Mailinglist Archive: opensuse-kernel (85 mails)

< Previous Next >
Re: [opensuse-kernel] after upgrade from 11.2 to 12.1: disk io hog/starvation issue on HW Raid ext4
  • From: "M. Edward (Ed) Borasky" <znmeb@xxxxxxxxxxxxxxxxxxxx>
  • Date: Wed, 18 Jan 2012 10:26:35 -0800
  • Message-id: <CAPmDps=dPfgbTbHWCbfHthtG7FCG8knN=e9E22PNW6UqayW73w@mail.gmail.com>
blktrace / seekwatcher are good places to start. See

https://github.com/znmeb/LinuxCon2009/tree/master/Linux_Server_Profiling_Using_Open_Source_Tools_for_Bottleneck_Analysis
https://github.com/znmeb/LinuxCon2009/tree/master/Modeling_the_Linux_Block_I-O_Layer_For_Enterprise_Applications

On Wed, Jan 18, 2012 at 1:32 AM, <rst@xxxxxxxxxxxxx> wrote:
Hi y'all,

I've been directed here by the opensuse forums about a problem we are having
with our server since we upgraded from opensuse 11.2 (kernel 2.6.31 I believe)
to 12.1.

The problem is that one process can hog all disk io and starves others. For
example progress database restore of multi GB DB  starves all others for
example mysqld.
We see latencies on fsync for mysqld of 15s + with cfq block io scheduler.
Still 5s+ with deadline block io scheduler and read_expire reduced to 20ms.

Been unable to reduce latency for other processes any further.

Our guess to the culprit is that the improvement that was made in 2.6.37 for
smp ext4 block io throughput (300-400% according to
Linux 2 6 37 - Linux Kernel Newbies ) has made it possible for one process to
be that fast and created this starvation problem.

Or maybe some kernel bug.

Anybody have any pointers about how to reign in disk-io hogs in 3.1?


best regards


remo

Some info about the Server:
Dell T710 with 2 Xeon 6 core procs, 48GB Memory. 6x300GB Disks in RAID10 on a
H700 Raid Controller.

We didn't mess with many default Suse Kernel values. Except swapiness,
default blocksize of Tape Driver, Max Semaphore and Shared Memory Segment
Values ( /proc/sys/kernel/shmmax shmmni shmall). And of course the
ioscheduler as deadline scheduler makes the system less unusable...

I'll gladly provide any other info y'all might need to help us improve this
starvation issue.

--
To unsubscribe, e-mail: opensuse-kernel+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-kernel+owner@xxxxxxxxxxxx



--
http://twitter.com/znmeb http://borasky-research.net

"A mathematician is a device for turning coffee into theorems." -- Paul Erdős
--
To unsubscribe, e-mail: opensuse-kernel+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-kernel+owner@xxxxxxxxxxxx

< Previous Next >
References