blktrace / seekwatcher are good places to start. See https://github.com/znmeb/LinuxCon2009/tree/master/Linux_Server_Profiling_Usi... https://github.com/znmeb/LinuxCon2009/tree/master/Modeling_the_Linux_Block_I... On Wed, Jan 18, 2012 at 1:32 AM, <rst@suissimage.ch> wrote:
Hi y'all,
I've been directed here by the opensuse forums about a problem we are having with our server since we upgraded from opensuse 11.2 (kernel 2.6.31 I believe) to 12.1.
The problem is that one process can hog all disk io and starves others. For example progress database restore of multi GB DB starves all others for example mysqld. We see latencies on fsync for mysqld of 15s + with cfq block io scheduler. Still 5s+ with deadline block io scheduler and read_expire reduced to 20ms.
Been unable to reduce latency for other processes any further.
Our guess to the culprit is that the improvement that was made in 2.6.37 for smp ext4 block io throughput (300-400% according to Linux 2 6 37 - Linux Kernel Newbies ) has made it possible for one process to be that fast and created this starvation problem.
Or maybe some kernel bug.
Anybody have any pointers about how to reign in disk-io hogs in 3.1?
best regards
remo
Some info about the Server: Dell T710 with 2 Xeon 6 core procs, 48GB Memory. 6x300GB Disks in RAID10 on a H700 Raid Controller.
We didn't mess with many default Suse Kernel values. Except swapiness, default blocksize of Tape Driver, Max Semaphore and Shared Memory Segment Values ( /proc/sys/kernel/shmmax shmmni shmall). And of course the ioscheduler as deadline scheduler makes the system less unusable...
I'll gladly provide any other info y'all might need to help us improve this starvation issue.
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
-- http://twitter.com/znmeb http://borasky-research.net "A mathematician is a device for turning coffee into theorems." -- Paul Erdős -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org