[Bug 668392] New: Using md raid yields to high system load
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c0 Summary: Using md raid yields to high system load Classification: openSUSE Product: openSUSE 11.3 Version: Final Platform: x86-64 OS/Version: openSUSE 11.3 Status: NEW Severity: Major Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: munderl@tnt.uni-hannover.de QAContact: qa@suse.de Found By: --- Blocker: --- User-Agent: Opera/9.80 (X11; Linux x86_64; U; de) Presto/2.7.62 Version/11.01 We upgraded some servers from 11.0 and 11.1 to 11.3 and recognized that they have become very slow when it comes to i/o operations. All of them use a md-raid1. The i/o performance is not only low on the md-raid itself but influences all i/o to any disk (even external scsi raids). We first recognized an md-resync taking months to complete because of just syncing with << 1 MB/s. Setting the min sync speed to 20 MB/s pushed it up to just 10 MB/s while the machine nearly stopped responding. System load is >3 with nearly no other process running. We also investigated our other 11.3 servers using md-raid1 and found that all behave poor. We have a internal webserver and mysql running on on md-raid with load >3 but nearly no access to the webserver or db. top always shows high load, but all other cpu values (even wa) are nearly 0 (except idle...). Then found that a lot of processes accessing the disks are in D state - they seem to wait for the disk. nmon shows 100% disk usage with only 5MB/s. This also happens when accessing an external raid which has nothing to do with the internal software raid. We removed one of the two disks of the md-raid1 with no performance increase. We then totally removed the md-raid and got full performance again and 0.00 load. After this we tried to use dm-raid for one of the machine which has bios-support for this. Using two mirror disk with this software raid also gives no performance problems. So just md seems to be effected. I'm totally stuck here because I can not find anyone else in the internet having this problem. But as we have it at least on 3 machines and only since we upgraded to 11.3 there must be a problem somewhere. Hardware has not changed. All machines have sata-drives but with different controllers (ahci/sata_mv). Reproducible: Always Steps to Reproduce: 1. Install md-raid1. 2. Boot openSUSE 11.3 3. Start several processes performing some i/o. 4. Start top and watch load go high (>3) and machine becoming slow Actual Results: Hmm. Except the raid1 there is no special setup involved. Expected Results: Give the same i/o results has without md like in 11.0. I don't know what you need on additional information. There is no error message involved, just this high load (with wa% low) and a lot of processes jumping on and off D state. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c1 Marco Munderloh <munderl@tnt.uni-hannover.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |munderl@tnt.uni-hannover.de --- Comment #1 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-01-31 17:37:32 UTC --- Oh, I forgot: we tried different kernels provided by openSUSE like the default and the desktop kernel, the vanilla kernel and even the kernel-of-the-day (2.6.37) with no effect. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c2 --- Comment #2 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-01-31 17:50:59 UTC --- Here is some output of top while a tar is running on an external raid (not the md). The high kswapd do not occur on the other machines, there other processes produce high load (like jbd2). I guess this depends on the filesystem used. If swap is turned off, the kswapd process persists. And swap is not used, anyway. tar just manages to get 5 MB/s at the moment. top - 18:42:56 up 32 days, 6:17, 3 users, load average: 3.27, 3.38, 3.38 Tasks: 168 total, 4 running, 164 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 98.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 1.3%si, 0.0%st Mem: 5996624k total, 5970408k used, 26216k free, 1162432k buffers Swap: 8393852k total, 7276k used, 8386576k free, 4393508k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 57 root 20 0 0 0 0 R 80 0.0 3325:44 kswapd0 58 root 20 0 0 0 0 R 60 0.0 3729:34 kswapd1 27826 root 20 0 10884 1540 708 R 39 0.0 1026:15 tar -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c3 --- Comment #3 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-01-31 17:55:39 UTC --- Output of sar: 18:55:00 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 18:55:01 dev8-16 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev8-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev8-32 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev8-48 139,39 5971,72 0,00 42,84 0,97 7,01 6,99 97,37 18:55:01 dev8-64 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev9-1 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev9-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev9-2 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev9-3 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev8-80 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 18:55:01 dev8-96 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c4 --- Comment #4 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-02-03 18:36:31 UTC --- Dropping the filesystem caches/buffers with echo 1 > /proc/sys/vm/drop_caches fixes the problem - but only for around 2 min. The kswapd go to 0% and the transfer rate of the tar jumps up to 80 MB/s. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c Jeff Mahoney <jeffm@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffm@novell.com AssignedTo|kernel-maintainers@forge.pr |nfbrown@novell.com |ovo.novell.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c5 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |munderl@tnt.uni-hannover.de --- Comment #5 from Neil Brown <nfbrown@novell.com> 2011-02-08 04:29:50 UTC --- This all sounds very strange. You seem to have a serious problem with writeback writing out dirty memory very inefficiently, though I cannot explain why that would happen. I cannot see how md could be implicated in such behaviour, yet you seem to have evidence that it is. One thing that might help to is to echo t > /proc/sysrq-trigger when things are going bad, and collect the output of dmesg which should have a stack trace for each process. It could well turn out that the dmesg buffer isn't big enough to hold all of the output. In that case you would need to boot with log_bug_len=8M or something like that. If you can collect that while they system is very sluggish and attach the result, that might help. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c6 Marco Munderloh <munderl@tnt.uni-hannover.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|munderl@tnt.uni-hannover.de | --- Comment #6 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-02-08 10:00:33 UTC --- Created an attachment (id=412751) --> (http://bugzilla.novell.com/attachment.cgi?id=412751) dmesg output of sysrq-t Find attached the dmesg output of the stack trace. The file starts with the end of an earlier stack trace, so log_buffer seems to be just big enough. Load was around 3 at that point with no process really doing something (maybe the smbd/nfsd). kswapd pops up to 100% every 10s for around 5s. I started an additional tar to get it to 100% all the time. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c7 --- Comment #7 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-02-08 10:06:49 UTC --- Created an attachment (id=412752) --> (http://bugzilla.novell.com/attachment.cgi?id=412752) stack trace after dropping buffers This is a stack trace after flushing the buffers. No kswapds, low load, tar running smoothly and fast. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c8 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |munderl@tnt.uni-hannover.de --- Comment #8 from Neil Brown <nfbrown@novell.com> 2011-02-22 01:55:37 UTC --- Thanks for the logs. Unfortunately they don't show much. 'tar' is the only process that is doing any significant. I note that the tar process is reading an XFS filesystem. I wonder if the problem could be related to xfs rather than md/raid. Are you able to test with a different filesystem and see if that makes any difference? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c9 --- Comment #9 from Neil Brown <nfbrown@novell.com> 2011-04-11 07:09:51 UTC --- Any news? Is this still a problem? Can we close this bug? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c10 Marco Munderloh <munderl@tnt.uni-hannover.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|munderl@tnt.uni-hannover.de | --- Comment #10 from Marco Munderloh <munderl@tnt.uni-hannover.de> 2011-04-14 15:02:50 UTC --- Sorry for the delay, I'm quite busy atm. So the problem does not seem to be md/raid but somthing in the io-scheduling subsystem. md/raid just triggers this issue as it produces more concurrent io load. We also have problems with other filesystems as ext4. The 100% processes differ throughout the systems and filesystems being used. They just show 100% as they have to wait for something (D state). This also causes the overall slowdown of the system. Dropping caches always helps, but only for a very small period of time (seconds). On one machine I installed the latest 2.3.32 kernel and all problems seem to be gone now (couldn't test that much yet). So it seems to be some change between 2.6.32 and 2.6.34. Wasn't something done on the big kernel lock in relation with the io-subsystem? Can this be the cause of the problem? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c11 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #11 from Neil Brown <nfbrown@novell.com> 2011-04-18 06:08:26 UTC --- The have been changed with the BKL lately - mostly removing it. While I wouldn't want to rule anything out, I see no reason to suspect the BKL changes particularly. As you have a work around, and we don't have much evidence to go on, I won't be giving this bug a lot of effort. It would be interesting to know if it is fixed in 11.4, but I don't really expect you to install that just to test. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=668392 https://bugzilla.novell.com/show_bug.cgi?id=668392#c12 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |WORKSFORME --- Comment #12 from Neil Brown <nfbrown@novell.com> 2011-06-02 06:53:46 UTC --- I don't think there is anything more we can do here and you seem to not be suffering from a problem any more, so I'm resolving this as WORKSFORME. Sorry would couldn't find a more complete resolution. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com