[opensuse-kernel] Re: 2.6.34-rc3: simple du (on a big xfs tree) triggers oom killer [bisected: 57817c68229984818fea9e614d6f95249c3fb098]
On Wednesday 07 April 2010, 03:45:33 Dave Chinner wrote:
However, if the memory pressure is purely inode cache (creating zero length files or read-only traversal), then the OOM killer kicks a while after the slab cache fills memory. This doesn't need highmem; I used a x86_64 kernel on a VM w/ 1GB RAM to reliably reproduce this. I'll add zero length file tests and traversals to my low memory testing.
I'm glad, that you're able to reproduce it. My initial failure was during disk to disk backup (with a simple cp -al & rsync combination).
The best way to fix this, I think, is to trigger a shrinker callback when memory is low to run the background inode reclaim. The problem is that these inode caches and the reclaim state are per-filesystem, not global state, and the current shrinker interface only works with global state.
Hence there are two patches to this fix - the first adds a context to the shrinker callout, and the second adds the XFS infrastructure to track the number of reclaimable inodes per filesystem and register/unregister shrinkers for each filesystem.
I see, the first one will be interesting to get into mainline, given the number of projects, that are involved.
With these patches, my reproducable test case which locked the machine up with a OOM panic in a couple of minutes has been running for over half an hour. I have much more confidence in this change with limited testing than the reverting of the background inode reclaim as the revert introduces
The patches below apply to the xfs-dev tree, which is currently at 34-rc1. If they don't apply, let me know and I'll redo them against a vanilla kernel tree. Can you test them to see if the problem goes away? If the problem is fixed, I'll push them for a proper review cycle...
Of course, you did the original patch for a reason... Therefor I would love to test your patches. I've tried to apply them to 2.6.33.2, but after fixing the same reject as noted below, I'm stuck here: /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux-2.6/xfs_sync.c: In function 'xfs_reclaim_inode_shrink': /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux-2.6/xfs_sync.c:805: error: implicit declaration of function 'xfs_perag_get' /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux-2.6/xfs_sync.c:805: warning: assignment makes pointer from integer without a cast /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux-2.6/xfs_sync.c:807: error: implicit declaration of function 'xfs_perag_put' Now I see, that there happened a rename of the offending functions, but also they've grown a radix_tree structure and locking. How do I handle that? BTW, your patches do not apply to Linus' current git tree either: patching file fs/xfs/quota/xfs_qm.c Hunk #1 succeeded at 72 (offset 3 lines). Hunk #2 FAILED at 2120. 1 out of 2 hunks FAILED -- saving rejects to file fs/xfs/quota/xfs_qm.c.rej I'm able to resolve this, but 2.6.34-current does give me some other trouble, that I need to get by (PS2 keyboard stops working eventually).. Anyway, thanks for your great support, Dave. This is much appreciated. Cheers, Pete -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
FYI: this issue is going to bite YOU with the SLE11-SP1 kernel as well. Just "du" a bigger tree, preferably on a high traffic server, and watch syslog. Note: any user can do this! My other issue of loosing the keyboard as already briefly mentioned here and reported to LKML is an issue of this kernel as well. See LKML: 2.6.34-rc3++ looses PS2 keyboard eventually in xterms As things stands, the most current kernel, that I'm able to use on my servers - which does not suffer from blatant stability issues - is 2.6.33.2 with the xfs reclaim patch reverted, while the best kernel for my (11.1) desktop is 2.6.31.13 because of the dreaded kbd issue there after. (Both are based on the opensuse-kernel git tree of the last two days with only minor modifications.) If somebody cares: all my kernel builds are available on BS: home:frispete:. (kernel-head is busted ATM due to Tejun's recent churn - I've lost interest to build, fix and test this tree, that just don't cut it for me...) These issues are giving me a real hard time. :-( Oh well, Pete On Thursday 08 April 2010, 00:02:20 Hans-Peter Jansen wrote:
On Wednesday 07 April 2010, 03:45:33 Dave Chinner wrote:
However, if the memory pressure is purely inode cache (creating zero length files or read-only traversal), then the OOM killer kicks a while after the slab cache fills memory. This doesn't need highmem; I used a x86_64 kernel on a VM w/ 1GB RAM to reliably reproduce this. I'll add zero length file tests and traversals to my low memory testing.
I'm glad, that you're able to reproduce it. My initial failure was during disk to disk backup (with a simple cp -al & rsync combination).
The best way to fix this, I think, is to trigger a shrinker callback when memory is low to run the background inode reclaim. The problem is that these inode caches and the reclaim state are per-filesystem, not global state, and the current shrinker interface only works with global state.
Hence there are two patches to this fix - the first adds a context to the shrinker callout, and the second adds the XFS infrastructure to track the number of reclaimable inodes per filesystem and register/unregister shrinkers for each filesystem.
I see, the first one will be interesting to get into mainline, given the number of projects, that are involved.
With these patches, my reproducable test case which locked the machine up with a OOM panic in a couple of minutes has been running for over half an hour. I have much more confidence in this change with limited testing than the reverting of the background inode reclaim as the revert introduces
The patches below apply to the xfs-dev tree, which is currently at 34-rc1. If they don't apply, let me know and I'll redo them against a vanilla kernel tree. Can you test them to see if the problem goes away? If the problem is fixed, I'll push them for a proper review cycle...
Of course, you did the original patch for a reason... Therefor I would love to test your patches. I've tried to apply them to 2.6.33.2, but after fixing the same reject as noted below, I'm stuck here:
/usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux -2.6/xfs_sync.c: In function 'xfs_reclaim_inode_shrink': /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux -2.6/xfs_sync.c:805: error: implicit declaration of function 'xfs_perag_get' /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux -2.6/xfs_sync.c:805: warning: assignment makes pointer from integer without a cast /usr/src/packages/BUILD/kernel-default-2.6.33.2/linux-2.6.33/fs/xfs/linux -2.6/xfs_sync.c:807: error: implicit declaration of function 'xfs_perag_put'
Now I see, that there happened a rename of the offending functions, but also they've grown a radix_tree structure and locking. How do I handle that?
BTW, your patches do not apply to Linus' current git tree either: patching file fs/xfs/quota/xfs_qm.c Hunk #1 succeeded at 72 (offset 3 lines). Hunk #2 FAILED at 2120. 1 out of 2 hunks FAILED -- saving rejects to file fs/xfs/quota/xfs_qm.c.rej I'm able to resolve this, but 2.6.34-current does give me some other trouble, that I need to get by (PS2 keyboard stops working eventually)..
Anyway, thanks for your great support, Dave. This is much appreciated.
Cheers, Pete
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (1)
-
Hans-Peter Jansen