https://bugzilla.novell.com/show_bug.cgi?id=763399 https://bugzilla.novell.com/show_bug.cgi?id=763399#c0 Summary: excessive swap out due to compaction/LRU aging Classification: openSUSE Product: openSUSE 12.1 Version: Final Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: mhocko@suse.com QAContact: qa-bugs@suse.de CC: ke@suse.com, mhocko@suse.com, mgorman@suse.com Found By: --- Blocker: --- This is a follow up for bug 719416. Karl is still seeing serious hick ups which seem to be caused by excessive swap out. From the vmstat logs (which he provided at https://bugzilla.novell.com/attachment.cgi?id=488809) we can clearly see that we really swap out a lot: $ grep pswpout bug-719416_vmstat.log | uniq -c 3 pswpout 4170 719 pswpout 5472 27 pswpout 5540 5 pswpout 5612 1 pswpout 5788 908 pswpout 5903 755 pswpout 6024 We swap out 1300 pages within a second. If we have a look at LRUs during that time period: $ grep nr_inactive_anon bug-719416_vmstat.log | head -n 6 nr_inactive_anon 55889 nr_inactive_anon 55889 nr_inactive_anon 55889 nr_inactive_anon 58193 <<< 2304 nr_inactive_anon 58193 nr_inactive_anon 58193 $ grep nr_active_anon bug-719416_vmstat.log | head -n 6 nr_active_anon 200718 nr_active_anon 200742 nr_active_anon 200742 nr_active_anon 109727 <<< -91015 nr_active_anon 109741 nr_active_anon 109763 First of all, inactive list is really low (active list is 4 times bigger!). The difference gets lower after 3rd second (when we start swapping). Inactive list gets slightly bigger while the active list get's down really considerably. $ grep nr_inactive_file bug-719416_vmstat.log | head -n 6 nr_inactive_file 219248 nr_inactive_file 219259 nr_inactive_file 219258 nr_inactive_file 266880 <<< 47622 nr_inactive_file 266880 nr_inactive_file 266883 $ grep nr_active_file bug-719416_vmstat.log | head -n 6 nr_active_file 192618 nr_active_file 192624 nr_active_file 192626 nr_active_file 218326 <<< 25700 nr_active_file 218326 nr_active_file 218326 Page cache went considerably up (by ~70k pages) which suggests that we had a hungry reader at that time. active vs. inactive lists seem to be balanced properly. $ grep nr_free_pages bug-719416_vmstat.log | head -n6 nr_free_pages 25275 nr_free_pages 25269 nr_free_pages 25267 nr_free_pages 33473 nr_free_pages 33440 nr_free_pages 33481 I guess we hit the water marks. Did we hit the direct reclaim as well? $ grep allocstall bug-719416_vmstat.log | head -n6 allocstall 45 allocstall 45 allocstall 45 allocstall 1992 <<< Yeah, we did And what about compaction? $ grep compact_pages_moved bug-719416_vmstat.log | head -n6 compact_pages_moved 131651 compact_pages_moved 131651 compact_pages_moved 131651 compact_pages_moved 186924 <<< 55273 compact_pages_moved 186924 compact_pages_moved 186924 $ grep compact_blocks_moved bug-719416_vmstat.log | head -n6 compact_blocks_moved 21117 compact_blocks_moved 21117 compact_blocks_moved 21117 compact_blocks_moved 26055 <<< 4938 compact_blocks_moved 26055 compact_blocks_moved 26055 Was the compaction successful? grep compact_fail bug-719416_vmstat.log | head -n6 compact_fail 53 compact_fail 53 compact_fail 53 compact_fail 83 <<< 30 compact_fail 83 compact_fail 83 $ grep compact_success bug-719416_vmstat.log | head -n6 compact_success 119 compact_success 119 compact_success 119 compact_success 172 <<< 53 compact_success 172 compact_success 172 Hmm, we failed only 30 times while succeeded 53. So the remaining question is who asked for the high order allocations? Check the usual suspects first: $ grep thp_fault_alloc bug-719416_vmstat.log | head -n6 thp_fault_alloc 1570 thp_fault_alloc 1570 thp_fault_alloc 1570 thp_fault_alloc 1851 <<< 281 thp_fault_alloc 1851 thp_fault_alloc 1851 Hmm, we wanted 281 huge pages, that's quite a lot. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.