Re: [opensuse-kernel] Memory stalls after upgrading from Kernel 4.4 to 4.12
Hi, Am 28.08.2018 um 08:25 schrieb Michal Hocko:
On Sat 25-08-18 09:54:34, Stefan Priebe - Profihost AG wrote:
Hello,
not sure if related but since upgrading from Kernel 4.4 to 4.12 based SLES15 kernel i had two times the following traces while the system was nearly unusable: [245513.362669] kvm: page allocation stalls for 194572ms, order:9, mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)
This is an THP allocation and from __GFP_DIRECT_RECLAIM I assume it is within MADV_HUGEPAGE mapping. What is your defrag setting? cat /sys/kernel/mm/transparent_hugepage/defrag
spending 194s reclaiming/compacting is definitely way too much for THP. This looks like an issue reported by Andrea recently http://lkml.kernel.org/r/20180820032204.9591-1-aarcange@redhat.com
We do not have a proper solution for these pathological cases yet though.
Yes this sounds pretty much exactly the issue i'm seeing on multiple kvm hosts. So this is a regression in SLES15? # cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise [madvise] never Greets, Stefan -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
[my previous email to the mailing list got stuck waiting for moderator so I am changing my From to suse.cz] On Tue 28-08-18 08:52:58, Stefan Priebe - Profihost AG wrote:
Hi,
Am 28.08.2018 um 08:25 schrieb Michal Hocko:
On Sat 25-08-18 09:54:34, Stefan Priebe - Profihost AG wrote:
Hello,
not sure if related but since upgrading from Kernel 4.4 to 4.12 based SLES15 kernel i had two times the following traces while the system was nearly unusable: [245513.362669] kvm: page allocation stalls for 194572ms, order:9, mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)
This is an THP allocation and from __GFP_DIRECT_RECLAIM I assume it is within MADV_HUGEPAGE mapping. What is your defrag setting? cat /sys/kernel/mm/transparent_hugepage/defrag
spending 194s reclaiming/compacting is definitely way too much for THP. This looks like an issue reported by Andrea recently http://lkml.kernel.org/r/20180820032204.9591-1-aarcange@redhat.com
We do not have a proper solution for these pathological cases yet though.
Yes this sounds pretty much exactly the issue i'm seeing on multiple kvm hosts. So this is a regression in SLES15?
Well, spending so much time in the allocation is clearly undesirable.
# cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise [madvise] never
OK, so the immediate workaround would be to weaken the defrag mode to defer. You will likely not get as many THP but stalls should be gone. If you are brave enough and willing to play a bit then you can try to apply the patch I was proposing https://marc.info/?l=linux-mm&m=153493606221018. I guess we will end up with a different solution in the end but seeing how this behaves if you have a workload which triggers the bad behavior would be valuable. Thanks! -- Michal Hocko SUSE Labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (2)
-
Michal Hocko -
Stefan Priebe - Profihost AG