http://bugzilla.opensuse.org/show_bug.cgi?id=1129214 http://bugzilla.opensuse.org/show_bug.cgi?id=1129214#c3 --- Comment #3 from Christian Wittmer --- (In reply to Michal Hocko from comment #2)

These all seem to be temporary allocation stalls 3 hitting one window Mar 8 and one mar 13. One hitting xfs, 2 skb and one vfs allocation paths. So nothing really systematic. Checking the free memory proves that the memory was mostly used.

I am strongly suspecting that some part of the memory reclaim (shrinkers probably) took excessive amount of time to make a forward progress. We would need vmsacan tracepoints data to know better though.

How can I provide such info about 'vmsacan tracepoints data' ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1129214 http://bugzilla.opensuse.org/show_bug.cgi?id=1129214#c7 Christian Wittmer changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(ailiopoulos@suse. | |com) --- Comment #7 from Christian Wittmer --- (In reply to Anthony Iliopoulos from comment #5)

Looks like rsync is exerting a lot of memory pressure, and potentially writeback at the same time cannot keep up (it seems that the md layer has pending data to be flushed). Perhaps also a stacktrace of the blocked tasks (echo w > /proc/sysrq-trigger) would give a bit more insight next time this occurs.

So when 'rsync' is exertimg memory again I should do the 'echo w ...' command ? Where will I see the 'bit more insight' then ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1129214 http://bugzilla.opensuse.org/show_bug.cgi?id=1129214#c9 Anthony Iliopoulos changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(mhocko@suse.com) | --- Comment #9 from Anthony Iliopoulos --- (In reply to Christian Wittmer from comment #6)

min)(In reply to Michal Hocko from comment #4)

...

There are two ways. You can either mount tracefs and enable $TRACEFS_MNT/events/vmscan/{mm_vmscan_direct_reclaim_begin, mm_vmscan_direct_reclaim_end,mm_shrink_slab_start,mm_shrink_slab_end} and How do I enable ?

...

read the output from $TRACE_MNT/trace_pipe or use trace-cmd to do the same. sorry I am not that professional. I would need more detailed info about what to do ...

You can install the trace-cmd package, and run something like: sudo trace-cmd start -e mm_vmscan_direct_reclaim_begin -e mm_vmscan_direct_reclaim_end -e mm_shrink_slab_start -e mm_shrink_slab_end This will enable tracing and recording of those enabled events in a dedicated kernel buffer. You can dump the buffer anytime via trace-cmd show. You can stop the tracing of the events via trace-cmd stop. The recorded events buffer will still be available for reading/dumping into a file via trace-cmd show, until you do trace-cmd reset that will clear everything.

...

Btw. is this reproducible? probably when the 'rsync' jobs are running again ...

some days ago the server died cause of a stack trace. I needed to 'reset' which caused a raid failure. After repairing the raid (rebuild is still running ... 900min) I will provide logs ...

I'd suggest setting up kdump (unless it's already set by default), see [1] for more info and setting up via yast etc. Next time you need to forcibly reset/reboot the system for whatever reason, you can trigger a crash via echo c > /proc/sysrq-trigger. This will force a kernel reboot as well as leverage kdump to keep a crash-dump of the kernel memory that we can use to analyze the problem in-depth. [1] https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.... -- You are receiving this mail because: You are on the CC list for the bug.