Re: [opensuse-factory] Deal with low memory situations
Michal's reply to factory is stuck as he's not subscribed, so forwarding: -------- Forwarded Message -------- Subject: Re: [opensuse-factory] Deal with low memory situations Date: Tue, 7 Jan 2020 15:21:36 +0100 From: Michal Hocko <mhocko@suse.com> To: Vlastimil Babka <vbabka@suse.cz> CC: Stasiek Michalski <hellcp@opensuse.org>, opensuse.lietuviu.kalba <opensuse.lietuviu.kalba@gmail.com>, opensuse-factory@opensuse.org On Tue 07-01-20 14:37:53, Vlastimil Babka wrote:
On 1/5/20 11:32 AM, Stasiek Michalski wrote:
On Sun, Jan 5, 2020 at 11:16, opensuse.lietuviu.kalba <opensuse.lietuviu.kalba@gmail.com> wrote:
Hi,
Fedora developers had been figuring out ways to improve Linux interactivity in low-RAM environments and Fedora 32 could begin shipping and using EarlyOOM by default to help in this area. What opinions are from openSUSE community to do similar steps?
It seems abundance of electron is getting some recognition in the OSS circles now :P
Joking aside, it would be a great addition to the default desktop patterns if anybody is willing to maintain it, OOM on Linux does tend to get annoying with low ram systems.
I would advise against enabling this by default. Simple conditions like "if both RAM and swap go below 10% free" are just not reliable enough and will cause premature OOM's. The current upstream direction towards pressure stall (PSI) tracking seems much better IMHO.
I would also add that simple threshold on the amount of the free memory is highly unlikely to fix the most common shortcoming of the oom killer which is a lack of threshing detection. There usually is a free memory in those situations which is rapidly recycled for the working set which is quickly thrown away. 10% also sound quite high, especially on machines with a lot of memory. On a 8GB machine (e.g. my laptop) the min_free_kbytes is 67584 (~68MB). This is where the direct reclaim kicks in a reclaims memory. The background reclaim kicks in when so called low watermark is hit and that is roughly 84MB on that machine. I have quickly checked how the daemon is implemented and it checks MemAvailable (from /proc/meminfo) which tries to estimate how much memory can be allocated before the systems starts swapping (essentially excludes page cache and reclaimable slab from MemFree). Now take 10% from 8GB and 780MB so _no_ reclaim doesn't even get chance to do anything before there is SIGTERM sent for workloads without a lot of page cache. While this might work for some cases I wouldn't be surprised if there was a lot false positives. Global oom killer is a last resort measure when the system is completely out of memory. If there is a more proactivity required and I can see why there is a need for that in some cases then I would recommend using something based on the PSI e.g oomd https://github.com/facebookincubator/oomd -- Michal Hocko SUSE Labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Jan 8, 2020 at 13:51, Vlastimil Babka <vbabka@suse.cz> wrote:
Michal's reply to factory is stuck as he's not subscribed, so forwarding:
-------- Forwarded Message -------- Subject: Re: [opensuse-factory] Deal with low memory situations Date: Tue, 7 Jan 2020 15:21:36 +0100 From: Michal Hocko <mhocko@suse.com> To: Vlastimil Babka <vbabka@suse.cz> CC: Stasiek Michalski <hellcp@opensuse.org>, opensuse.lietuviu.kalba <opensuse.lietuviu.kalba@gmail.com>, opensuse-factory@opensuse.org
On Tue 07-01-20 14:37:53, Vlastimil Babka wrote:
On 1/5/20 11:32 AM, Stasiek Michalski wrote:
On Sun, Jan 5, 2020 at 11:16, opensuse.lietuviu.kalba <opensuse.lietuviu.kalba@gmail.com> wrote:
Hi,
Fedora developers had been figuring out ways to improve Linux interactivity in low-RAM environments and Fedora 32 could begin shipping and using EarlyOOM by default to help in this area. What opinions are from openSUSE community to do similar steps?
It seems abundance of electron is getting some recognition in the
OSS
circles now :P
Joking aside, it would be a great addition to the default desktop patterns if anybody is willing to maintain it, OOM on Linux does tend to get annoying with low ram systems.
I would advise against enabling this by default. Simple conditions like "if both RAM and swap go below 10% free" are just not reliable enough and will cause premature OOM's. The current upstream direction towards pressure stall (PSI) tracking seems much better IMHO.
I would also add that simple threshold on the amount of the free memory is highly unlikely to fix the most common shortcoming of the oom killer which is a lack of threshing detection. There usually is a free memory in those situations which is rapidly recycled for the working set which is quickly thrown away. 10% also sound quite high, especially on machines with a lot of memory. On a 8GB machine (e.g. my laptop) the min_free_kbytes is 67584 (~68MB). This is where the direct reclaim kicks in a reclaims memory. The background reclaim kicks in when so called low watermark is hit and that is roughly 84MB on that machine.
I have quickly checked how the daemon is implemented and it checks MemAvailable (from /proc/meminfo) which tries to estimate how much memory can be allocated before the systems starts swapping (essentially excludes page cache and reclaimable slab from MemFree). Now take 10% from 8GB and 780MB so _no_ reclaim doesn't even get chance to do anything before there is SIGTERM sent for workloads without a lot of page cache.
While this might work for some cases I wouldn't be surprised if there was a lot false positives.
Global oom killer is a last resort measure when the system is completely out of memory. If there is a more proactivity required and I can see why there is a need for that in some cases then I would recommend using something based on the PSI e.g oomd https://github.com/facebookincubator/oomd
Alright, let's address this properly, oomd is not a good solution for the desktop. It does not support SIGTERM before SIGKILL, which is why Fedora decided not to use it in the end, and it's why Facebook decided to put this functionality on the roadmap (which, because it's Facebook, will take them a few months at best). low-memory-monitor is a great idea, but it relies on applications handling killing themselves, which will hopefully be handled from larger frameworks (KF, GTK, Electron) to simplify the implementations, but isn't yet. The unfortunate reality of the kernel killer is that it does tend to be a bit overzealous, which is absolutely unacceptable for the desktop use cases, like the one presented by one of the issues on low-memory-monitor [1]. That's the reason why the preferred method is to have the applications handle this themselves, and low-memory-monitor no longer asks the kernel to kill anything by default, instead being only responsible for asking applications to kill themselves. And if you really prefer to have all of the instances of your terminal killed instead of the process that's generating pressure stall, you are free to submit appropriate changes adding a tool like oomd to the base patterns. Otherwise, I suggest we go the suggested route of having something basic that kills the processes responsible for the ram usage instead. [1] https://gitlab.freedesktop.org/hadess/low-memory-monitor/issues/8 LCP [Stasiek] https://lcp.world -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (2)
-
Stasiek Michalski -
Vlastimil Babka