On Thu, Jan 30, 2020 at 6:53 PM Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> wrote:
In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot?
It's in pages, so multiply by 4K. Active anonymous memory alone is around 6GB.
Ah, thanks - I was thinking it was kilobytes.
This is known problem. There is a lot of active memory so every time kernel has to search a lot to find free page (or page that can be reclaimed).
that could lead to an oom condition?
Rather this is indication of low memory. In this case free RAM is below absolute minimum that kernel attempts to keep: Node 0 Normal free:102036kB min:106080kB Which means on every allocation request kernel will synchronously attempt to free enough memory to go above watermark. It looks like it tries it actually: writeback:1251356kB writeback:1671120kB writeback:1876996kB writeback:1982080kB writeback:1961032kB writeback:1964848kB writeback:1914756kB writeback:1914500kB writeback:250628kB writeback:317124kB writeback:372888kB writeback:402920kB writeback:558956kB writeback:656724kB writeback:696416kB writeback:736480kB writeback:792932kB writeback:1045984kB writeback:1035232kB writeback:278136kB writeback:274188kB This started with almost 2GB of data that is being written to disk. At some point kernel managed to complete it just to get more and more new data to be written. If kernel needs to wait while dirty pages are written to disk it pretty much explains those "allocation stalled" warnings. This goes on until no more free swap is available (which probably means - kernel cannot free memory to go above watermark at all) at which point OOM kicks in. So it appears like some application(s) really use that much memory. IOW system is overloaded. As for "If you sudo swapoff -a the swap turns into memory, the system turns responsive" it is far too vague to make any useful comment. It is impossible to do "swapoff" under conditions shown in dmesg output. If this was supposed to mean "swapoff before this situation is reached" - sure, it will eliminate stalls due to writing data to disk, it will also cause OOM earlier. If RAM+swap is not enough for current workload, then RAM alone is obviously not enough as well. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org