-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/01/2020 08.04, Andrei Borzenkov wrote: | On Thu, Jan 30, 2020 at 6:53 PM Per Jessen <per@computer.org> | wrote: |> |> Andrei Borzenkov wrote: |> |>> On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> |>> wrote: |>> |>>> In the process lists from dmesg, I see 266 processes using up |>>> about 1Gb (RSS) which doesn't seem like a lot? |>> |>> It's in pages, so multiply by 4K. Active anonymous memory alone |>> is around 6GB. |> |> Ah, thanks - I was thinking it was kilobytes. |> |>> This is known problem. There is a lot of active memory so every |>> time kernel has to search a lot to find free page (or page that |>> can be reclaimed). |> |> that could lead to an oom condition? |> | | Rather this is indication of low memory. | | In this case free RAM is below absolute minimum that kernel | attempts to keep: | | Node 0 Normal free:102036kB min:106080kB Ah! That's the start condition? I also have that in the log of my incident. <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921018] Node 0 Normal free:105704kB min:106040kB low:116644kB high:127248kB That's about 100 Mb, right? active_anon:1885752kB inactive_anon:626456kB active_file:371848kB inactive_file:1478016kB unevictable:10880kB writepending:239740kB present:5242880kB managed:5110480kB mlocked:10880kB slab_reclaimable:170100kB slab_unreclaimable:162136kB kernel_stack:21080kB pagetables:91024kB bounce:0kB free_pcp:2776kB local_pcp:1280kB free_cma:0kB Any document you know that explains these? And no knowing how the machine got there. | | Which means on every allocation request kernel will synchronously | attempt to free enough memory to go above watermark. It looks like | it tries it actually: | | writeback:1251356kB writeback:1671120kB writeback:1876996kB | writeback:1982080kB writeback:1961032kB I have one instance of that word. <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921004] Node 0 active_anon:2560096kB inactive_anon:1335884kB active_file:461648kB inactive_file:2786820kB unevictable:10880kB isolated(anon):0kB isolated(file):256kB mapped:495892kB dirty:23756kB writeback:391168kB shmem:212684kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 382976kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no | | This started with almost 2GB of data that is being written to | disk. How do you know that? :-? (I want to find it in my case) My machine was at the time probably copying a large directory from one disk to another, using 'mc', and nobody seated. | At some point kernel managed to complete it just to get more and | more new data to be written. If kernel needs to wait while dirty | pages are written to disk it pretty much explains those "allocation | stalled" warnings. | | This goes on until no more free swap is available (which probably | means - kernel cannot free memory to go above watermark at all) at | which point OOM kicks in. I don't think that happened in my case, but I do not know. I have 25 GB of swap. | So it appears like some application(s) really use that much | memory. IOW system is overloaded. | | As for "If you sudo swapoff -a the swap turns into memory, the | system turns responsive" it is far too vague to make any useful | comment. It is impossible to do "swapoff" under conditions shown in | dmesg output. If this was supposed to mean "swapoff before this | situation is reached" - sure, it will eliminate stalls due to | writing data to disk, it will also cause OOM earlier. If RAM+swap | is not enough for current workload, then RAM alone is obviously not | enough as well. This may be because read/write to swap with rotating disk is way too slow and involves many seeks. I observed that in my machine when I switched to Leap, and stopped when I bought an SSD disk and put the swap there. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjQQaQAKCRC1MxgcbY1H 1dlTAJwOC6qwNEkmm7lPvOKMtwh6z3B0+wCfQhWQ0MNjYns3IlD6MHrL1G+TvZM= =GZmz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org