[opensuse] Hardware failure or software problem? Swap fills up, system unresponsive and dmesg is flooded
I did create a paste in https://paste.opensuse.org/22334414 I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive. sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.) I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available? _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postf�cher sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive.
I'm unsure I understand :-?
sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.)
You could have edited them out ;-)
I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available?
Because what is stored there is not requested by anything, so better stay there out of the way. More RAM is free as a result. - -- Cheers, Carlos E. R. (from openSUSE 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjK15hwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVQkAAn3LXJ8i+ZYAHlPhfuIDx XyKkTLDAAJ9/Ds4ZdnoVvGygVIYfT4agXdGobg== =tfgH -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data giovedì 30 gennaio 2020 11:54:30 CET, Carlos E. R. ha scritto:
On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive.
I'm unsure I understand :-?
sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.)
You could have edited them out ;-)
I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available?
Because what is stored there is not requested by anything, so better stay there out of the way. More RAM is free as a result.
-- Cheers, Carlos E. R. (from openSUSE 15.1 x86_64 at Telcontar)
Unevictable, unreclamable, unresponsive........seems to be more a grave than a swap. Complains about a qt problem but I do not understand which kind of. While the system swaps so heavily the memory is free, not even a problem of cached memory. The system has 8GB and when this happens it has up to 25% of unallocated(!) memory, completely free. So this is definitely not normal. If I force back all to ram I have still 15 to 20% of memory free and unallocated. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data giovedì 30 gennaio 2020 12:05:42 CET, stakanov ha scritto:
In data giovedì 30 gennaio 2020 11:54:30 CET, Carlos E. R. ha scritto:
On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
Just an excerpt of the endless(!) /var/log/messages: local_pcp:1496kB free_cma:0kB 2020-01-29T18:25:02.652312+01:00 roadrunner kernel: [114550.815012] lowmem_reserve[]: 0 0 0 0 0 2020-01-29T18:25:02.652318+01:00 roadrunner kernel: [114550.815016] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB 2020-01-29T18:25:02.652321+01:00 roadrunner kernel: [114550.815028] Node 0 DMA32: 13861*4kB (UME) 2316*8kB (UME) 458*16kB (UME) 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 81332kB 2020-01-29T18:25:02.652322+01:00 roadrunner kernel: [114550.815038] Node 0 Normal: 17083*4kB (UME) 4592*8kB (UM) 11*16kB (UM) 2*32kB (UM) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 105372kB 2020-01-29T18:25:02.652323+01:00 roadrunner kernel: [114550.815051] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 2020-01-29T18:25:02.652323+01:00 roadrunner kernel: [114550.815052] 754045 total pagecache pages 2020-01-29T18:25:02.652324+01:00 roadrunner kernel: [114550.815056] 633151 pages in swap cache 2020-01-29T18:25:02.652325+01:00 roadrunner kernel: [114550.815057] Swap cache stats: add 1379720, delete 751984, find 164078/212978 2020-01-29T18:25:02.652325+01:00 roadrunner kernel: [114550.815058] Free swap = 3653232kB 2020-01-29T18:25:02.652326+01:00 roadrunner kernel: [114550.815059] Total swap = 7950332kB 2020-01-29T18:25:02.652327+01:00 roadrunner kernel: [114550.815060] 2043819 pages RAM 2020-01-29T18:25:02.718981+01:00 roadrunner kernel: [114550.815060] 0 pages HighMem/MovableOnly 2020-01-29T18:25:02.719005+01:00 roadrunner kernel: [114550.815061] 56631 pages reserved 2020-01-29T18:25:02.719006+01:00 roadrunner kernel: [114550.815062] 0 pages hwpoisoned 2020-01-29T18:25:02.719007+01:00 roadrunner kernel: [114551.313408] vivaldi- bin: page allocation stalls for 13448ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:02.719008+01:00 roadrunner kernel: [114551.313416] vivaldi- bin cpuset=/ mems_allowed=0 2020-01-29T18:25:02.719008+01:00 roadrunner kernel: [114551.313423] CPU: 3 PID: 27493 Comm: vivaldi-bin Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:02.719009+01:00 roadrunner kernel: [114551.313424] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:02.764044+01:00 roadrunner kernel: [114551.313426] Call Trace: 2020-01-29T18:25:02.764067+01:00 roadrunner kernel: [114551.313440] dump_stack+0x5c/0x86 2020-01-29T18:25:02.764069+01:00 roadrunner kernel: [114551.313445] warn_alloc+0xe0/0x170 2020-01-29T18:25:02.764069+01:00 roadrunner kernel: [114551.313449] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:02.764071+01:00 roadrunner kernel: [114551.313453] ? __do_page_cache_readahead+0xe0/0x270 2020-01-29T18:25:02.764072+01:00 roadrunner kernel: [114551.313456] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:02.764073+01:00 roadrunner kernel: [114551.313461] alloc_pages_current+0x72/0x140 2020-01-29T18:25:02.796052+01:00 roadrunner kernel: [114551.313466] filemap_fault+0x319/0x6b0 2020-01-29T18:25:02.796076+01:00 roadrunner kernel: [114551.313469] ? filemap_map_pages+0x19c/0x3a0 2020-01-29T18:25:02.796078+01:00 roadrunner kernel: [114551.313473] ext4_filemap_fault+0x2c/0x40 2020-01-29T18:25:02.796079+01:00 roadrunner kernel: [114551.313477] __do_fault+0x1f/0xe0 2020-01-29T18:25:02.796080+01:00 roadrunner kernel: [114551.313479] __handle_mm_fault+0xc7a/0x1210 2020-01-29T18:25:02.852138+01:00 roadrunner kernel: [114551.313482] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:02.852461+01:00 roadrunner kernel: [114551.313486] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:02.852465+01:00 roadrunner kernel: [114551.313489] do_page_fault+0x2a/0x70 2020-01-29T18:25:02.852466+01:00 roadrunner kernel: [114551.313494] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:02.852466+01:00 roadrunner kernel: [114551.313499] ? page_fault+0x2f/0x50 2020-01-29T18:25:02.852467+01:00 roadrunner kernel: [114551.313500] page_fault+0x45/0x50 2020-01-29T18:25:02.852467+01:00 roadrunner kernel: [114551.313503] RIP: afa0:0x3fc0a5f3e190 2020-01-29T18:25:02.852468+01:00 roadrunner kernel: [114551.313504] RSP: dcb40ee0:00007fffdcb40e00 EFLAGS: 7fffdcb40ea8 2020-01-29T18:25:02.887877+01:00 roadrunner kernel: [114566.829045] QThread: page allocation stalls for 28988ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:02.887878+01:00 roadrunner kernel: [114566.829054] QThread cpuset=/ mems_allowed=0 2020-01-29T18:25:02.887881+01:00 roadrunner kernel: [114566.829060] CPU: 2 PID: 10313 Comm: QThread Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:02.887882+01:00 roadrunner kernel: [114566.829061] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:02.887883+01:00 roadrunner kernel: [114566.829063] Call Trace: 2020-01-29T18:25:02.887884+01:00 roadrunner kernel: [114566.829076] dump_stack+0x5c/0x86 2020-01-29T18:25:02.887884+01:00 roadrunner kernel: [114566.829081] warn_alloc+0xe0/0x170 2020-01-29T18:25:02.887885+01:00 roadrunner kernel: [114566.829085] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:02.887886+01:00 roadrunner kernel: [114566.829089] ? __do_page_cache_readahead+0xe0/0x270 2020-01-29T18:25:02.887886+01:00 roadrunner kernel: [114566.829091] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:02.887887+01:00 roadrunner kernel: [114566.829096] alloc_pages_current+0x72/0x140 2020-01-29T18:25:02.887888+01:00 roadrunner kernel: [114566.829100] filemap_fault+0x319/0x6b0 2020-01-29T18:25:02.988206+01:00 roadrunner kernel: [114566.829103] ? filemap_map_pages+0x19c/0x3a0 2020-01-29T18:25:02.993088+01:00 roadrunner kernel: [114566.829107] ext4_filemap_fault+0x2c/0x40 2020-01-29T18:25:02.993107+01:00 roadrunner kernel: [114566.829110] __do_fault+0x1f/0xe0 2020-01-29T18:25:02.993109+01:00 roadrunner kernel: [114566.829113] __handle_mm_fault+0xc7a/0x1210 2020-01-29T18:25:02.993111+01:00 roadrunner kernel: [114566.829116] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:02.993112+01:00 roadrunner kernel: [114566.829120] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:02.993114+01:00 roadrunner kernel: [114566.829122] do_page_fault+0x2a/0x70 2020-01-29T18:25:02.993115+01:00 roadrunner kernel: [114566.829127] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.010085+01:00 roadrunner kernel: [114566.829131] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.010110+01:00 roadrunner kernel: [114566.829133] page_fault+0x45/0x50 2020-01-29T18:25:03.010112+01:00 roadrunner kernel: [114566.829136] RIP: 5ce8:0x55b2fb1bdd08 2020-01-29T18:25:03.010113+01:00 roadrunner kernel: [114566.829137] RSP: 8f085cf0:000055b2fb1bdcf0 EFLAGS: 7f208f085cf0 2020-01-29T18:25:03.010114+01:00 roadrunner kernel: [114566.829140] warn_alloc_show_mem: 1 callbacks suppressed 2020-01-29T18:25:03.010116+01:00 roadrunner kernel: [114566.829140] Mem-Info: 2020-01-29T18:25:03.027077+01:00 roadrunner kernel: [114566.829146] active_anon:1103789 inactive_anon:183908 isolated_anon:21750 2020-01-29T18:25:03.027101+01:00 roadrunner kernel: [114566.829146] active_file:273 inactive_file:2397 isolated_file:32 2020-01-29T18:25:03.027103+01:00 roadrunner kernel: [114566.829146] unevictable:2709 dirty:0 writeback:1273966 unstable:0 2020-01-29T18:25:03.027104+01:00 roadrunner kernel: [114566.829146] slab_reclaimable:16253 slab_unreclaimable:532119 2020-01-29T18:25:03.027105+01:00 roadrunner kernel: [114566.829146] mapped: 2482 shmem:5835 pagetables:47040 bounce:0 2020-01-29T18:25:03.027106+01:00 roadrunner kernel: [114566.829146] free: 50816 free_pcp:1054 free_cma:0 2020-01-29T18:25:03.027107+01:00 roadrunner kernel: [114566.829151] Node 0 active_anon:4415156kB inactive_anon:735632kB active_file:1092kB inactive_file: 9588kB unevictable:10836kB isolated(anon):87000kB isolated(file):128kB mapped: 9928kB dirty:0kB writeback:5095864kB shmem:23340kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 38912kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no 2020-01-29T18:25:03.035903+01:00 roadrunner kernel: [114566.829151] Node 0 DMA free:15904kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present: 15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable: 0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 2020-01-29T18:25:03.035925+01:00 roadrunner kernel: [114566.829157] lowmem_reserve[]: 0 2866 7729 7729 7729 2020-01-29T18:25:03.035927+01:00 roadrunner kernel: [114566.829161] Node 0 DMA32 free:81864kB min:62528kB low:68780kB high:75032kB active_anon:1884760kB inactive_anon:332240kB active_file:68kB inactive_file:1372kB unevictable:0kB writepending:2196100kB present:3047480kB managed:2953452kB mlocked:0kB slab_reclaimable:9628kB slab_unreclaimable:538368kB kernel_stack:3512kB pagetables:21336kB bounce:0kB free_pcp:1636kB local_pcp:300kB free_cma:0kB 2020-01-29T18:25:03.035929+01:00 roadrunner kernel: [114566.829166] lowmem_reserve[]: 0 0 4862 4862 4862 2020-01-29T18:25:03.036033+01:00 roadrunner kernel: [114566.829170] Node 0 Normal free:105496kB min:106080kB low:116688kB high:127296kB active_anon: 2530876kB inactive_anon:402796kB active_file:1024kB inactive_file:8216kB unevictable:10836kB writepending:2899108kB present:5111808kB managed:4979396kB mlocked:10836kB slab_reclaimable:55384kB slab_unreclaimable:1590108kB kernel_stack:23704kB pagetables:166824kB bounce:0kB free_pcp:2580kB local_pcp: 864kB free_cma:0kB 2020-01-29T18:25:03.066457+01:00 roadrunner kernel: [114566.829175] lowmem_reserve[]: 0 0 0 0 0 2020-01-29T18:25:03.066467+01:00 roadrunner kernel: [114566.829179] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB 2020-01-29T18:25:03.066469+01:00 roadrunner kernel: [114566.829191] Node 0 DMA32: 14611*4kB (UME) 2997*8kB (UM) 4*16kB (UM) 3*32kB (UM) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 82644kB 2020-01-29T18:25:03.066471+01:00 roadrunner kernel: [114566.829202] Node 0 Normal: 20889*4kB (UME) 2734*8kB (UM) 2*16kB (M) 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 105492kB 2020-01-29T18:25:03.066472+01:00 roadrunner kernel: [114566.829214] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 2020-01-29T18:25:03.066473+01:00 roadrunner kernel: [114566.829215] 1284823 total pagecache pages 2020-01-29T18:25:03.066475+01:00 roadrunner kernel: [114566.829218] 1274064 pages in swap cache 2020-01-29T18:25:03.066476+01:00 roadrunner kernel: [114566.829219] Swap cache stats: add 2270549, delete 1005333, find 164145/213061 2020-01-29T18:25:03.066478+01:00 roadrunner kernel: [114566.829220] Free swap = 76668kB 2020-01-29T18:25:03.066479+01:00 roadrunner kernel: [114566.829220] Total swap = 7950332kB 2020-01-29T18:25:03.066480+01:00 roadrunner kernel: [114566.829221] 2043819 pages RAM 2020-01-29T18:25:03.073403+01:00 roadrunner kernel: [114566.829222] 0 pages HighMem/MovableOnly 2020-01-29T18:25:03.073463+01:00 roadrunner kernel: [114566.829222] 56631 pages reserved 2020-01-29T18:25:03.073466+01:00 roadrunner kernel: [114566.829223] 0 pages hwpoisoned 2020-01-29T18:25:03.073479+01:00 roadrunner kernel: [114569.012784] akonadi_maildir: page allocation stalls for 20252ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.073480+01:00 roadrunner kernel: [114569.012792] akonadi_maildir cpuset=/ mems_allowed=0 2020-01-29T18:25:03.073481+01:00 roadrunner kernel: [114569.012798] CPU: 1 PID: 4361 Comm: akonadi_maildir Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.073483+01:00 roadrunner kernel: [114569.012800] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.073485+01:00 roadrunner kernel: [114569.012801] Call Trace: 2020-01-29T18:25:03.073486+01:00 roadrunner kernel: [114569.012813] dump_stack+0x5c/0x86 2020-01-29T18:25:03.073487+01:00 roadrunner kernel: [114569.012818] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.073488+01:00 roadrunner kernel: [114569.012822] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.073489+01:00 roadrunner kernel: [114569.012826] ? free_pcppages_bulk+0x31a/0x5e0 2020-01-29T18:25:03.073491+01:00 roadrunner kernel: [114569.012828] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.073492+01:00 roadrunner kernel: [114569.012833] alloc_pages_vma+0x8f/0x2a0 2020-01-29T18:25:03.073493+01:00 roadrunner kernel: [114569.012838] __read_swap_cache_async+0x146/0x200 2020-01-29T18:25:03.073495+01:00 roadrunner kernel: [114569.012840] read_swap_cache_async+0x14/0x30 2020-01-29T18:25:03.073496+01:00 roadrunner kernel: [114569.012842] swapin_readahead+0x115/0x1f0 2020-01-29T18:25:03.073497+01:00 roadrunner kernel: [114569.012846] ? do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.073499+01:00 roadrunner kernel: [114569.012848] ? lookup_swap_cache+0x3f/0x70 2020-01-29T18:25:03.073500+01:00 roadrunner kernel: [114569.012850] do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.073501+01:00 roadrunner kernel: [114569.012852] __handle_mm_fault+0x784/0x1210 2020-01-29T18:25:03.073502+01:00 roadrunner kernel: [114569.012857] ? wake_up_q+0x70/0x70 2020-01-29T18:25:03.073504+01:00 roadrunner kernel: [114569.012859] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.073505+01:00 roadrunner kernel: [114569.012863] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.073506+01:00 roadrunner kernel: [114569.012865] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.073507+01:00 roadrunner kernel: [114569.012870] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.073509+01:00 roadrunner kernel: [114569.012875] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.073510+01:00 roadrunner kernel: [114569.012876] page_fault+0x45/0x50 2020-01-29T18:25:03.073512+01:00 roadrunner kernel: [114569.012879] RIP: 0003:0x7fe842d6a930 2020-01-29T18:25:03.073513+01:00 roadrunner kernel: [114569.012880] RSP: 34004fc0:000000007fffffff EFLAGS: 55d1925f0930 2020-01-29T18:25:03.073514+01:00 roadrunner kernel: [114569.012882] Mem-Info: 2020-01-29T18:25:03.073516+01:00 roadrunner kernel: [114569.012888] active_anon:1103653 inactive_anon:183959 isolated_anon:15496 2020-01-29T18:25:03.073517+01:00 roadrunner kernel: [114569.012888] active_file:332 inactive_file:2475 isolated_file:0 2020-01-29T18:25:03.073518+01:00 roadrunner kernel: [114569.012888] unevictable:2709 dirty:0 writeback:1286960 unstable:0 2020-01-29T18:25:03.073519+01:00 roadrunner kernel: [114569.012888] slab_reclaimable:16254 slab_unreclaimable:537433 2020-01-29T18:25:03.073521+01:00 roadrunner kernel: [114569.012888] mapped: 2483 shmem:2236 pagetables:47040 bounce:0 2020-01-29T18:25:03.073522+01:00 roadrunner kernel: [114569.012888] free: 50908 free_pcp:1924 free_cma:0 2020-01-29T18:25:03.073524+01:00 roadrunner kernel: [114569.012893] Node 0 active_anon:4414612kB inactive_anon:735836kB active_file:1328kB inactive_file: 9900kB unevictable:10836kB isolated(anon):61984kB isolated(file):0kB mapped: 9932kB dirty:0kB writeback:5147840kB shmem:8944kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 16384kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no 2020-01-29T18:25:03.073525+01:00 roadrunner kernel: [114569.012894] Node 0 DMA free:15904kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present: 15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable: 0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 2020-01-29T18:25:03.073527+01:00 roadrunner kernel: [114569.012899] lowmem_reserve[]: 0 2866 7729 7729 7729 2020-01-29T18:25:03.073529+01:00 roadrunner kernel: [114569.012903] Node 0 DMA32 free:81924kB min:62528kB low:68780kB high:75032kB active_anon:1923372kB inactive_anon:307440kB active_file:72kB inactive_file:1368kB unevictable:0kB writepending:2226832kB present:3047480kB managed:2953452kB mlocked:0kB slab_reclaimable:9608kB slab_unreclaimable:545612kB kernel_stack:3512kB pagetables:21336kB bounce:0kB free_pcp:4576kB local_pcp:1444kB free_cma:0kB 2020-01-29T18:25:03.073530+01:00 roadrunner kernel: [114569.012909] lowmem_reserve[]: 0 0 4862 4862 4862 2020-01-29T18:25:03.073532+01:00 roadrunner kernel: [114569.012912] Node 0 Normal free:105804kB min:106080kB low:116688kB high:127296kB active_anon: 2489808kB inactive_anon:428364kB active_file:1256kB inactive_file:8532kB unevictable:10836kB writepending:2921184kB present:5111808kB managed:4979396kB mlocked:10836kB slab_reclaimable:55408kB slab_unreclaimable:1604120kB kernel_stack:23704kB pagetables:166824kB bounce:0kB free_pcp:3120kB local_pcp: 768kB free_cma:0kB 2020-01-29T18:25:03.073533+01:00 roadrunner kernel: [114569.012918] lowmem_reserve[]: 0 0 0 0 0 2020-01-29T18:25:03.073535+01:00 roadrunner kernel: [114569.012921] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB 2020-01-29T18:25:03.073536+01:00 roadrunner kernel: [114569.012933] Node 0 DMA32: 14584*4kB (ME) 2981*8kB (UME) 9*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 82328kB 2020-01-29T18:25:03.073537+01:00 roadrunner kernel: [114569.012943] Node 0 Normal: 21007*4kB (UME) 2790*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 106348kB 2020-01-29T18:25:03.073538+01:00 roadrunner kernel: [114569.012955] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 2020-01-29T18:25:03.073538+01:00 roadrunner kernel: [114569.012956] 1294159 total pagecache pages 2020-01-29T18:25:03.073539+01:00 roadrunner kernel: [114569.012958] 1287017 pages in swap cache 2020-01-29T18:25:03.073539+01:00 roadrunner kernel: [114569.012960] Swap cache stats: add 2289769, delete 1011600, find 164147/213063 2020-01-29T18:25:03.073540+01:00 roadrunner kernel: [114569.012961] Free swap = 0kB 2020-01-29T18:25:03.073541+01:00 roadrunner kernel: [114569.012961] Total swap = 7950332kB 2020-01-29T18:25:03.073542+01:00 roadrunner kernel: [114569.012962] 2043819 pages RAM 2020-01-29T18:25:03.073542+01:00 roadrunner kernel: [114569.012962] 0 pages HighMem/MovableOnly 2020-01-29T18:25:03.073543+01:00 roadrunner kernel: [114569.012963] 56631 pages reserved 2020-01-29T18:25:03.073544+01:00 roadrunner kernel: [114569.012964] 0 pages hwpoisoned 2020-01-29T18:25:03.073545+01:00 roadrunner kernel: [114569.027921] akonadiserver: page allocation stalls for 15564ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.073545+01:00 roadrunner kernel: [114569.027929] akonadiserver cpuset=/ mems_allowed=0 2020-01-29T18:25:03.073547+01:00 roadrunner kernel: [114569.027936] CPU: 0 PID: 4149 Comm: akonadiserver Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.073548+01:00 roadrunner kernel: [114569.027937] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.073550+01:00 roadrunner kernel: [114569.027938] Call Trace: 2020-01-29T18:25:03.073551+01:00 roadrunner kernel: [114569.027951] dump_stack+0x5c/0x86 2020-01-29T18:25:03.073552+01:00 roadrunner kernel: [114569.027956] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.073553+01:00 roadrunner kernel: [114569.027959] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.073688+01:00 roadrunner kernel: [114569.027964] ? __switch_to_asm+0x40/0x70 2020-01-29T18:25:03.073693+01:00 roadrunner kernel: [114569.027966] ? __switch_to_asm+0x34/0x70 2020-01-29T18:25:03.073694+01:00 roadrunner kernel: [114569.027968] ? __switch_to_asm+0x34/0x70 2020-01-29T18:25:03.080739+01:00 roadrunner kernel: [114569.027970] ? __switch_to_asm+0x40/0x70 2020-01-29T18:25:03.080761+01:00 roadrunner kernel: [114569.027971] ? __switch_to_asm+0x34/0x70 2020-01-29T18:25:03.080763+01:00 roadrunner kernel: [114569.027973] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.080763+01:00 roadrunner kernel: [114569.027978] alloc_pages_vma+0x8f/0x2a0 2020-01-29T18:25:03.080764+01:00 roadrunner kernel: [114569.027982] __read_swap_cache_async+0x146/0x200 2020-01-29T18:25:03.080764+01:00 roadrunner kernel: [114569.027985] read_swap_cache_async+0x14/0x30 2020-01-29T18:25:03.085615+01:00 roadrunner kernel: [114569.027987] swapin_readahead+0x115/0x1f0 2020-01-29T18:25:03.085632+01:00 roadrunner kernel: [114569.027991] ? do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.085633+01:00 roadrunner kernel: [114569.027993] ? lookup_swap_cache+0x3f/0x70 2020-01-29T18:25:03.085634+01:00 roadrunner kernel: [114569.027994] do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.085636+01:00 roadrunner kernel: [114569.027997] __handle_mm_fault+0x784/0x1210 2020-01-29T18:25:03.085637+01:00 roadrunner kernel: [114569.027999] ? __switch_to_asm+0x40/0x70 2020-01-29T18:25:03.085638+01:00 roadrunner kernel: [114569.028001] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.109218+01:00 roadrunner kernel: [114569.028005] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.109267+01:00 roadrunner kernel: [114569.028007] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.109270+01:00 roadrunner kernel: [114569.028012] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.109272+01:00 roadrunner kernel: [114569.028015] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.109273+01:00 roadrunner kernel: [114569.028016] page_fault+0x45/0x50 2020-01-29T18:25:03.109274+01:00 roadrunner kernel: [114569.028019] RIP: 4090:0x7ffe8c5a2970 2020-01-29T18:25:03.120184+01:00 roadrunner kernel: [114569.028021] RSP: 8c5a2ab0:00005625c5898930 EFLAGS: 5625c58f6b50 2020-01-29T18:25:03.120217+01:00 roadrunner kernel: [114569.352581] akonadi_maildir: page allocation stalls for 20592ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.120220+01:00 roadrunner kernel: [114569.352589] akonadi_maildir cpuset=/ mems_allowed=0 2020-01-29T18:25:03.120221+01:00 roadrunner kernel: [114569.352597] CPU: 1 PID: 4361 Comm: akonadi_maildir Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.120223+01:00 roadrunner kernel: [114569.352598] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.120224+01:00 roadrunner kernel: [114569.352599] Call Trace: 2020-01-29T18:25:03.126205+01:00 roadrunner kernel: [114569.352612] dump_stack+0x5c/0x86 2020-01-29T18:25:03.126486+01:00 roadrunner kernel: [114569.352618] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.126489+01:00 roadrunner kernel: [114569.352621] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.126490+01:00 roadrunner kernel: [114569.352625] ? free_pcppages_bulk+0x31a/0x5e0 2020-01-29T18:25:03.126491+01:00 roadrunner kernel: [114569.352627] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.126492+01:00 roadrunner kernel: [114569.352632] alloc_pages_vma+0x8f/0x2a0 2020-01-29T18:25:03.126493+01:00 roadrunner kernel: [114569.352636] __read_swap_cache_async+0x146/0x200 2020-01-29T18:25:03.126495+01:00 roadrunner kernel: [114569.352639] read_swap_cache_async+0x14/0x30 2020-01-29T18:25:03.126495+01:00 roadrunner kernel: [114569.352641] swapin_readahead+0x115/0x1f0 2020-01-29T18:25:03.126497+01:00 roadrunner kernel: [114569.352645] ? do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.126497+01:00 roadrunner kernel: [114569.352647] ? lookup_swap_cache+0x3f/0x70 2020-01-29T18:25:03.126499+01:00 roadrunner kernel: [114569.352648] do_swap_page+0x284/0x8e0 2020-01-29T18:25:03.126500+01:00 roadrunner kernel: [114569.352650] __handle_mm_fault+0x784/0x1210 2020-01-29T18:25:03.126501+01:00 roadrunner kernel: [114569.352655] ? wake_up_q+0x70/0x70 2020-01-29T18:25:03.126502+01:00 roadrunner kernel: [114569.352657] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.126503+01:00 roadrunner kernel: [114569.352660] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.126504+01:00 roadrunner kernel: [114569.352663] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.126505+01:00 roadrunner kernel: [114569.352668] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.126506+01:00 roadrunner kernel: [114569.352672] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.126507+01:00 roadrunner kernel: [114569.352674] page_fault+0x45/0x50 2020-01-29T18:25:03.126508+01:00 roadrunner kernel: [114569.352677] RIP: 0003:0x7fe842d6a930 2020-01-29T18:25:03.126508+01:00 roadrunner kernel: [114569.352678] RSP: 34004fc0:000000007fffffff EFLAGS: 55d1925f0930 2020-01-29T18:25:03.126510+01:00 roadrunner kernel: [114569.723782] vivaldi- bin: page allocation stalls for 31856ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.126511+01:00 roadrunner kernel: [114569.723790] vivaldi- bin cpuset=/ mems_allowed=0 2020-01-29T18:25:03.126512+01:00 roadrunner kernel: [114569.723796] CPU: 0 PID: 27493 Comm: vivaldi-bin Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.126513+01:00 roadrunner kernel: [114569.723797] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.126515+01:00 roadrunner kernel: [114569.723798] Call Trace: 2020-01-29T18:25:03.126516+01:00 roadrunner kernel: [114569.723811] dump_stack+0x5c/0x86 2020-01-29T18:25:03.126517+01:00 roadrunner kernel: [114569.723816] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.126518+01:00 roadrunner kernel: [114569.723820] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.126519+01:00 roadrunner kernel: [114569.723824] ? __do_page_cache_readahead+0xe0/0x270 2020-01-29T18:25:03.126520+01:00 roadrunner kernel: [114569.723826] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.126521+01:00 roadrunner kernel: [114569.723831] alloc_pages_current+0x72/0x140 2020-01-29T18:25:03.126522+01:00 roadrunner kernel: [114569.723836] filemap_fault+0x319/0x6b0 2020-01-29T18:25:03.126523+01:00 roadrunner kernel: [114569.723839] ? filemap_map_pages+0x19c/0x3a0 2020-01-29T18:25:03.126524+01:00 roadrunner kernel: [114569.723842] ext4_filemap_fault+0x2c/0x40 2020-01-29T18:25:03.126526+01:00 roadrunner kernel: [114569.723846] __do_fault+0x1f/0xe0 2020-01-29T18:25:03.126527+01:00 roadrunner kernel: [114569.723849] __handle_mm_fault+0xc7a/0x1210 2020-01-29T18:25:03.126528+01:00 roadrunner kernel: [114569.723851] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.126529+01:00 roadrunner kernel: [114569.723855] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.126530+01:00 roadrunner kernel: [114569.723857] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.126531+01:00 roadrunner kernel: [114569.723862] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.126532+01:00 roadrunner kernel: [114569.723867] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.126533+01:00 roadrunner kernel: [114569.723868] page_fault+0x45/0x50 2020-01-29T18:25:03.126534+01:00 roadrunner kernel: [114569.723871] RIP: afa0:0x3fc0a5f3e190 2020-01-29T18:25:03.137319+01:00 roadrunner kernel: [114569.723872] RSP: dcb40ee0:00007fffdcb40e00 EFLAGS: 7fffdcb40ea8 2020-01-29T18:25:03.138241+01:00 roadrunner kernel: [114569.984096] vivaldi- bin: page allocation stalls for 32116ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.138248+01:00 roadrunner kernel: [114569.984105] vivaldi- bin cpuset=/ mems_allowed=0 2020-01-29T18:25:03.138249+01:00 roadrunner kernel: [114569.984112] CPU: 1 PID: 27493 Comm: vivaldi-bin Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.138251+01:00 roadrunner kernel: [114569.984113] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.138252+01:00 roadrunner kernel: [114569.984114] Call Trace: 2020-01-29T18:25:03.138253+01:00 roadrunner kernel: [114569.984126] dump_stack+0x5c/0x86 2020-01-29T18:25:03.138254+01:00 roadrunner kernel: [114569.984132] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.138255+01:00 roadrunner kernel: [114569.984135] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.138257+01:00 roadrunner kernel: [114569.984140] ? __do_page_cache_readahead+0xe0/0x270 2020-01-29T18:25:03.138258+01:00 roadrunner kernel: [114569.984142] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.138260+01:00 roadrunner kernel: [114569.984147] alloc_pages_current+0x72/0x140 2020-01-29T18:25:03.138261+01:00 roadrunner kernel: [114569.984151] filemap_fault+0x319/0x6b0 2020-01-29T18:25:03.138262+01:00 roadrunner kernel: [114569.984155] ? filemap_map_pages+0x19c/0x3a0 2020-01-29T18:25:03.138263+01:00 roadrunner kernel: [114569.984157] ext4_filemap_fault+0x2c/0x40 2020-01-29T18:25:03.138264+01:00 roadrunner kernel: [114569.984161] __do_fault+0x1f/0xe0 2020-01-29T18:25:03.138266+01:00 roadrunner kernel: [114569.984163] __handle_mm_fault+0xc7a/0x1210 2020-01-29T18:25:03.138267+01:00 roadrunner kernel: [114569.984166] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.138268+01:00 roadrunner kernel: [114569.984170] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.138270+01:00 roadrunner kernel: [114569.984173] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.138271+01:00 roadrunner kernel: [114569.984178] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.138272+01:00 roadrunner kernel: [114569.984182] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.138273+01:00 roadrunner kernel: [114569.984183] page_fault+0x45/0x50 2020-01-29T18:25:03.138274+01:00 roadrunner kernel: [114569.984186] RIP: afa0:0x3fc0a5f3e190 2020-01-29T18:25:03.138276+01:00 roadrunner kernel: [114569.984187] RSP: dcb40ee0:00007fffdcb40e00 EFLAGS: 7fffdcb40ea8 2020-01-29T18:25:03.138277+01:00 roadrunner kernel: [114570.712588] QtWebEngineProc: page allocation stalls for 32688ms, order:0, mode: 0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null) 2020-01-29T18:25:03.138278+01:00 roadrunner kernel: [114570.712595] QtWebEngineProc cpuset=/ mems_allowed=0 2020-01-29T18:25:03.138280+01:00 roadrunner kernel: [114570.712603] CPU: 0 PID: 32216 Comm: QtWebEngineProc Not tainted 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 2020-01-29T18:25:03.138281+01:00 roadrunner kernel: [114570.712604] Hardware name: LENOVO 3680W1J/3680W1J, BIOS 6QET70WW (1.40 ) 10/11/2012 2020-01-29T18:25:03.138282+01:00 roadrunner kernel: [114570.712605] Call Trace: 2020-01-29T18:25:03.138283+01:00 roadrunner kernel: [114570.712618] dump_stack+0x5c/0x86 2020-01-29T18:25:03.138284+01:00 roadrunner kernel: [114570.712623] warn_alloc+0xe0/0x170 2020-01-29T18:25:03.138285+01:00 roadrunner kernel: [114570.712627] __alloc_pages_slowpath+0x7e2/0xc10 2020-01-29T18:25:03.138287+01:00 roadrunner kernel: [114570.712632] ? __do_page_cache_readahead+0xe0/0x270 2020-01-29T18:25:03.138294+01:00 roadrunner kernel: [114570.712635] __alloc_pages_nodemask+0x246/0x260 2020-01-29T18:25:03.138296+01:00 roadrunner kernel: [114570.712640] alloc_pages_current+0x72/0x140 2020-01-29T18:25:03.138297+01:00 roadrunner kernel: [114570.712645] filemap_fault+0x319/0x6b0 2020-01-29T18:25:03.138298+01:00 roadrunner kernel: [114570.712649] ? filemap_map_pages+0x382/0x3a0 2020-01-29T18:25:03.138299+01:00 roadrunner kernel: [114570.712652] ext4_filemap_fault+0x2c/0x40 2020-01-29T18:25:03.138301+01:00 roadrunner kernel: [114570.712655] __do_fault+0x1f/0xe0 2020-01-29T18:25:03.138302+01:00 roadrunner kernel: [114570.712658] __handle_mm_fault+0xc7a/0x1210 2020-01-29T18:25:03.138303+01:00 roadrunner kernel: [114570.712662] handle_mm_fault+0xa6/0x1d0 2020-01-29T18:25:03.138304+01:00 roadrunner kernel: [114570.712666] __do_page_fault+0x233/0x4c0 2020-01-29T18:25:03.138305+01:00 roadrunner kernel: [114570.712668] do_page_fault+0x2a/0x70 2020-01-29T18:25:03.138306+01:00 roadrunner kernel: [114570.712674] ? do_syscall_64+0x7b/0x160 2020-01-29T18:25:03.138307+01:00 roadrunner kernel: [114570.712678] ? page_fault+0x2f/0x50 2020-01-29T18:25:03.138308+01:00 roadrunner kernel: [114570.712680] page_fault+0x45/0x50 2020-01-29T18:25:03.138310+01:00 roadrunner kernel: [114570.712683] RIP: 5a80:0x7fff8191bdf8 2020-01-29T18:25:03.138311+01:00 roadrunner kernel: [114570.712684] RSP: aae65a78:00007fff8191bcd8 EFLAGS: 1aaae51890 2020-01-29T18:25:03.138312+01:00 roadrunner kernel: [114570.712687] warn_alloc_show_mem: 4 callbacks suppressed 2020-01-29T18:25:03.138313+01:00 roadrunner kernel: [114570.712688] Mem-Info: 2020-01-29T18:25:03.138314+01:00 roadrunner kernel: [114570.712695] active_anon:1091205 inactive_anon:181882 isolated_anon:2880 2020-01-29T18:25:03.138316+01:00 roadrunner kernel: [114570.712695] active_file:62 inactive_file:2784 isolated_file:0 2020-01-29T18:25:03.138317+01:00 roadrunner kernel: [114570.712695] unevictable:2709 dirty:0 writeback:1259726 unstable:0 2020-01-29T18:25:03.138318+01:00 roadrunner kernel: [114570.712695] slab_reclaimable:16250 slab_unreclaimable:529718 2020-01-29T18:25:03.138319+01:00 roadrunner kernel: [114570.712695] mapped: 2369 shmem:2106 pagetables:47040 bounce:0 Mentiones akonadi, vivaldi, it is a bit of who is who of what is running on the system. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/01/2020 12.13, stakanov wrote: | In data giovedì 30 gennaio 2020 12:05:42 CET, stakanov ha scritto: |> In data giovedì 30 gennaio 2020 11:54:30 CET, Carlos E. R. ha |> scritto: |>> On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote: |>>> I did create a paste in |>>> |>>> https://paste.opensuse.org/22334414 |>> |>> Just a comment: I think it is better to obtain this from the |>> /var/log/messages file instead of dmesg output. It has |>> readable timestamps, for starters, and more context. | | | | Just an excerpt of the endless(!) /var/log/messages: | | | | local_pcp:1496kB free_cma:0kB 2020-01-29T18:25:02.652312+01:00 | roadrunner kernel: [114550.815012] lowmem_reserve[]: 0 0 0 0 0 | 2020-01-29T18:25:02.652318+01:00 roadrunner kernel: [114550.815016] | Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) | 1*256kB (U) 0*512kB ... | 2020-01-29T18:25:03.138319+01:00 roadrunner kernel: [114570.712695] | mapped: 2369 shmem:2106 pagetables:47040 bounce:0 | | Mentiones akonadi, vivaldi, it is a bit of who is who of what is | running on the system. I think you should report in Bugzilla. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjLJFAAKCRC1MxgcbY1H 1bquAJ9ZTeFuQC/ALkZumWa5Bjoic9tQ1wCbBOAh9doG5uu/T6XKkoGiiijUMRs= =PpF2 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 7:16 a.m., Carlos E. R. wrote:
On 30/01/2020 12.13, stakanov wrote: | In data giovedì 30 gennaio 2020 12:05:42 CET, stakanov ha scritto: |> In data giovedì 30 gennaio 2020 11:54:30 CET, Carlos E. R. ha |> scritto: |>> On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote: |>>> I did create a paste in |>>> |>>> https://paste.opensuse.org/22334414 |>> |>> Just a comment: I think it is better to obtain this from the |>> /var/log/messages file instead of dmesg output. It has |>> readable timestamps, for starters, and more context. | | | | Just an excerpt of the endless(!) /var/log/messages: | | | | local_pcp:1496kB free_cma:0kB 2020-01-29T18:25:02.652312+01:00 | roadrunner kernel: [114550.815012] lowmem_reserve[]: 0 0 0 0 0 | 2020-01-29T18:25:02.652318+01:00 roadrunner kernel: [114550.815016] | Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) | 1*256kB (U) 0*512kB ... | 2020-01-29T18:25:03.138319+01:00 roadrunner kernel: [114570.712695] | mapped: 2369 shmem:2106 pagetables:47040 bounce:0 | | Mentiones akonadi, vivaldi, it is a bit of who is who of what is | running on the system.
I think you should report in Bugzilla.
Yes. The VM system relies on a known, allocated amount of memory in low address space to act as the page mapping tables. certain types of what are best described as 'fragmentation' can cause that to overflow. Of course, it is a configurable parameter :-) If you know ahead of time that your application spread is going to place that sort of demand on memory (and you know about VM algorithms and all about this, which to be honest I need more, MUCH MORE coffee in my system to be able to advise on) (and which the kernel docco doesn't discuss! [and yes, I've checked]) and know how to tune the parameters, then, well, you can tune the parameters. Of course if this were the Real World(tm) you would be paying for support from SUSE (or possibly, being the real World(tm) you would actually be using Redhat and using IBM support) and could call and expect professional advise from someone who knows more about the VM system than the likes of Thee or Mee. Meanwhile, after picking your self up from the floor and calming down your giggles, you might tell us about the application set and environment, because, as your know, and that I keep say (but sometimes I think to no avail) Context is Everything
-- Cheers / Saludos,
Carlos E. R. (from 15.1 x86_64 at Telcontar)
-- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 6:05 a.m., stakanov wrote:
Unevictable, unreclamable, unresponsive........seems to be more a grave than a swap. Complains about a qt problem but I do not understand which kind of. While the system swaps so heavily the memory is free, not even a problem of cached memory. The system has 8GB and when this happens it has up to 25% of unallocated(!) memory, completely free. So this is definitely not normal. If I force back all to ram I have still 15 to 20% of memory free and unallocated.
All the evidence is that the swap is a ratchet mechanism. Once something is swapped out it being swapped back in makes no difference to the amount of swap allocated. Or maybe it just works in terms a high water mark. That you CAN do a swapoff and the system works fine after with zero in swap is very significant. Sometimes it seems Linux does 'pre-emptive' swapping', making guesses, rightly or wrongly, about what might not be needed in the near future. The more you jack up 'swapiness' the more likely that is. I understand the reasoning behind pre-emptive swapping but disagree with the wisdom. Once again I point out that code does not get swapped out, only volatile data. Or that's how it should be. I strongly suggest running 'vmstat -SM -a 5' in a terminal window. Oh, and read the man page as well :-) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 4:54 a.m., Carlos E. R. wrote:
On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
dmesg -T -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Carlos E. R. <robin.listas@telefonica.net> [01-30-20 05:57]:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
look at "dmesg -T" -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data giovedì 30 gennaio 2020 16:50:57 CET, Patrick Shanahan ha scritto:
* Carlos E. R. <robin.listas@telefonica.net> [01-30-20 05:57]:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2020-01-30 at 11:18 +0100, stakanov wrote:
I did create a paste in
Just a comment: I think it is better to obtain this from the /var/log/messages file instead of dmesg output. It has readable timestamps, for starters, and more context.
look at "dmesg -T" Unfortunately (always for those masses of reject messages from the vpn) it rolled over and created an xz compressed file (messages) I think. But I will hold that in mind, did not know about it thank you.
_________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
stakanov composed on 2020-01-30 11:18 (UTC+0100):
I did create a paste in
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead.
Since the first crash appears shortly after mysqld: page allocation stalls for 21920ms, mysqld cpuset=/ mems_allowed=0 CPU: 0 PID: 4130 Comm: mysqld Not tainted I would try disabling mysqld to see if the crashing stops and swap stays unused. -- Evolution as taught in public schools is religion, not science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
stakanov composed on 2020-01-30 11:18 (UTC+0100):
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead.
Since the first crash appears shortly after mysqld: page allocation stalls for 21920ms, mysqld cpuset=/ mems_allowed=0 CPU: 0 PID: 4130 Comm: mysqld Not tainted I would try disabling mysqld to see if the crashing stops and swap stays unused.
In data giovedì 30 gennaio 2020 12:42:26 CET, Felix Miata ha scritto: this is a bit difficult since akonadi is used on two users for mail. But I can try, although that would mean, to have a typical example, to leave the machine running over night, with akonadi shut down I guess (or do I have to end msql actively) to see if it swaps at all. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jan 30, 2020 at 6:53 PM stakanov <stakanov@eclipso.eu> wrote:
stakanov composed on 2020-01-30 11:18 (UTC+0100):
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead.
Since the first crash appears shortly after mysqld: page allocation stalls for 21920ms, mysqld cpuset=/ mems_allowed=0 CPU: 0 PID: 4130 Comm: mysqld Not tainted I would try disabling mysqld to see if the crashing stops and swap stays unused.
In data giovedì 30 gennaio 2020 12:42:26 CET, Felix Miata ha scritto: this is a bit difficult since akonadi is used on two users for mail.
But I can try, although that would mean, to have a typical example, to leave the machine running over night, with akonadi shut down I guess (or do I have to end msql actively) to see if it swaps at all.
Hi, As someone who get in touch with database daily (mainly PostgreSQL), seems to me there is something wrong with MySQL. Do you check your physical RAM? Try memtester or memtest86+ (https://www.memtest.org/). Anyway I try to avoid disk swap on my database, and push it to use hugepages and TLB extensively. I also use zram and change the unused RAM become swap. best, -- Edwin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 11:21 p.m., medwinz wrote:
Anyway I try to avoid disk swap on my database, and push it to use hugepages and TLB extensively. I also use zram and change the unused RAM become swap.
Could you detail each of those two matters, please. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 11:44 AM Anton Aylward <opensuse@antonaylward.com> wrote:
On 2020-01-30 11:21 p.m., medwinz wrote:
Anyway I try to avoid disk swap on my database, and push it to use hugepages and TLB extensively. I also use zram and change the unused RAM become swap.
Could you detail each of those two matters, please.
Hi, There are many sources on the internet, postgresql documentation mentions it on https://www.postgresql.org/docs/12/kernel-resources.html#LINUX-HUGE-PAGES I made an article in indonesia openSUSE community blog (but it is in Indonesian language) https://opensuse.id/2020/01/04/tuning-konfigurasi-postgresql/ For zram it is very straight forward just install it from official repo, configure it, and start the service. best, -- Edwin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 11:55 AM medwinz <medwinz@gmail.com> wrote:
On Fri, Jan 31, 2020 at 11:44 AM Anton Aylward <opensuse@antonaylward.com> wrote:
....
Could you detail each of those two matters, please.
.... For zram it is very straight forward just install it from official repo, configure it, and start the service.
Hi, My mistake and can be mislead. It is not from official repo it is from obs repo filesystem. The package name is systemd-zram-service. It will install some packages, which are /usr/sbin/zramswapon, /usr/sbin/zramswapoff and also a systemd service file zramswap.service. Just start and enable that service. best, -- Edwin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 11:55 AM medwinz <medwinz@gmail.com> wrote:
On Fri, Jan 31, 2020 at 11:44 AM Anton Aylward
<opensuse@antonaylward.com> wrote: ....
Could you detail each of those two matters, please.
....
For zram it is very straight forward just install it from official repo, configure it, and start the service.
Hi,
My mistake and can be mislead. It is not from official repo it is from obs repo filesystem. The package name is systemd-zram-service. It will install some packages, which are /usr/sbin/zramswapon, /usr/sbin/zramswapoff and also a systemd service file zramswap.service. Just start and enable that service.
best, -- Edwin I thought honestly that mysql has hugepages enabled by default but I can be wrong. How could I quickcheck if it has? I do not recall the command any more, I
In data venerdì 31 gennaio 2020 08:08:06 CET, medwinz ha scritto: think I checked for it, when I first run into this, because time ago I already had this doubt. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 10:28 AM stakanov <stakanov@eclipso.eu> wrote:
I thought honestly that mysql has hugepages enabled by default but I can be wrong. How could I quickcheck if it has?
According to your dmesg output no hugepage is allocated. Node 0 hugepages_total=0 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 11:55 p.m., medwinz wrote:
On Fri, Jan 31, 2020 at 11:44 AM Anton Aylward <opensuse@antonaylward.com> wrote:
On 2020-01-30 11:21 p.m., medwinz wrote:
Anyway I try to avoid disk swap on my database, and push it to use hugepages and TLB extensively. I also use zram and change the unused RAM become swap.
Could you detail each of those two matters, please.
Hi,
There are many sources on the internet, postgresql documentation mentions it on https://www.postgresql.org/docs/12/kernel-resources.html#LINUX-HUGE-PAGES I made an article in indonesia openSUSE community blog (but it is in Indonesian language) https://opensuse.id/2020/01/04/tuning-konfigurasi-postgresql/ For zram it is very straight forward just install it from official repo, configure it, and start the service.
Yes, but I'm interested, I suspect others are interested too, in what YOU specifically did, what size BIGPAGES you used, how you set that up, what libraries or other binaries you assigned to such pages, and how YOU felt it affected the performance of YOUR system. Please don't cc me, just reply to the list. I don't need two copies of your reply :-) -- "Now look," Forrester said patiently, "progress is an outmoded idea. We've got to be in step with the times. We've got to ask ourselves what progress ever did for us." -- Randall Garett, "Pagan Passions", 1959 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains.
It looks like an out-of-memory condition, and your system has used up all the swap space. Something is gobbleing up memory. -- Per Jessen, Zürich (10.4°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data giovedì 30 gennaio 2020 13:16:38 CET, Per Jessen ha scritto:
stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains.
It looks like an out-of-memory condition, and your system has used up all the swap space. Something is gobbleing up memory.
I am not keen no molesting people. If you vote in favor like Carlos however, that this grants a bug report, I will do that. Just want to avoid to open something that is visibly hardware related. So that is why I posted here instead. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
stakanov wrote:
In data giovedì 30 gennaio 2020 13:16:38 CET, Per Jessen ha scritto:
stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains.
It looks like an out-of-memory condition, and your system has used up all the swap space. Something is gobbleing up memory.
I am not keen no molesting people. If you vote in favor like Carlos however, that this grants a bug report, I will do that. Just want to avoid to open something that is visibly hardware related. So that is why I posted here instead.
I don't see any hints that this could be hardware related. Your system is unresponsive, that suggests it is busy otherwise, which could likely be swapping. Try the simple thing first - "top", then shift-M, to see which processes are using up all your memory. In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot? It looks like your system has 8Gb ? -- Per Jessen, Zürich (10.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> wrote:
In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot?
It's in pages, so multiply by 4K. Active anonymous memory alone is around 6GB. This is known problem. There is a lot of active memory so every time kernel has to search a lot to find free page (or page that can be reclaimed).
It looks like your system has 8Gb ?
yes. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> wrote:
In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot?
It's in pages, so multiply by 4K. Active anonymous memory alone is around 6GB.
Ah, thanks - I was thinking it was kilobytes.
This is known problem. There is a lot of active memory so every time kernel has to search a lot to find free page (or page that can be reclaimed).
that could lead to an oom condition? -- Per Jessen, Zürich (8.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
It's a HARDWARE problem. Sort of. Or a limitation of the software design imposed by a hardware limitation. On 2020-01-30 10:53 a.m., Per Jessen wrote:
Andrei Borzenkov wrote:
On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> wrote:
In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot?
It's in pages, so multiply by 4K. Active anonymous memory alone is around 6GB.
Ah, thanks - I was thinking it was kilobytes.
This is known problem. There is a lot of active memory so every time kernel has to search a lot to find free page (or page that can be reclaimed).
that could lead to an oom condition?
You asked earlier about 'fragmentation. it SHOULD NOT happen with a VM system but there IS a syndromic situation where it COULD happen and this seems to be it. Some applications need HUGEPAGES, that is block of pages that correspond to large areas of physical memory. It's a hardware problem! Why is it a hardware problem? Well think of DMA. As long as DMA is doing the 4K buffered IO it can do that behind the scenes of the VM system. The VM-IO can tell the DMA the 'real address' and that's what the DMA uses. The DMA does not use the VM mapping because the hardware doesn't work that way. <sidebar> Not all hardware worked that way. To the best of my knowledge, the old DEC VAX had some way round this but I don't recall what it was[1]. It had other problems though that involved a not-very-reliable method of tagging the page descriptors. Kludges about to make up for hardware shortcomings. </sidebar> But some application such as some database apps are more 'raw' and need huge spans of address space to IO into (or out of). If the DMA was using the same mapping tables then everything would be OK, but its not, it's 'behind the scenes'. So the VM system has to deal with HUGEPAGES, that is a block of pages that corresponds to a large block of physical memory. And it has to create those. Now the regular VM operation might well have broken up memory so there isn't such a span. It will go into a frenzy of deallocation and swapping stuff out to try to create such a span, perhaps more than one. The deallocation might cause anomalies in other execution as code pages vanish. meanwhile, the application has requested this and is waiting, waiting ... .... mysqld: page allocation stalls for 21920ms, .... The irony is that there might be a great amount 'free' (for some value of that meaning) memory. The VM system isn't good at, because that's not what it was designed for, juggling that around to create the large available spans. A human would be good at that, but this is a general purpose computer, and it's just gone beyond the more general use-case it was intended for. We can do the Times Crossword puzzle. (OK, some of us can. other's weren't designed for that.) Can you 'tune' for this? Probably. I don't know. I think I'm coming down with the 'flu, so please don't ask me to find out. Try reading the kernel VM docco yourself. I'm off to get some hot lemon flu cure. [1] I could look it up but I'm not going to. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jan 30, 2020 at 6:53 PM Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> wrote:
In the process lists from dmesg, I see 266 processes using up about 1Gb (RSS) which doesn't seem like a lot?
It's in pages, so multiply by 4K. Active anonymous memory alone is around 6GB.
Ah, thanks - I was thinking it was kilobytes.
This is known problem. There is a lot of active memory so every time kernel has to search a lot to find free page (or page that can be reclaimed).
that could lead to an oom condition?
Rather this is indication of low memory. In this case free RAM is below absolute minimum that kernel attempts to keep: Node 0 Normal free:102036kB min:106080kB Which means on every allocation request kernel will synchronously attempt to free enough memory to go above watermark. It looks like it tries it actually: writeback:1251356kB writeback:1671120kB writeback:1876996kB writeback:1982080kB writeback:1961032kB writeback:1964848kB writeback:1914756kB writeback:1914500kB writeback:250628kB writeback:317124kB writeback:372888kB writeback:402920kB writeback:558956kB writeback:656724kB writeback:696416kB writeback:736480kB writeback:792932kB writeback:1045984kB writeback:1035232kB writeback:278136kB writeback:274188kB This started with almost 2GB of data that is being written to disk. At some point kernel managed to complete it just to get more and more new data to be written. If kernel needs to wait while dirty pages are written to disk it pretty much explains those "allocation stalled" warnings. This goes on until no more free swap is available (which probably means - kernel cannot free memory to go above watermark at all) at which point OOM kicks in. So it appears like some application(s) really use that much memory. IOW system is overloaded. As for "If you sudo swapoff -a the swap turns into memory, the system turns responsive" it is far too vague to make any useful comment. It is impossible to do "swapoff" under conditions shown in dmesg output. If this was supposed to mean "swapoff before this situation is reached" - sure, it will eliminate stalls due to writing data to disk, it will also cause OOM earlier. If RAM+swap is not enough for current workload, then RAM alone is obviously not enough as well. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 08:04:27 CET, Andrei Borzenkov ha scritto:
the system turns responsive" it is far too vague to make any useful comment. It
On this one: when you have the aforementioned condition (that is curiously triggered always when you use the touchpad (but that can be a causality thing because you will note consciously it does not respond any more, while if you do not use the touchpad probably you wont). But for the sake of it: what happens if it is unresponsive and what do I mean with responsive: Unresponsive: heavy disc activity, no response of the mouse pointer. No possibility to change between sessions with alt + cntrl + Fn. No applications do open. That is, you can (after a while (about 5 minutes) and with a 30 seconds delay) execute commands and open applications. But caution is needed because two commands in this time will make it much worse. If you do not kill at least one application that consumes a lot of cpu time, then you are done. The system will go down (shutdown for oom condition). You know that it happened again because it switched off during your absence while on sector. At the time that happened the last time, the application was used was firefox. But it does not mean it has to be firefox. The CPU time was 24%. After killing it (because application do respond too slow to avoid the complete freeze) the system turned normal. That is, cursor has normal speed, more applications could be opened, you can shift between sessions. However what you can't do: leave it like that. If you sudo swapoff -a and wait (takes about half an hour) to discharge the amount that is still in swap to the ram, then it is over, you might work for hours, after also sudo swapon -a. No new swap will accumulate. But if you do not this procedure, that starts over after a short time... and you are left with the same problem. Or, if you leave it, the best you can hope is for slow growth of swap. I fairly never saw the swap written back. If you tell me how to quantify unresponsiveness quantitatively then please tell me the command and next time (and no problem, there will be) I give you the values. Frequency: happens on a "regular irregular" basis. That is, it is repetitive, quite frequent (from several times per week to one time per week). When using suspend to disk and long uptimes then it is much more frequent, about after an uptime of 2 days, very frequent. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 10:22 AM stakanov <stakanov@eclipso.eu> wrote:
In data venerdì 31 gennaio 2020 08:04:27 CET, Andrei Borzenkov ha scritto:
the system turns responsive" it is far too vague to make any useful comment. It
On this one: when you have the aforementioned condition (that is curiously triggered always when you use the touchpad (but that can be a causality thing because you will note consciously it does not respond any more, while if you do not use the touchpad probably you wont). But for the sake of it: what happens if it is unresponsive and what do I mean with responsive:
Unresponsive: heavy disc activity, no response of the mouse pointer. No possibility to change between sessions with alt + cntrl + Fn. No applications do open. That is, you can (after a while (about 5 minutes) and with a 30 seconds delay) execute commands and open applications. But caution is needed because two commands in this time will make it much worse. If you do not kill at least one application that consumes a lot of cpu time, then you are done. The system will go down (shutdown for oom condition). You know that it happened again because it switched off during your absence while on sector.
OOM does not switch system off. According to your dmesg output OOM just killed one of processes related to web browsing. And system apparently was up and running after that for quite some time. And you also were able to collect dmesg output.
At the time that happened the last time, the application was used was firefox. But it does not mean it has to be firefox. The CPU time was 24%. After killing it (because application do respond too slow to avoid the complete freeze) the system turned normal. That is, cursor has normal speed, more applications could be opened, you can shift between sessions. However what you can't do: leave it like that. If you sudo swapoff -a and wait (takes about half an hour) to discharge the amount that is still in swap to the ram, then it is over, you might work for hours, after also sudo swapon -a.
Was that dmesg output you provided after "swapoff"?
No new swap will accumulate.
Without any more details the obvious guess - some rogue process that over-allocated memory was killed during this procedure. Process list before and after would certainly be useful. As well as logs after swapoff finished.
But if you do not this procedure, that starts over after a short time... and you are left with the same problem. Or, if you leave it, the best you can hope is for slow growth of swap. I fairly never saw the swap written back.
I am not sure what "swap written back" means. If you mean "swap is freed" - this happens when process memory is freed which almost always means, process is ended (normal high level memory allocators do not actually return memory back to system). Or when page content changes in RAM. So once some pages were swapped out, they almost certainly remain there - unless application reads them back. So it is entirely up to application. Swap is memory that application does not use currently but used at some point in the past. You can monitor swap in/out rate using vmstat or install sar (I think part of sysstat package).
If you tell me how to quantify unresponsiveness quantitatively then please tell me the command and next time (and no problem, there will be) I give you the values.
Frequency: happens on a "regular irregular" basis. That is, it is repetitive, quite frequent (from several times per week to one time per week). When using suspend to disk and long uptimes then it is much more frequent, about after an uptime of 2 days, very frequent.
In this case I would certainly install sar (and changed stats collection interval to be more frequent than default 15 minutes) to see how situation evolves. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 08:58:23 CET, hai scritto:
Was that dmesg output you provided after "swapoff"?
Oh, well, good question now. I think yes, but thinking is not knowing. As I have the output with timestamps from /var/log/messages, if there is a way (not that I am aware of) to see the command history of yakuake with time- stamp, I can tell you with certainty. For what is the /var/log/messages, the mailing list is difficult when it comes to such long files. I can however try to select all and paste it into a file in opensuse.paste. Would this be useful?
I am not sure what "swap written back" means. If you mean "swap is freed" yes, this is what I want to say. Swap is freed.
SAR: ok, it is sysstat. I am installing it now. Thank you for telling me about it. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 09:25:07 CET, stakanov ha scritto:
SAR:
sar -V sysstat versione 12.3.1 (C) Sebastien Godard (sysstat <at> orange.fr) Now what should I run exactly and can I run it as a automatic task once I start the system (cron?). Should be but I am worried about the size. As I have more space in /home than in root, it would probably a good idea to save the output in /home? Or is the output small. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Jan 31, 2020 at 11:31 AM stakanov <stakanov@eclipso.eu> wrote:
In data venerdì 31 gennaio 2020 09:25:07 CET, stakanov ha scritto:
SAR:
sar -V sysstat versione 12.3.1 (C) Sebastien Godard (sysstat <at> orange.fr)
Now what should I run exactly and can I run it as a automatic task once I start the system (cron?).
It should have installed cron job.
Should be but I am worried about the size. As I have more space in /home than in root, it would probably a good idea to save the output in /home? Or is the output small.
It goes into /var/log/sa by default, I suppose you can customize it, but size is not that large for default 15 minutes interval. Collecting stats more often will require more space. Just let it run for some time and check. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 31/01/2020 09.39, Andrei Borzenkov wrote:
On Fri, Jan 31, 2020 at 11:31 AM stakanov <stakanov@eclipso.eu> wrote:
In data venerdì 31 gennaio 2020 09:25:07 CET, stakanov ha scritto:
SAR:
sar -V sysstat versione 12.3.1 (C) Sebastien Godard (sysstat <at> orange.fr)
Now what should I run exactly and can I run it as a automatic task
once I
start the system (cron?).
It should have installed cron job.
I have sysstat installed since long ago, but no output to /var/log/sa Telcontar: # rpm -ql sysstat | grep cron /etc/sysstat/sysstat.cron I guess that file has to be copied or linked to crontab directory. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On Fri, Jan 31, 2020 at 2:24 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
On 31/01/2020 09.39, Andrei Borzenkov wrote:
On Fri, Jan 31, 2020 at 11:31 AM stakanov <stakanov@eclipso.eu> wrote:
In data venerdì 31 gennaio 2020 09:25:07 CET, stakanov ha scritto:
SAR:
sar -V sysstat versione 12.3.1 (C) Sebastien Godard (sysstat <at> orange.fr)
Now what should I run exactly and can I run it as a automatic task
once I
start the system (cron?).
It should have installed cron job.
I have sysstat installed since long ago, but no output to /var/log/sa
Telcontar: # rpm -ql sysstat | grep cron /etc/sysstat/sysstat.cron
I guess that file has to be copied or linked to crontab directory.
Looks like it has been converted to systemd, so you need to enable sysstat.service which itself enabled cron job. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/01/2020 12.36, Andrei Borzenkov wrote:
On Fri, Jan 31, 2020 at 2:24 PM Carlos E. R. <> wrote:
On 31/01/2020 09.39, Andrei Borzenkov wrote:
On Fri, Jan 31, 2020 at 11:31 AM stakanov <> wrote:
I have sysstat installed since long ago, but no output to /var/log/sa
Telcontar: # rpm -ql sysstat | grep cron /etc/sysstat/sysstat.cron
I guess that file has to be copied or linked to crontab directory.
Looks like it has been converted to systemd, so you need to enable sysstat.service which itself enabled cron job.
Telcontar:~ # systemctl enable sysstat.service Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /usr/lib/systemd/system/sysstat.service. Telcontar:~ # systemctl status sysstat.service ● sysstat.service - Write information about system start to sysstat log Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: disabled) Active: inactive (dead) Telcontar:~ # Information about system start? I thought it analyzed during normal system work. I don't see new cron file installed. Telcontar:~ # ls /etc/cron.* | grep sys Telcontar:~ # Telcontar:~ # systemctl list-timers --all NEXT LEFT LAST PASSED UNIT ACTIVATES Fri 2020-01-31 14:00:00 CET 14min left Fri 2020-01-31 13:00:00 CET 45min ago snapper-timeline.timer snapper-timeline.service Sat 2020-02-01 00:00:00 CET 10h left Mon 2020-01-27 11:40:01 CET 4 days ago btrfs-balance.timer btrfs-balance.service Sat 2020-02-01 00:00:00 CET 10h left Wed 2020-01-01 13:09:12 CET 4 weeks 2 days ago btrfs-scrub.timer btrfs-scrub.service Sat 2020-02-01 00:00:00 CET 10h left Mon 2020-01-27 11:40:02 CET 4 days ago btrfs-trim.timer btrfs-trim.service Sat 2020-02-01 00:00:00 CET 10h left Fri 2020-01-31 00:00:01 CET 13h ago logrotate.timer logrotate.service Sat 2020-02-01 00:00:00 CET 10h left Fri 2020-01-31 00:00:01 CET 13h ago mandb.timer mandb.service Sat 2020-02-01 00:00:00 CET 10h left Fri 2020-01-31 00:00:01 CET 13h ago mlocate.timer mlocate.service Sat 2020-02-01 00:00:00 CET 10h left Fri 2020-01-31 00:00:01 CET 13h ago unbound-anchor.timer unbound-anchor.service Sat 2020-02-01 00:29:17 CET 10h left Fri 2020-01-31 01:13:59 CET 12h ago backup-sysconfig.timer backup-sysconfig.service Sat 2020-02-01 00:50:37 CET 11h left Fri 2020-01-31 01:20:00 CET 12h ago check-battery.timer check-battery.service Sat 2020-02-01 01:42:55 CET 11h left Fri 2020-01-31 01:16:30 CET 12h ago backup-rpmdb.timer backup-rpmdb.service Sat 2020-02-01 08:50:25 CET 19h left Fri 2020-01-31 08:50:25 CET 4h 55min ago snapper-cleanup.timer snapper-cleanup.service Sat 2020-02-01 08:55:16 CET 19h left Fri 2020-01-31 08:55:16 CET 4h 50min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Mon 2020-02-03 00:00:00 CET 2 days left Mon 2020-01-27 11:40:01 CET 4 days ago fstrim.timer fstrim.service n/a n/a n/a n/a mdadm-last-resort@md0.timer mdadm-last-resort@md0.service 15 timers listed. Telcontar:~ # sysstat is not there :-? - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjQiNAAKCRC1MxgcbY1H 1VdJAJ4sbIx0KN/I82TZpxutlcwqYe8z3wCfVNy+KWvSfRvhZ18sOryBBevHmYE= =LksN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am Freitag, 31. Januar 2020, 13:49:03 CET schrieb Carlos E. R.:
Telcontar:~ # systemctl enable sysstat.service Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /usr/lib/systemd/system/sysstat.service. Telcontar:~ # systemctl status sysstat.service ● sysstat.service - Write information about system start to sysstat log Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: disabled) Active: inactive (dead) Telcontar:~ #
Information about system start? I thought it analyzed during normal system work.
I don't see new cron file installed.
I have $ l /etc/cron.d/sysstat lrwxrwxrwx 1 root root 25 Jan 29 15:18 /etc/cron.d/sysstat -> /etc/sysstat/sysstat.cron Cheers, Pete -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 15:21:05 CET, Hans-Peter Jansen ha scritto:
Am Freitag, 31. Januar 2020, 13:49:03 CET schrieb Carlos E. R.:
Telcontar:~ # systemctl enable sysstat.service Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /usr/lib/systemd/system/sysstat.service. Telcontar:~ # systemctl status sysstat.service ● sysstat.service - Write information about system start to sysstat log
Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor
preset: disabled) Active: inactive (dead) Telcontar:~ #
Information about system start? I thought it analyzed during normal system work.
I don't see new cron file installed.
I have
$ l /etc/cron.d/sysstat lrwxrwxrwx 1 root root 25 Jan 29 15:18 /etc/cron.d/sysstat -> /etc/sysstat/sysstat.cron
Cheers, Pete
# l /etc/cron.d/sysstat ls: cannot access '/etc/cron.d/sysstat': No such file or directory Not with me in any case.... _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postf�cher sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/01/2020 17.26, stakanov wrote: | In data venerdì 31 gennaio 2020 15:21:05 CET, Hans-Peter Jansen ha | scritto: |> Am Freitag, 31. Januar 2020, 13:49:03 CET schrieb Carlos E. R.: |>> Telcontar:~ # systemctl enable sysstat.service Created symlink |>> /etc/systemd/system/multi-user.target.wants/sysstat.service → |>> /usr/lib/systemd/system/sysstat.service. Telcontar:~ # |>> systemctl status sysstat.service ● sysstat.service - Write |>> information about system start to sysstat log |>> |>> Loaded: loaded (/usr/lib/systemd/system/sysstat.service; |>> enabled; vendor |>> |>> preset: disabled) Active: inactive (dead) Telcontar:~ # |>> |>> |>> |>> Information about system start? I thought it analyzed during |>> normal system work. |>> |>> I don't see new cron file installed. |> |> I have |> |> $ l /etc/cron.d/sysstat lrwxrwxrwx 1 root root 25 Jan 29 15:18 |> /etc/cron.d/sysstat -> /etc/sysstat/sysstat.cron | | # l /etc/cron.d/sysstat ls: cannot access '/etc/cron.d/sysstat': | No such file or directory | | Not with me in any case.... I have just created it. But /usr/share/doc/packages/sysstat/README.md does say to start the service via systemd and that's it. I'm confused. But now indeed files appear in the log. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjRirgAKCRC1MxgcbY1H 1QT+AKCY2AqbfUU7w5Z7FjyUbB9ktYaXSgCeIpy4tsvxrqOba9xR7bC6Zxuezwo= =OA8m -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/01/2020 08.04, Andrei Borzenkov wrote: | On Thu, Jan 30, 2020 at 6:53 PM Per Jessen <per@computer.org> | wrote: |> |> Andrei Borzenkov wrote: |> |>> On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> |>> wrote: |>> |>>> In the process lists from dmesg, I see 266 processes using up |>>> about 1Gb (RSS) which doesn't seem like a lot? |>> |>> It's in pages, so multiply by 4K. Active anonymous memory alone |>> is around 6GB. |> |> Ah, thanks - I was thinking it was kilobytes. |> |>> This is known problem. There is a lot of active memory so every |>> time kernel has to search a lot to find free page (or page that |>> can be reclaimed). |> |> that could lead to an oom condition? |> | | Rather this is indication of low memory. | | In this case free RAM is below absolute minimum that kernel | attempts to keep: | | Node 0 Normal free:102036kB min:106080kB Ah! That's the start condition? I also have that in the log of my incident. <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921018] Node 0 Normal free:105704kB min:106040kB low:116644kB high:127248kB That's about 100 Mb, right? active_anon:1885752kB inactive_anon:626456kB active_file:371848kB inactive_file:1478016kB unevictable:10880kB writepending:239740kB present:5242880kB managed:5110480kB mlocked:10880kB slab_reclaimable:170100kB slab_unreclaimable:162136kB kernel_stack:21080kB pagetables:91024kB bounce:0kB free_pcp:2776kB local_pcp:1280kB free_cma:0kB Any document you know that explains these? And no knowing how the machine got there. | | Which means on every allocation request kernel will synchronously | attempt to free enough memory to go above watermark. It looks like | it tries it actually: | | writeback:1251356kB writeback:1671120kB writeback:1876996kB | writeback:1982080kB writeback:1961032kB I have one instance of that word. <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921004] Node 0 active_anon:2560096kB inactive_anon:1335884kB active_file:461648kB inactive_file:2786820kB unevictable:10880kB isolated(anon):0kB isolated(file):256kB mapped:495892kB dirty:23756kB writeback:391168kB shmem:212684kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 382976kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no | | This started with almost 2GB of data that is being written to | disk. How do you know that? :-? (I want to find it in my case) My machine was at the time probably copying a large directory from one disk to another, using 'mc', and nobody seated. | At some point kernel managed to complete it just to get more and | more new data to be written. If kernel needs to wait while dirty | pages are written to disk it pretty much explains those "allocation | stalled" warnings. | | This goes on until no more free swap is available (which probably | means - kernel cannot free memory to go above watermark at all) at | which point OOM kicks in. I don't think that happened in my case, but I do not know. I have 25 GB of swap. | So it appears like some application(s) really use that much | memory. IOW system is overloaded. | | As for "If you sudo swapoff -a the swap turns into memory, the | system turns responsive" it is far too vague to make any useful | comment. It is impossible to do "swapoff" under conditions shown in | dmesg output. If this was supposed to mean "swapoff before this | situation is reached" - sure, it will eliminate stalls due to | writing data to disk, it will also cause OOM earlier. If RAM+swap | is not enough for current workload, then RAM alone is obviously not | enough as well. This may be because read/write to swap with rotating disk is way too slow and involves many seeks. I observed that in my machine when I switched to Leap, and stopped when I bought an SSD disk and put the swap there. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjQQaQAKCRC1MxgcbY1H 1dlTAJwOC6qwNEkmm7lPvOKMtwh6z3B0+wCfQhWQ0MNjYns3IlD6MHrL1G+TvZM= =GZmz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 12:32:59 CET, Carlos E. R. ha scritto:
On 31/01/2020 08.04, Andrei Borzenkov wrote: | On Thu, Jan 30, 2020 at 6:53 PM Per Jessen <per@computer.org> | | wrote: |> Andrei Borzenkov wrote: |>> On Thu, Jan 30, 2020 at 4:29 PM Per Jessen <per@computer.org> |>> |>> wrote: |>>> In the process lists from dmesg, I see 266 processes using up |>>> about 1Gb (RSS) which doesn't seem like a lot? |>> |>> It's in pages, so multiply by 4K. Active anonymous memory alone |>> is around 6GB. |> |> Ah, thanks - I was thinking it was kilobytes. |> |>> This is known problem. There is a lot of active memory so every |>> time kernel has to search a lot to find free page (or page that |>> can be reclaimed). |> |> that could lead to an oom condition? | | Rather this is indication of low memory. | | In this case free RAM is below absolute minimum that kernel | attempts to keep: | | Node 0 Normal free:102036kB min:106080kB
Ah! That's the start condition? I also have that in the log of my incident.
<0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921018] Node 0 Normal free:105704kB min:106040kB low:116644kB high:127248kB
That's about 100 Mb, right?
active_anon:1885752kB inactive_anon:626456kB active_file:371848kB inactive_file:1478016kB unevictable:10880kB writepending:239740kB present:5242880kB managed:5110480kB mlocked:10880kB slab_reclaimable:170100kB slab_unreclaimable:162136kB kernel_stack:21080kB pagetables:91024kB bounce:0kB free_pcp:2776kB local_pcp:1280kB free_cma:0kB
Any document you know that explains these?
And no knowing how the machine got there.
| Which means on every allocation request kernel will synchronously | attempt to free enough memory to go above watermark. It looks like | it tries it actually: | | writeback:1251356kB writeback:1671120kB writeback:1876996kB | writeback:1982080kB writeback:1961032kB
I have one instance of that word.
<0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921004] Node 0 active_anon:2560096kB inactive_anon:1335884kB active_file:461648kB inactive_file:2786820kB unevictable:10880kB isolated(anon):0kB isolated(file):256kB mapped:495892kB dirty:23756kB writeback:391168kB shmem:212684kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 382976kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
| This started with almost 2GB of data that is being written to | disk.
How do you know that? :-? (I want to find it in my case)
My machine was at the time probably copying a large directory from one disk to another, using 'mc', and nobody seated.
| At some point kernel managed to complete it just to get more and | more new data to be written. If kernel needs to wait while dirty | pages are written to disk it pretty much explains those "allocation | stalled" warnings. | | This goes on until no more free swap is available (which probably | means - kernel cannot free memory to go above watermark at all) at | which point OOM kicks in.
I don't think that happened in my case, but I do not know. I have 25 GB of swap.
| So it appears like some application(s) really use that much | memory. IOW system is overloaded. | | As for "If you sudo swapoff -a the swap turns into memory, the | system turns responsive" it is far too vague to make any useful | comment. It is impossible to do "swapoff" under conditions shown in | dmesg output. If this was supposed to mean "swapoff before this | situation is reached" - sure, it will eliminate stalls due to | writing data to disk, it will also cause OOM earlier. If RAM+swap | is not enough for current workload, then RAM alone is obviously not | enough as well.
This may be because read/write to swap with rotating disk is way too slow and involves many seeks. I observed that in my machine when I switched to Leap, and stopped when I bought an SSD disk and put the swap there.
-- Cheers / Saludos,
Carlos E. R. (from 15.1 x86_64 at Telcontar) I am using a Samsung 850 Pro SSD (just for completeness.
_________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/01/2020 13.16, Per Jessen wrote: | stakanov wrote: | |> I did create a paste in |> |> https://paste.opensuse.org/22334414 |> |> |> I am not good enough to understand what the system actually |> complains. | | It looks like an out-of-memory condition, and your system has used | up all the swap space. Something is gobbleing up memory. Not memory, just swap. Apparently he has 25% of RAM free, before and after. He needs to post command sequence (commands and their output in a single copy paste operation) proving this, specially for the bugzilla. The machine becoming unresponsive can be a side effect of swap being on rotating rust media; it is much faster on SSD. My personal belief is that there was some change in swap handling on Leap 42, related to something causing many swap disk seeks (maybe fragmentation?). This belief is based on the fact that the day I upgraded from the 13.x series to the Leap series, I noticed the impact on the machine. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjLPWQAKCRC1MxgcbY1H 1QL8AJ93Gt2B5ZudWCVoWrAN4euTNT02YwCfU4BhuL3SPin+VeWjCJO0FTA9yew= =opB1 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 30/01/2020 13.16, Per Jessen wrote: | stakanov wrote: | |> I did create a paste in |> |> https://paste.opensuse.org/22334414 |> |> |> I am not good enough to understand what the system actually |> complains. | | It looks like an out-of-memory condition, and your system has used | up all the swap space. Something is gobbleing up memory.
Not memory, just swap.
I see the oom killer in the output posted. If swap is full, so it likely memory. Anyway, just my opinion. -- Per Jessen, Zürich (10.2°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/01/2020 14.13, Per Jessen wrote: | Carlos E. R. wrote: |> On 30/01/2020 13.16, Per Jessen wrote: | stakanov wrote: | |> I |> did create a paste in |> |> https://paste.opensuse.org/22334414 |> |> |> |> I am not good enough to understand what the system |> actually |> complains. | | It looks like an out-of-memory |> condition, and your system has used | up all the swap space. |> Something is gobbleing up memory. |> |> Not memory, just swap. | | I see the oom killer in the output posted. If swap is full, so it | likely memory. Anyway, just my opinion. I don't know, I'm going by his description. - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjMSMgAKCRC1MxgcbY1H 1X+KAJ93HtRQCjhK5k+vxRLNbtc7GWIz8ACfbuLB75XcQcxNnGeKDUUf7afChR4= =N8EJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/30/2020 04:18 AM, stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive. sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.)
I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available?
Ouch! Not a hardware problem, a "Running Out Of Memory" problem. Something is really, really memory hungry. Here a couple of bits from your file: Something is occupying over 3G of memory and then kernel has nearly 900M of writes pending. The big users seem to be Baloo (which is the file indexing thing...) and QtWebEngine -- not sure what browser that goes to, but likely a browser or somehow connected to Baloo. The big tell is the kernel message: Out of memory: Kill process 4478 (QtWebEngineProc) score 302 or sacrifice child Out of memory: Kill process 5251 (QtWebEngineProc) score 301 or sacrifice child Do you have a lot of tabs open in your browser? Do you need indexing? -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-30 8:48 p.m., David C. Rankin wrote:
The big tell is the kernel message:
Out of memory: Kill process 4478 (QtWebEngineProc) score 302 or sacrifice child
Out of memory: Kill process 5251 (QtWebEngineProc) score 301 or sacrifice child
UNLESS he has reconfigured to OOM_Killer settings to the non-default value which means that the process causing the OOM is the one that gets killed, this is only marginally significant. The algorithm 'picks one', iteratively, according to its own evaluation criteria. I've mentioned this difference in previous postings. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/01/2020 11.18, stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive. sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.)
I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available?
I just noticed that today (actually yesterday) I had a similar incident. Log: <0.6> 2020-01-30 15:02:31 Telcontar kernel - - - [282156.986472] usb 4-2: USB disconnect, device number 3 That was the scanner, means I powered off the auxiliaries before leaving the room. <3.6> 2020-01-30 15:17:45 Telcontar smartd 1471 - - Device: /dev/sdc [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 76 to 77 <3.6> 2020-01-30 15:17:45 Telcontar smartd 1471 - - Device: /dev/sdc [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 76 to 77 <3.6> 2020-01-30 15:17:47 Telcontar smartd 1471 - - Device: /dev/sde [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 75 <10.6> 2020-01-30 15:20:01 Telcontar cron 22415 - - pam_unix(crond:session): session opened for user cer by (uid=0) <10.6> 2020-01-30 15:20:01 Telcontar CRON 22415 - - pam_unix(crond:session): session closed for user cer <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920913] lxterminal: page allocation stalls for 13732ms, order:0, mode:0x14000c0(GFP_KERNEL), nodemask=(null) <0.6> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920919] lxterminal cpuset=/ mems_allowed=0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920924] CPU: 3 PID: 5491 Comm: lxterminal Tainted: P O 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920925] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7516/MS-7516, BIOS V1.5 10/10/2008 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920926] Call Trace: <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920935] dump_stack+0x5c/0x86 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920938] warn_alloc+0xe0/0x170 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920940] __alloc_pages_slowpath+0x7e2/0xc10 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920943] ? __switch_to_asm+0x40/0x70 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920944] ? __switch_to_asm+0x34/0x70 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920946] __alloc_pages_nodemask+0x246/0x260 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920948] alloc_pages_current+0x72/0x140 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920950] __get_free_pages+0xa/0x40 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920953] __pollwait+0x4d/0xe0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920956] n_tty_poll+0x3f/0x1d0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920957] tty_poll+0x6d/0x90 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920959] do_sys_poll+0x239/0x530 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920963] ? mutex_lock+0xe/0x30 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920965] ? unix_stream_read_generic+0x1e8/0x860 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920966] ? compat_set_fd_set+0x80/0x80 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920968] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920969] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920971] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920972] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920973] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920975] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920976] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920978] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920979] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920980] ? SyS_poll+0x70/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920982] SyS_poll+0x70/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920985] do_syscall_64+0x7b/0x160 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920986] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920989] RIP: 0033:0x7f2433ee419b <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920990] RSP: 002b:00007fff89cf0ce0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920992] RAX: ffffffffffffffda RBX: 000055dc7b0f60c0 RCX: 00007f2433ee419b <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920993] RDX: 000000000000001e RSI: 000000000000000d RDI: 000055dc7b0f60c0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920993] RBP: 000000000000000d R08: 0000000000000000 R09: 000055dc7ae28f80 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920994] R10: 000055dc7ad93b60 R11: 0000000000000293 R12: 000055dc7b0f60c0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920995] R13: 000000000000001e R14: 00007f2435684930 R15: 000000000000000d <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920996] Mem-Info: <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] active_anon:640024 inactive_anon:333971 isolated_anon:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] active_file:115412 inactive_file:696705 isolated_file:64 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] unevictable:2720 dirty:5939 writeback:97792 unstable:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] slab_reclaimable:65197 slab_unreclaimable:52386 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] mapped:123973 shmem:53171 pagetables:25742 bounce:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] free:50936 free_pcp:1095 free_cma:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921004] Node 0 active_anon:2560096kB inactive_anon:1335884kB active_file:461648kB inactive_file:2786820kB unevictable:10880kB isolated(anon):0kB isolated(file):256kB mapped:495892kB dirty:23756kB writeback:391168kB shmem:212684kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 382976kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921005] Node 0 DMA free:15876kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921008] lowmem_reserve[]: 0 2945 7936 7936 7936 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921011] Node 0 DMA32 free:82164kB min:62576kB low:68832kB high:75088kB active_anon:674324kB inactive_anon:709408kB active_file:89800kB inactive_file:1309124kB unevictable:0kB writepending:174448kB present:3128896kB managed:3034868kB mlocked:0kB slab_reclaimable:90688kB slab_unreclaimable:47408kB kernel_stack:3048kB pagetables:11944kB bounce:0kB free_pcp:1604kB local_pcp:668kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921015] lowmem_reserve[]: 0 0 4990 4990 4990 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921018] Node 0 Normal free:105704kB min:106040kB low:116644kB high:127248kB active_anon:1885752kB inactive_anon:626456kB active_file:371848kB inactive_file:1478016kB unevictable:10880kB writepending:239740kB present:5242880kB managed:5110480kB mlocked:10880kB slab_reclaimable:170100kB slab_unreclaimable:162136kB kernel_stack:21080kB pagetables:91024kB bounce:0kB free_pcp:2776kB local_pcp:1280kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921022] lowmem_reserve[]: 0 0 0 0 0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921024] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15876kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921034] Node 0 DMA32: 1318*4kB (UME) 5388*8kB (UME) 1571*16kB (UE) 235*32kB (UE) 2*64kB (U) 0*128kB 1*256kB (M) 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 82440kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921043] Node 0 Normal: 7530*4kB (UMH) 5345*8kB (UMEH) 1659*16kB (UMEH) 200*32kB (UMEH) 4*64kB (M) 3*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 106720kB <0.6> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921053] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921054] 1064990 total pagecache pages <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921061] 197345 pages in swap cache <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921062] Swap cache stats: add 24031290, delete 23835992, find 116458533/122287290 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921063] Free swap = 18066196kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921063] Total swap = 25165820kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921064] 2096942 pages RAM <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921065] 0 pages HighMem/MovableOnly <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921065] 56628 pages reserved <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921066] 0 pages hwpoisoned <10.6> 2020-01-30 15:30:01 Telcontar cron 22750 - - pam_unix(crond:session): session opened for user cer by (uid=0) <9.6> 2020-01-30 15:30:01 Telcontar CRON 22759 - - (cer) CMD (/home/cer/bin/dar_la_hora_en_cron hora) I think I was having lunch at the time, nobody was using the machine. Maybe I was copying many files with 'mc' in the lxterminal. I suspect it finished at 15:20:42 (because of timestamp of files as shown with "ls -ct --full-time", just seconds after the incident: drwxr-xr-x 3 root root 63 2020-01-30 15:20:42.546288581 +0100 Jazz2_small/ drwxr-xr-x 4 root root 50 2020-01-30 15:20:42.518289652 +0100 Jazz2_Big/ - -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXjOQ8wAKCRC1MxgcbY1H 1SblAJ9VDSF2Jl8Fbhtx++2CDhRklZ4NGwCeNQvjQaIOZqEUVXLEYLbkKJVPmZE= =eFtc -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
In data venerdì 31 gennaio 2020 03:29:20 CET, Carlos E. R. ha scritto:
On 30/01/2020 11.18, stakanov wrote:
I did create a paste in
https://paste.opensuse.org/22334414
I am not good enough to understand what the system actually complains. I do not think it is a memory problem, could be related to the OS instead. The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive. sudo dmesg reveals a flood of entries (disregard the martians, these are due to a vpn.)
I do not understand if, and if which, hardware is failing. Memory does nor reveal errors. Sometimes I get a complaint about CPU3 should not be sleeping. This is all I know. Thanks in advance if somebody understands (and maybe can a bit explain) the output. Why does the swap not return to memory once memory is abundantly available?
I just noticed that today (actually yesterday) I had a similar incident.
Log:
<0.6> 2020-01-30 15:02:31 Telcontar kernel - - - [282156.986472] usb 4-2: USB disconnect, device number 3
That was the scanner, means I powered off the auxiliaries before leaving the room.
<3.6> 2020-01-30 15:17:45 Telcontar smartd 1471 - - Device: /dev/sdc [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 76 to 77 <3.6> 2020-01-30 15:17:45 Telcontar smartd 1471 - - Device: /dev/sdc [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 76 to 77 <3.6> 2020-01-30 15:17:47 Telcontar smartd 1471 - - Device: /dev/sde [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 75 <10.6> 2020-01-30 15:20:01 Telcontar cron 22415 - - pam_unix(crond:session): session opened for user cer by (uid=0) <10.6> 2020-01-30 15:20:01 Telcontar CRON 22415 - - pam_unix(crond:session): session closed for user cer <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920913] lxterminal: page allocation stalls for 13732ms, order:0, mode:0x14000c0(GFP_KERNEL), nodemask=(null) <0.6> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920919] lxterminal cpuset=/ mems_allowed=0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920924] CPU: 3 PID: 5491 Comm: lxterminal Tainted: P O 4.12.14-lp151.28.36-default #1 openSUSE Leap 15.1 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920925] Hardware name: MICRO-STAR INTERNATIONAL CO.,LTD MS-7516/MS-7516, BIOS V1.5 10/10/2008 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920926] Call Trace: <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920935] dump_stack+0x5c/0x86 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920938] warn_alloc+0xe0/0x170 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920940] __alloc_pages_slowpath+0x7e2/0xc10 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920943] ? __switch_to_asm+0x40/0x70 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920944] ? __switch_to_asm+0x34/0x70 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920946] __alloc_pages_nodemask+0x246/0x260 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920948] alloc_pages_current+0x72/0x140 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920950] __get_free_pages+0xa/0x40 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920953] __pollwait+0x4d/0xe0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920956] n_tty_poll+0x3f/0x1d0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920957] tty_poll+0x6d/0x90 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920959] do_sys_poll+0x239/0x530 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920963] ? mutex_lock+0xe/0x30 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920965] ? unix_stream_read_generic+0x1e8/0x860 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920966] ? compat_set_fd_set+0x80/0x80 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920968] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920969] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920971] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920972] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920973] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920975] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920976] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920978] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920979] ? compat_poll_select_copy_remaining+0x100/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920980] ? SyS_poll+0x70/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920982] SyS_poll+0x70/0x100 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920985] do_syscall_64+0x7b/0x160 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920986] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920989] RIP: 0033:0x7f2433ee419b <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920990] RSP: 002b:00007fff89cf0ce0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920992] RAX: ffffffffffffffda RBX: 000055dc7b0f60c0 RCX: 00007f2433ee419b <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920993] RDX: 000000000000001e RSI: 000000000000000d RDI: 000055dc7b0f60c0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920993] RBP: 000000000000000d R08: 0000000000000000 R09: 000055dc7ae28f80 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920994] R10: 000055dc7ad93b60 R11: 0000000000000293 R12: 000055dc7b0f60c0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920995] R13: 000000000000001e R14: 00007f2435684930 R15: 000000000000000d <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.920996] Mem-Info: <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] active_anon:640024 inactive_anon:333971 isolated_anon:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] active_file:115412 inactive_file:696705 isolated_file:64 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] unevictable:2720 dirty:5939 writeback:97792 unstable:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] slab_reclaimable:65197 slab_unreclaimable:52386 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] mapped:123973 shmem:53171 pagetables:25742 bounce:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921001] free:50936 free_pcp:1095 free_cma:0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921004] Node 0 active_anon:2560096kB inactive_anon:1335884kB active_file:461648kB inactive_file:2786820kB unevictable:10880kB isolated(anon):0kB isolated(file):256kB mapped:495892kB dirty:23756kB writeback:391168kB shmem:212684kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 382976kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921005] Node 0 DMA free:15876kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921008] lowmem_reserve[]: 0 2945 7936 7936 7936 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921011] Node 0 DMA32 free:82164kB min:62576kB low:68832kB high:75088kB active_anon:674324kB inactive_anon:709408kB active_file:89800kB inactive_file:1309124kB unevictable:0kB writepending:174448kB present:3128896kB managed:3034868kB mlocked:0kB slab_reclaimable:90688kB slab_unreclaimable:47408kB kernel_stack:3048kB pagetables:11944kB bounce:0kB free_pcp:1604kB local_pcp:668kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921015] lowmem_reserve[]: 0 0 4990 4990 4990 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921018] Node 0 Normal free:105704kB min:106040kB low:116644kB high:127248kB active_anon:1885752kB inactive_anon:626456kB active_file:371848kB inactive_file:1478016kB unevictable:10880kB writepending:239740kB present:5242880kB managed:5110480kB mlocked:10880kB slab_reclaimable:170100kB slab_unreclaimable:162136kB kernel_stack:21080kB pagetables:91024kB bounce:0kB free_pcp:2776kB local_pcp:1280kB free_cma:0kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921022] lowmem_reserve[]: 0 0 0 0 0 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921024] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15876kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921034] Node 0 DMA32: 1318*4kB (UME) 5388*8kB (UME) 1571*16kB (UE) 235*32kB (UE) 2*64kB (U) 0*128kB 1*256kB (M) 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 82440kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921043] Node 0 Normal: 7530*4kB (UMH) 5345*8kB (UMEH) 1659*16kB (UMEH) 200*32kB (UMEH) 4*64kB (M) 3*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 106720kB <0.6> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921053] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921054] 1064990 total pagecache pages <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921061] 197345 pages in swap cache <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921062] Swap cache stats: add 24031290, delete 23835992, find 116458533/122287290 <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921063] Free swap = 18066196kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921063] Total swap = 25165820kB <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921064] 2096942 pages RAM <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921065] 0 pages HighMem/MovableOnly <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921065] 56628 pages reserved <0.4> 2020-01-30 15:20:37 Telcontar kernel - - - [283242.921066] 0 pages hwpoisoned <10.6> 2020-01-30 15:30:01 Telcontar cron 22750 - - pam_unix(crond:session): session opened for user cer by (uid=0) <9.6> 2020-01-30 15:30:01 Telcontar CRON 22759 - - (cer) CMD (/home/cer/bin/dar_la_hora_en_cron hora)
I think I was having lunch at the time, nobody was using the machine. Maybe I was copying many files with 'mc' in the lxterminal. I suspect it finished at 15:20:42 (because of timestamp of files as shown with "ls -ct --full-time", just seconds after the incident:
drwxr-xr-x 3 root root 63 2020-01-30 15:20:42.546288581 +0100 Jazz2_small/ drwxr-xr-x 4 root root 50 2020-01-30 15:20:42.518289652 +0100 Jazz2_Big/
-- Cheers / Saludos,
Carlos E. R. (from 15.1 x86_64 at Telcontar)
For me this condition is much more frequent when the machine previously idled and when it has been suspended the day before. Do not know if this helps. _________________________________________________________________ ________________________________________________________ Ihre E-Mail-Postfächer sicher & zentral an einem Ort. Jetzt wechseln und alte E-Mail-Adresse mitnehmen! https://www.eclipso.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/30/20 11:18, stakanov wrote:
The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive.
I got this issue the day I updated to kernel 5.3. Going back to kernel 5.2 fixed the issue. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-31 01:32 PM, Christoph Feck wrote:
On 01/30/20 11:18, stakanov wrote:
The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive.
I got this issue the day I updated to kernel 5.3. Going back to kernel 5.2 fixed the issue.
I also turned off swap today, after reading that note. My system seems to be much more responsive. However, I'll have to give it a few days to be sure. I've had some discussion about this here before and haven't found the cause. However, closing browsers seems to help. My kernel is 4.12 with Leap 15.1. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* James Knott <james.knott@jknott.net> [01-31-20 13:42]:
On 2020-01-31 01:32 PM, Christoph Feck wrote:
On 01/30/20 11:18, stakanov wrote:
The swap fills although the memory remains free. If you sudo swapoff -a the swap turns into memory, the system turns responsive.
I got this issue the day I updated to kernel 5.3. Going back to kernel 5.2 fixed the issue.
I also turned off swap today, after reading that note. My system seems to be much more responsive. However, I'll have to give it a few days to be sure. I've had some discussion about this here before and haven't found the cause. However, closing browsers seems to help.
My kernel is 4.12 with Leap 15.1.
no swap kernel-default-5.4.14-1.1.x86_64 Tw no problem -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-31 1:41 p.m., James Knott wrote:
I also turned off swap today, after reading that note. My system seems to be much more responsive.
that is the extreme of setting 'swapiness' to zero, which I've touched n. There's evidence that if you let it, if 'swapiness' is large enough, then Linux does what might be called 'pre-emotive' swapping, swapping out what it thinks you might not be using at some time in the future, maybe, perhaps. Or is that a guess? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2020-01-31 10:06 PM, Anton Aylward wrote:
I also turned off swap today, after reading that note. My system seems to be much more responsive.
On 2020-01-31 1:41 p.m., James Knott wrote: that is the extreme of setting 'swapiness' to zero, which I've touched n.
There's evidence that if you let it, if 'swapiness' is large enough, then Linux does what might be called 'pre-emotive' swapping, swapping out what it thinks you might not be using at some time in the future, maybe, perhaps. Or is that a guess?
I have absolutely no idea. All I know is so far my system appears to be running better without swap than with. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (13)
-
Andrei Borzenkov -
Anton Aylward -
Carlos E. R. -
Christoph Feck -
Darryl Gregorash -
David C. Rankin -
Felix Miata -
Hans-Peter Jansen -
James Knott -
medwinz -
Patrick Shanahan -
Per Jessen -
stakanov