[opensuse-kernel] VM swapping behaviour broken in 2.6.22.12-0.1-default?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I am desperately looking for clues on a very strange behaviour that I now have experienced with openSUSE 10.3 for several times now (actually, every 2-3 days on my Laptop, which is my primary workstation and up 24x7). The system works fine for 1-2 days, then all the sudden it becomes less and less responsive due to massive swap activity. Looking at the output of vmstat and free, I observe that the VM decided to swap out large amounts of memory pages, while there still was plenty of RAM freely available (not used for buffers or cache). So the system ends up with over half of the RAM being marked as free, while the swap has filled up quite significantly. This makes the system become *very* sluggish. It happened to me several times now that I returned to my desk in the morning and the system was completely unresponsive and the disk was rattling as mad. Only a hard reboot helped to regain control (Since I can't always wait ~1 hour for the Alt+SysRq+E to take effect). I filed a bug report about this a few days ago, it would be great if some kernel expert could investigate this and let me know, what other information I need to provide. https://bugzilla.novell.com/show_bug.cgi?id=340143 I toyed around with various swappiness values, but they don't seem to make a difference. Interestingly, even setting /proc/sys/vm/swappiness to "0" does not completely prohibit the system from swapping, while there still is plenty of free memory available: lenz@metis:~> uname -a Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686 i686 i386 GNU/Linux lenz@metis:~> uptime 8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1014084 21248 0 87160 197488 - -/+ buffers/cache: 729436 305896 Swap: 1253028 131076 1121952 lenz@metis:~> cat /proc/vmstat nr_free_pages 5250 nr_inactive 26913 nr_active 214996 nr_anon_pages 169683 nr_mapped 29874 nr_file_pages 79046 nr_dirty 721 nr_writeback 0 nr_slab_reclaimable 2394 nr_slab_unreclaimable 3436 nr_page_table_pages 965 nr_unstable 0 nr_bounce 0 nr_vmscan_write 135771 pgpgin 25483536 pgpgout 10607861 pswpin 14777 pswpout 36166 pgalloc_dma 169320 pgalloc_normal 29403345 pgalloc_high 2876515 pgfree 32454506 pgactivate 1380976 pgdeactivate 1156011 pgfault 35317571 pgmajfault 13960 pgrefill_dma 571625 pgrefill_normal 29974997 pgrefill_high 5063004 pgsteal_dma 91237 pgsteal_normal 7634076 pgsteal_high 333204 pgscan_kswapd_dma 191877 pgscan_kswapd_normal 7868615 pgscan_kswapd_high 430492 pgscan_direct_dma 3991 pgscan_direct_normal 289792 pgscan_direct_high 19388 pginodesteal 6 slabs_scanned 4068352 kswapd_steal 7819731 kswapd_inodesteal 17211 pageoutrun 132841 allocstall 3593 pgrotated 115105 Is there anything else you need? Any help would be appreciated - I don't seem to be the only one experiencing this. I spotted one message on the general openSUSE List, and there was a thread on this list about "Resume take a long time to bring in the images to main memory" which sounded like it may suffer from the same problem. Thanks! Bye, LenZ - -- - ------------------------------------------------------------------ Lenz Grimmer <lenz@grimmer.com> -o) [ICQ: 160767607 | Jabber: LenZGr@jabber.org] /\\ http://www.lenzg.org/ V_V -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHQe0FSVDhKrJykfIRAlW9AJ9/+lp+g311lHy1z7byMq0AZyLHoACfYuxF J7kLQb5fjAjQaXHQhjsBm7Y= =mpMz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862 --jaimon Lenz Grimmer wrote, On 11/20/2007 01:54 AM:
Hi,
I am desperately looking for clues on a very strange behaviour that I now have experienced with openSUSE 10.3 for several times now (actually, every 2-3 days on my Laptop, which is my primary workstation and up 24x7).
The system works fine for 1-2 days, then all the sudden it becomes less and less responsive due to massive swap activity. Looking at the output of vmstat and free, I observe that the VM decided to swap out large amounts of memory pages, while there still was plenty of RAM freely available (not used for buffers or cache). So the system ends up with over half of the RAM being marked as free, while the swap has filled up quite significantly. This makes the system become *very* sluggish.
It happened to me several times now that I returned to my desk in the morning and the system was completely unresponsive and the disk was rattling as mad. Only a hard reboot helped to regain control (Since I can't always wait ~1 hour for the Alt+SysRq+E to take effect).
I filed a bug report about this a few days ago, it would be great if some kernel expert could investigate this and let me know, what other information I need to provide.
https://bugzilla.novell.com/show_bug.cgi?id=340143
I toyed around with various swappiness values, but they don't seem to make a difference. Interestingly, even setting /proc/sys/vm/swappiness to "0" does not completely prohibit the system from swapping, while there still is plenty of free memory available:
lenz@metis:~> uname -a Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686 i686 i386 GNU/Linux lenz@metis:~> uptime 8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1014084 21248 0 87160 197488 -/+ buffers/cache: 729436 305896 Swap: 1253028 131076 1121952 lenz@metis:~> cat /proc/vmstat nr_free_pages 5250 nr_inactive 26913 nr_active 214996 nr_anon_pages 169683 nr_mapped 29874 nr_file_pages 79046 nr_dirty 721 nr_writeback 0 nr_slab_reclaimable 2394 nr_slab_unreclaimable 3436 nr_page_table_pages 965 nr_unstable 0 nr_bounce 0 nr_vmscan_write 135771 pgpgin 25483536 pgpgout 10607861 pswpin 14777 pswpout 36166 pgalloc_dma 169320 pgalloc_normal 29403345 pgalloc_high 2876515 pgfree 32454506 pgactivate 1380976 pgdeactivate 1156011 pgfault 35317571 pgmajfault 13960 pgrefill_dma 571625 pgrefill_normal 29974997 pgrefill_high 5063004 pgsteal_dma 91237 pgsteal_normal 7634076 pgsteal_high 333204 pgscan_kswapd_dma 191877 pgscan_kswapd_normal 7868615 pgscan_kswapd_high 430492 pgscan_direct_dma 3991 pgscan_direct_normal 289792 pgscan_direct_high 19388 pginodesteal 6 slabs_scanned 4068352 kswapd_steal 7819731 kswapd_inodesteal 17211 pageoutrun 132841 allocstall 3593 pgrotated 115105
Is there anything else you need? Any help would be appreciated - I don't seem to be the only one experiencing this. I spotted one message on the general openSUSE List, and there was a thread on this list about "Resume take a long time to bring in the images to main memory" which sounded like it may suffer from the same problem.
Thanks!
Bye, LenZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jaimon, thanks for your message! Jaimon Jose wrote:
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862
Nope, I even disabled DRI in my X Server configuration some time ago (as it's still very unstable when using the opensource radeon driver). I really suspect a VM bug here. Why is it using swap even when there is no memory pressure and swappiness is set to zero? lenz@metis:~> uptime 10:26am up 1 day 0:21, 8 users, load average: 0.40, 0.73, 0.98 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1020224 15108 0 150936 222276 - -/+ buffers/cache: 647012 388320 Swap: 1253028 349804 903224 Note that this is just about 2:30 hours after I did a "swapoff -a ; swapon -a" this morning (in the attempt to move the swapped out pages back into memory). Bye, LenZ - -- - ------------------------------------------------------------------ Lenz Grimmer <lenz@grimmer.com> -o) [ICQ: 160767607 | Jabber: LenZGr@jabber.org] /\\ http://www.lenzg.org/ V_V -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHQqirSVDhKrJykfIRAgrXAJ9y1tlwX5zO/fnslK9gdic20tCBlACfbJ3V WZ+AFn7kX9sl9wWi2SJEK7U= =B156 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jaimon, Lenz Grimmer wrote:
I really suspect a VM bug here. Why is it using swap even when there is no memory pressure and swappiness is set to zero?
lenz@metis:~> uptime 10:26am up 1 day 0:21, 8 users, load average: 0.40, 0.73, 0.98 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1020224 15108 0 150936 222276 -/+ buffers/cache: 647012 388320 Swap: 1253028 349804 903224
Note that this is just about 2:30 hours after I did a "swapoff -a ; swapon -a" this morning (in the attempt to move the swapped out pages back into memory).
The system again started to swap heavily and more and more memory appeared as "free" during the course of the day. I was able to regain control by termination several apps (Firefox, OOo, Thunderbird) and disabling swap temporarily (to force the VM to swap in all pages again), but this is becoming a major annoyance. I wonder if I am the only one experiencing this? Any idea why it ignores the swappiness setting? Or why the VM decides to start swapping in the first place, if there is plenty of free memory? I'd be happy to assist you in finding the cause of this, just let me know what other info you would need or what info to watch. Thanks! Bye, LenZ - -- - ------------------------------------------------------------------ Lenz Grimmer <lenz@grimmer.com> -o) [ICQ: 160767607 | Jabber: LenZGr@jabber.org] /\\ http://www.lenzg.org/ V_V -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHRGMsSVDhKrJykfIRApg3AJ4yCQl8DU/Q79Z7WHGRQG1/boqYsACeKOTc UvKuI2shIzr22aYCAZfq2bQ= =DZch -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit. I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it Anders -- Madness takes its toll -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be. I'm now in 32 bit installation (box is Athlon 64) and swap is not used. One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 02:09:39 pm Rajko M. wrote:
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be.
I'm now in 32 bit installation (box is Athlon 64) and swap is not used.
One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works.
1) ~> free total used free shared buffers cached Mem: 905804 884992 20812 0 18116 651424 -/+ buffers/cache: 215452 690352 Swap: 2104472 420 2104052 2) ~> free total used free shared buffers cached Mem: 905804 825180 80624 0 76312 523004 -/+ buffers/cache: 225864 679940 Swap: 2104472 86576 2017896 System was not used between 2 outputs for hours, though, it was probably running daily cron jobs. Currently (after another few hours) when system was used: 3) ~> free total used free shared buffers cached Mem: 905804 893640 12164 0 77552 540008 -/+ buffers/cache: 276080 629724 Swap: 2104472 85800 2018672 The /proc/meminfo shows different for 1) and 2) MemFree:71220 kB Some 400 MB was used by virtual machine, so 'free' report seems way out of line. It is OK for the rest of the system, but I can't see this 400 MB as used. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Friday 23 November 2007 13:22:22 Rajko M. wrote:
On Thursday 22 November 2007 02:09:39 pm Rajko M. wrote:
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be.
I'm now in 32 bit installation (box is Athlon 64) and swap is not used.
One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works.
1) ~> free total used free shared buffers cached Mem: 905804 884992 20812 0 18116 651424 -/+ buffers/cache: 215452 690352 Swap: 2104472 420 2104052
2) ~> free total used free shared buffers cached Mem: 905804 825180 80624 0 76312 523004 -/+ buffers/cache: 225864 679940 Swap: 2104472 86576 2017896
System was not used between 2 outputs for hours, though, it was probably running daily cron jobs.
Currently (after another few hours) when system was used:
3) ~> free total used free shared buffers cached Mem: 905804 893640 12164 0 77552 540008 -/+ buffers/cache: 276080 629724 Swap: 2104472 85800 2018672
The /proc/meminfo shows different for 1) and 2) MemFree:71220 kB Some 400 MB was used by virtual machine, so 'free' report seems way out of line. It is OK for the rest of the system, but I can't see this 400 MB as used.
The interesting part from meminfo is LowFree. If that gets very very low, the memory pressure will be high on the machine regardless of how much highmem is free Anders -- Madness takes its toll -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Hi!
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862
And I have something s2ram-related reported, too... Are you sure userspace is not responsible? Reproducing it w/o X would be nice... Pavel
Lenz Grimmer wrote, On 11/20/2007 01:54 AM:
Hi,
I am desperately looking for clues on a very strange behaviour that I now have experienced with openSUSE 10.3 for several times now (actually, every 2-3 days on my Laptop, which is my primary workstation and up 24x7).
The system works fine for 1-2 days, then all the sudden it becomes less and less responsive due to massive swap activity. Looking at the output of vmstat and free, I observe that the VM decided to swap out large amounts of memory pages, while there still was plenty of RAM freely available (not used for buffers or cache). So the system ends up with over half of the RAM being marked as free, while the swap has filled up quite significantly. This makes the system become *very* sluggish.
It happened to me several times now that I returned to my desk in the morning and the system was completely unresponsive and the disk was rattling as mad. Only a hard reboot helped to regain control (Since I can't always wait ~1 hour for the Alt+SysRq+E to take effect).
I filed a bug report about this a few days ago, it would be great if some kernel expert could investigate this and let me know, what other information I need to provide.
https://bugzilla.novell.com/show_bug.cgi?id=340143
I toyed around with various swappiness values, but they don't seem to make a difference. Interestingly, even setting /proc/sys/vm/swappiness to "0" does not completely prohibit the system from swapping, while there still is plenty of free memory available:
lenz@metis:~> uname -a Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686 i686 i386 GNU/Linux lenz@metis:~> uptime 8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1014084 21248 0 87160 197488 -/+ buffers/cache: 729436 305896 Swap: 1253028 131076 1121952 lenz@metis:~> cat /proc/vmstat nr_free_pages 5250 nr_inactive 26913 nr_active 214996 nr_anon_pages 169683 nr_mapped 29874 nr_file_pages 79046 nr_dirty 721 nr_writeback 0 nr_slab_reclaimable 2394 nr_slab_unreclaimable 3436 nr_page_table_pages 965 nr_unstable 0 nr_bounce 0 nr_vmscan_write 135771 pgpgin 25483536 pgpgout 10607861 pswpin 14777 pswpout 36166 pgalloc_dma 169320 pgalloc_normal 29403345 pgalloc_high 2876515 pgfree 32454506 pgactivate 1380976 pgdeactivate 1156011 pgfault 35317571 pgmajfault 13960 pgrefill_dma 571625 pgrefill_normal 29974997 pgrefill_high 5063004 pgsteal_dma 91237 pgsteal_normal 7634076 pgsteal_high 333204 pgscan_kswapd_dma 191877 pgscan_kswapd_normal 7868615 pgscan_kswapd_high 430492 pgscan_direct_dma 3991 pgscan_direct_normal 289792 pgscan_direct_high 19388 pginodesteal 6 slabs_scanned 4068352 kswapd_steal 7819731 kswapd_inodesteal 17211 pageoutrun 132841 allocstall 3593 pgrotated 115105
Is there anything else you need? Any help would be appreciated - I don't seem to be the only one experiencing this. I spotted one message on the general openSUSE List, and there was a thread on this list about "Resume take a long time to bring in the images to main memory" which sounded like it may suffer from the same problem.
Thanks!
Bye, LenZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Pavel, thanks for your reply! Pavel Machek wrote:
Are you sure userspace is not responsible?
Of course I am not, all I can say is that this is a change in behaviour from 10.2 to 10.3, while I did not change my way of working with this system as a Desktop machine (e.g. which applications I keep running)...
Reproducing it w/o X would be nice...
Unfortunately this is something I am not able to test. I need this box as my primary workstation and not being able to use X would severely hamper this purpose :( Would it help to also give you a list of all running processes and their memory usage, next time it happens? Just let me know which output of which command would be helpful. Somebody informed me that he also noticed similar reports on the LKML. I asked him if he could provide pointers, which I will add to the bug report. Bye, LenZ - -- - ------------------------------------------------------------------ Lenz Grimmer <lenz@grimmer.com> -o) [ICQ: 160767607 | Jabber: LenZGr@jabber.org] /\\ http://www.lenzg.org/ V_V -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHRITbSVDhKrJykfIRAlcgAJwNVa9uCoxcsx6mb/t6NTLLjIfafQCff47F 9RjbYCel5kSBZD58yYhSnew= =BHAt -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Replying to myself) Lenz Grimmer wrote:
Of course I am not, all I can say is that this is a change in behaviour from 10.2 to 10.3, while I did not change my way of working with this system as a Desktop machine (e.g. which applications I keep running)...
It happened again this morning - returning to my desk which I left 6 hours before in a working state, the system again was very unresponsive and the hard disk was rattling as mad. I managed to capture some stats before rebooting it: lenz@metis:~> date Wed Nov 28 10:00:59 CET 2007 lenz@metis:~> uptime 10:01am up 2 days 0:11, 7 users, load average: 8.16, 6.54, 4.97 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 664756 370576 0 25200 103252 - -/+ buffers/cache: 536304 499028 Swap: 1253028 777676 475352 lenz@metis:~> cat /proc/meminfo MemTotal: 1035332 kB MemFree: 514212 kB Buffers: 18372 kB Cached: 67852 kB SwapCached: 100220 kB Active: 433092 kB Inactive: 39948 kB HighTotal: 130368 kB HighFree: 11964 kB LowTotal: 904964 kB LowFree: 502248 kB SwapTotal: 1253028 kB SwapFree: 476124 kB Dirty: 25136 kB Writeback: 620 kB AnonPages: 380552 kB Mapped: 44084 kB Slab: 24052 kB SReclaimable: 7976 kB SUnreclaim: 16076 kB PageTables: 4472 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 1770692 kB Committed_AS: 1731800 kB VmallocTotal: 114680 kB VmallocUsed: 24568 kB VmallocChunk: 88568 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 4096 kB lenz@metis:~> cat /proc/vmstat nr_free_pages 128481 nr_inactive 9272 nr_active 109029 nr_anon_pages 95354 nr_mapped 11057 nr_file_pages 46490 nr_dirty 5699 nr_writeback 159 nr_slab_reclaimable 1994 nr_slab_unreclaimable 4009 nr_page_table_pages 1118 nr_unstable 0 nr_bounce 0 nr_vmscan_write 1888108 pgpgin 96418508 pgpgout 45637844 pswpin 2774246 pswpout 1125860 pgalloc_dma 851875 pgalloc_normal 87607894 pgalloc_high 11340630 pgfree 99929041 pgactivate 17272483 pgdeactivate 16874995 pgfault 108709328 pgmajfault 1059067 pgrefill_dma 2592619 pgrefill_normal 380007871 pgrefill_high 128076759 pgsteal_dma 354571 pgsteal_normal 28255238 pgsteal_high 2627230 pgscan_kswapd_dma 706840 pgscan_kswapd_normal 38108000 pgscan_kswapd_high 4184311 pgscan_direct_dma 50950 pgscan_direct_normal 2046397 pgscan_direct_high 131039 pginodesteal 1971 slabs_scanned 16685952 kswapd_steal 29692461 kswapd_inodesteal 231462 pageoutrun 468658 allocstall 22220 pgrotated 1566095 lenz@metis:~> cat /proc/sys/vm/swappiness 0 At this point I terminated a number of running apps (openoffice, firefox, thunderbird), trying to regain control. However, the swapping activity remained high: lenz@metis:~> uptime 10:13am up 2 days 0:23, 4 users, load average: 8.65, 7.53, 6.21 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 352576 682756 0 6376 76444 - -/+ buffers/cache: 269756 765576 Swap: 1253028 350440 902588 lenz@metis:~> cat /proc/meminfo MemTotal: 1035332 kB MemFree: 681840 kB Buffers: 8584 kB Cached: 72404 kB SwapCached: 61036 kB Active: 292924 kB Inactive: 14960 kB HighTotal: 130368 kB HighFree: 260 kB LowTotal: 904964 kB LowFree: 681580 kB SwapTotal: 1253028 kB SwapFree: 902624 kB Dirty: 2628 kB Writeback: 448 kB AnonPages: 225440 kB Mapped: 54392 kB Slab: 23200 kB SReclaimable: 7364 kB SUnreclaim: 15836 kB PageTables: 3692 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 1770692 kB Committed_AS: 1088488 kB VmallocTotal: 114680 kB VmallocUsed: 24568 kB VmallocChunk: 88568 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 4096 kB lenz@metis:~> cat /proc/vmstat nr_free_pages 171792 nr_inactive 3542 nr_active 72030 nr_anon_pages 55568 nr_mapped 13418 nr_file_pages 35014 nr_dirty 432 nr_writeback 75 nr_slab_reclaimable 1841 nr_slab_unreclaimable 3964 nr_page_table_pages 915 nr_unstable 0 nr_bounce 0 nr_vmscan_write 1935933 pgpgin 97455380 pgpgout 45834820 pswpin 2850118 pswpout 1128092 pgalloc_dma 851875 pgalloc_normal 87812685 pgalloc_high 11548807 pgfree 100385240 pgactivate 17871993 pgdeactivate 17518347 pgfault 108993634 pgmajfault 1088574 pgrefill_dma 2593158 pgrefill_normal 403564383 pgrefill_high 136467021 pgsteal_dma 354572 pgsteal_normal 28388247 pgsteal_high 2763581 pgscan_kswapd_dma 707914 pgscan_kswapd_normal 38687360 pgscan_kswapd_high 4502615 pgscan_direct_dma 50950 pgscan_direct_normal 2046397 pgscan_direct_high 131039 pginodesteal 1971 slabs_scanned 16719872 kswapd_steal 29961822 kswapd_inodesteal 231462 pageoutrun 472969 allocstall 22220 pgrotated 1598360 lenz@metis:~> vmstat 3 procs -----------memory---------- ---swap-- -----io---- -system-- - ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 7 350404 688560 7412 68808 60 24 60 15 112 219 9 2 76 13 3 4 350404 687660 7760 69812 180 0 1237 679 371 875 5 5 0 89 1 6 350404 688420 7792 68956 119 0 1168 587 345 835 13 5 0 82 0 9 350404 689644 6860 69556 364 0 1372 159 351 988 6 4 0 91 0 6 350404 689148 6448 69876 271 0 1424 127 368 684 7 4 0 90 2 7 350404 689328 6336 70584 145 0 1275 39 313 651 5 4 0 91 0 4 350404 690612 6136 68820 457 0 1368 164 394 756 8 3 0 89 0 5 350404 689452 5872 70292 204 0 1599 35 331 590 7 2 0 91 2 4 350404 688496 6116 70448 296 0 1391 79 394 1780 11 4 0 86 2 4 350404 685608 8216 70716 309 0 2181 41 404 917 6 4 0 90 1 6 350404 683164 8868 71568 463 0 1389 567 377 1065 10 4 0 86 1 6 350404 682280 7948 71764 663 0 1579 729 445 861 9 5 0 86 0 8 350404 685556 6116 70716 324 0 1353 736 418 872 9 5 0 87 1 5 350360 684536 6124 71276 525 0 1273 97 367 772 11 2 0 87 0 8 350360 683780 5908 71736 499 0 1331 77 376 681 10 5 0 85 0 4 350300 682348 6052 73180 175 0 1652 67 335 617 6 5 0 89 0 5 350300 681396 5896 73252 517 0 1828 43 386 810 7 6 0 87 0 5 350268 681272 5992 72996 380 0 1423 136 349 634 7 3 0 90 1 4 342204 688196 5816 73300 207 0 1109 60 327 611 6 3 0 91 0 3 342204 690740 6024 70100 433 0 813 141 334 764 9 2 0 89 I am clueless and highly annoyed. What's wrong here? Bye, LenZ - -- - ------------------------------------------------------------------ Lenz Grimmer <lenz@grimmer.com> -o) [ICQ: 160767607 | Jabber: LenZGr@jabber.org] /\\ http://www.lenzg.org/ V_V -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHTWthSVDhKrJykfIRAl+vAJ9xaiWGL7aNCglmfWWFf9tGh+jTuQCfW7QW LDDCXdF0NyK2/k4z4WROYDE= =WQGx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Nov 28, 2007 at 02:21:39PM +0100, Lenz Grimmer wrote:
lenz@metis:~> free total used free shared buffers cached Mem: 1035332 664756 370576 0 25200 103252 - -/+ buffers/cache: 536304 499028 Swap: 1253028 777676 475352
370M free (so no real mem pressure). System isn't responsive because 700m is in swap and swapins are required to bring it in ram.
LowFree: 502248 kB
now free ram increase to 500M. So you must have closed some app taking >100M.
At this point I terminated a number of running apps (openoffice, firefox, thunderbird), trying to regain control. However, the swapping activity remained high:
lenz@metis:~> uptime 10:13am up 2 days 0:23, 4 users, load average: 8.65, 7.53, 6.21 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 352576 682756 0 6376 76444 - -/+ buffers/cache: 269756 765576 Swap: 1253028 350440 902588
free ram now increases to 680M and swap is reduced from 777 to 350M. so OO/firefox/thunderbid were taking around half a gig. You've still about 600M allocated, which seems too much for an empty desktop and only GUI + daemons running.
lenz@metis:~> vmstat 3 procs -----------memory---------- ---swap-- -----io---- -system-- - ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 7 350404 688560 7412 68808 60 24 60 15 112 219 9 2 76 13 3 4 350404 687660 7760 69812 180 0 1237 679 371 875 5 5 0 89 1 6 350404 688420 7792 68956 119 0 1168 587 345 835 13 5 0 82 0 9 350404 689644 6860 69556 364 0 1372 159 351 988 6 4 0 91 0 6 350404 689148 6448 69876 271 0 1424 127 368 684 7 4 0 90 2 7 350404 689328 6336 70584 145 0 1275 39 313 651 5 4 0 91 0 4 350404 690612 6136 68820 457 0 1368 164 394 756 8 3 0 89 0 5 350404 689452 5872 70292 204 0 1599 35 331 590 7 2 0 91 2 4 350404 688496 6116 70448 296 0 1391 79 394 1780 11 4 0 86 2 4 350404 685608 8216 70716 309 0 2181 41 404 917 6 4 0 90 1 6 350404 683164 8868 71568 463 0 1389 567 377 1065 10 4 0 86 1 6 350404 682280 7948 71764 663 0 1579 729 445 861 9 5 0 86 0 8 350404 685556 6116 70716 324 0 1353 736 418 872 9 5 0 87 1 5 350360 684536 6124 71276 525 0 1273 97 367 772 11 2 0 87 0 8 350360 683780 5908 71736 499 0 1331 77 376 681 10 5 0 85 0 4 350300 682348 6052 73180 175 0 1652 67 335 617 6 5 0 89 0 5 350300 681396 5896 73252 517 0 1828 43 386 810 7 6 0 87 0 5 350268 681272 5992 72996 380 0 1423 136 349 634 7 3 0 90 1 4 342204 688196 5816 73300 207 0 1109 60 327 611 6 3 0 91 0 3 342204 690740 6024 70100 433 0 813 141 334 764 9 2 0 89
see tons of ram free, only swapins. They're actually quite slow swapins, hope you're not swapping on a fragmented file in the filesystem but this may be normal.
I am clueless and highly annoyed. What's wrong here?
ps xau will tell you which is the app that is leaking memory, look for both RSS and VSZ going up (RSS will peak at some point when system starts swapping while VSZ will keep growing). firefox/thunderbird/OO cumulatively taking half a gig is probably expected, so they're probably not guilty. If nothing showup in "ps xau" look for `df /dev/shm` and `ipcs`. To verify it's not kernel related SYSRQ+E then after a few seconds SYSRQ+I will do the trick, then you can login in the console and verify that after that `free` tells ~950M free and ~0 swap. Hope this helps. Andrea -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (6)
-
Anders Johansson -
Andrea Arcangeli -
Jaimon Jose -
Lenz Grimmer -
Pavel Machek -
Rajko M.