[opensuse-kernel] VM swapping behaviour broken in 2.6.22.12-0.1-default?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
I am desperately looking for clues on a very strange behaviour that I
now have experienced with openSUSE 10.3 for several times now (actually,
every 2-3 days on my Laptop, which is my primary workstation and up 24x7).
The system works fine for 1-2 days, then all the sudden it becomes less
and less responsive due to massive swap activity. Looking at the output
of vmstat and free, I observe that the VM decided to swap out large
amounts of memory pages, while there still was plenty of RAM freely
available (not used for buffers or cache). So the system ends up with
over half of the RAM being marked as free, while the swap has filled up
quite significantly. This makes the system become *very* sluggish.
It happened to me several times now that I returned to my desk in the
morning and the system was completely unresponsive and the disk was
rattling as mad. Only a hard reboot helped to regain control (Since I
can't always wait ~1 hour for the Alt+SysRq+E to take effect).
I filed a bug report about this a few days ago, it would be great if
some kernel expert could investigate this and let me know, what other
information I need to provide.
https://bugzilla.novell.com/show_bug.cgi?id=340143
I toyed around with various swappiness values, but they don't seem to
make a difference. Interestingly, even setting /proc/sys/vm/swappiness
to "0" does not completely prohibit the system from swapping, while
there still is plenty of free memory available:
lenz@metis:~> uname -a
Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686
i686 i386 GNU/Linux
lenz@metis:~> uptime
8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63
lenz@metis:~> cat /proc/sys/vm/swappiness
0
lenz@metis:~> free
total used free shared buffers cached
Mem: 1035332 1014084 21248 0 87160 197488
- -/+ buffers/cache: 729436 305896
Swap: 1253028 131076 1121952
lenz@metis:~> cat /proc/vmstat
nr_free_pages 5250
nr_inactive 26913
nr_active 214996
nr_anon_pages 169683
nr_mapped 29874
nr_file_pages 79046
nr_dirty 721
nr_writeback 0
nr_slab_reclaimable 2394
nr_slab_unreclaimable 3436
nr_page_table_pages 965
nr_unstable 0
nr_bounce 0
nr_vmscan_write 135771
pgpgin 25483536
pgpgout 10607861
pswpin 14777
pswpout 36166
pgalloc_dma 169320
pgalloc_normal 29403345
pgalloc_high 2876515
pgfree 32454506
pgactivate 1380976
pgdeactivate 1156011
pgfault 35317571
pgmajfault 13960
pgrefill_dma 571625
pgrefill_normal 29974997
pgrefill_high 5063004
pgsteal_dma 91237
pgsteal_normal 7634076
pgsteal_high 333204
pgscan_kswapd_dma 191877
pgscan_kswapd_normal 7868615
pgscan_kswapd_high 430492
pgscan_direct_dma 3991
pgscan_direct_normal 289792
pgscan_direct_high 19388
pginodesteal 6
slabs_scanned 4068352
kswapd_steal 7819731
kswapd_inodesteal 17211
pageoutrun 132841
allocstall 3593
pgrotated 115105
Is there anything else you need? Any help would be appreciated - I don't
seem to be the only one experiencing this. I spotted one message on the
general openSUSE List, and there was a thread on this list about "Resume
take a long time to bring in the images to main memory" which sounded
like it may suffer from the same problem.
Thanks!
Bye,
LenZ
- --
- ------------------------------------------------------------------
Lenz Grimmer
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862 --jaimon Lenz Grimmer wrote, On 11/20/2007 01:54 AM:
Hi,
I am desperately looking for clues on a very strange behaviour that I now have experienced with openSUSE 10.3 for several times now (actually, every 2-3 days on my Laptop, which is my primary workstation and up 24x7).
The system works fine for 1-2 days, then all the sudden it becomes less and less responsive due to massive swap activity. Looking at the output of vmstat and free, I observe that the VM decided to swap out large amounts of memory pages, while there still was plenty of RAM freely available (not used for buffers or cache). So the system ends up with over half of the RAM being marked as free, while the swap has filled up quite significantly. This makes the system become *very* sluggish.
It happened to me several times now that I returned to my desk in the morning and the system was completely unresponsive and the disk was rattling as mad. Only a hard reboot helped to regain control (Since I can't always wait ~1 hour for the Alt+SysRq+E to take effect).
I filed a bug report about this a few days ago, it would be great if some kernel expert could investigate this and let me know, what other information I need to provide.
https://bugzilla.novell.com/show_bug.cgi?id=340143
I toyed around with various swappiness values, but they don't seem to make a difference. Interestingly, even setting /proc/sys/vm/swappiness to "0" does not completely prohibit the system from swapping, while there still is plenty of free memory available:
lenz@metis:~> uname -a Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686 i686 i386 GNU/Linux lenz@metis:~> uptime 8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1014084 21248 0 87160 197488 -/+ buffers/cache: 729436 305896 Swap: 1253028 131076 1121952 lenz@metis:~> cat /proc/vmstat nr_free_pages 5250 nr_inactive 26913 nr_active 214996 nr_anon_pages 169683 nr_mapped 29874 nr_file_pages 79046 nr_dirty 721 nr_writeback 0 nr_slab_reclaimable 2394 nr_slab_unreclaimable 3436 nr_page_table_pages 965 nr_unstable 0 nr_bounce 0 nr_vmscan_write 135771 pgpgin 25483536 pgpgout 10607861 pswpin 14777 pswpout 36166 pgalloc_dma 169320 pgalloc_normal 29403345 pgalloc_high 2876515 pgfree 32454506 pgactivate 1380976 pgdeactivate 1156011 pgfault 35317571 pgmajfault 13960 pgrefill_dma 571625 pgrefill_normal 29974997 pgrefill_high 5063004 pgsteal_dma 91237 pgsteal_normal 7634076 pgsteal_high 333204 pgscan_kswapd_dma 191877 pgscan_kswapd_normal 7868615 pgscan_kswapd_high 430492 pgscan_direct_dma 3991 pgscan_direct_normal 289792 pgscan_direct_high 19388 pginodesteal 6 slabs_scanned 4068352 kswapd_steal 7819731 kswapd_inodesteal 17211 pageoutrun 132841 allocstall 3593 pgrotated 115105
Is there anything else you need? Any help would be appreciated - I don't seem to be the only one experiencing this. I spotted one message on the general openSUSE List, and there was a thread on this list about "Resume take a long time to bring in the images to main memory" which sounded like it may suffer from the same problem.
Thanks!
Bye, LenZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jaimon, thanks for your message! Jaimon Jose wrote:
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862
Nope, I even disabled DRI in my X Server configuration some time ago (as
it's still very unstable when using the opensource radeon driver).
I really suspect a VM bug here. Why is it using swap even when there is
no memory pressure and swappiness is set to zero?
lenz@metis:~> uptime
10:26am up 1 day 0:21, 8 users, load average: 0.40, 0.73, 0.98
lenz@metis:~> cat /proc/sys/vm/swappiness
0
lenz@metis:~> free
total used free shared buffers cached
Mem: 1035332 1020224 15108 0 150936 222276
- -/+ buffers/cache: 647012 388320
Swap: 1253028 349804 903224
Note that this is just about 2:30 hours after I did a "swapoff -a ;
swapon -a" this morning (in the attempt to move the swapped out pages
back into memory).
Bye,
LenZ
- --
- ------------------------------------------------------------------
Lenz Grimmer
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jaimon, Lenz Grimmer wrote:
I really suspect a VM bug here. Why is it using swap even when there is no memory pressure and swappiness is set to zero?
lenz@metis:~> uptime 10:26am up 1 day 0:21, 8 users, load average: 0.40, 0.73, 0.98 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1020224 15108 0 150936 222276 -/+ buffers/cache: 647012 388320 Swap: 1253028 349804 903224
Note that this is just about 2:30 hours after I did a "swapoff -a ; swapon -a" this morning (in the attempt to move the swapped out pages back into memory).
The system again started to swap heavily and more and more memory
appeared as "free" during the course of the day. I was able to regain
control by termination several apps (Firefox, OOo, Thunderbird) and
disabling swap temporarily (to force the VM to swap in all pages again),
but this is becoming a major annoyance. I wonder if I am the only one
experiencing this?
Any idea why it ignores the swappiness setting? Or why the VM decides to
start swapping in the first place, if there is plenty of free memory?
I'd be happy to assist you in finding the cause of this, just let me
know what other info you would need or what info to watch. Thanks!
Bye,
LenZ
- --
- ------------------------------------------------------------------
Lenz Grimmer
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit. I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it Anders -- Madness takes its toll -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be. I'm now in 32 bit installation (box is Athlon 64) and swap is not used. One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thursday 22 November 2007 02:09:39 pm Rajko M. wrote:
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be.
I'm now in 32 bit installation (box is Athlon 64) and swap is not used.
One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works.
1) ~> free total used free shared buffers cached Mem: 905804 884992 20812 0 18116 651424 -/+ buffers/cache: 215452 690352 Swap: 2104472 420 2104052 2) ~> free total used free shared buffers cached Mem: 905804 825180 80624 0 76312 523004 -/+ buffers/cache: 225864 679940 Swap: 2104472 86576 2017896 System was not used between 2 outputs for hours, though, it was probably running daily cron jobs. Currently (after another few hours) when system was used: 3) ~> free total used free shared buffers cached Mem: 905804 893640 12164 0 77552 540008 -/+ buffers/cache: 276080 629724 Swap: 2104472 85800 2018672 The /proc/meminfo shows different for 1) and 2) MemFree:71220 kB Some 400 MB was used by virtual machine, so 'free' report seems way out of line. It is OK for the rest of the system, but I can't see this 400 MB as used. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Friday 23 November 2007 13:22:22 Rajko M. wrote:
On Thursday 22 November 2007 02:09:39 pm Rajko M. wrote:
On Thursday 22 November 2007 12:25:43 pm Anders Johansson wrote:
On Thursday 22 November 2007 03:11:54 Rajko M. wrote:
On Wednesday 21 November 2007 10:56:13 am Lenz Grimmer wrote:
I wonder if I am the only one experiencing this?
No. I didn't tried to debug it as I switch between installations often enough, but I'll post 'free' output when it happens.
/proc/meminfo would probably be more interesting
The systems posted so far have been 32 bit systems. I wonder if this happens on 64 bit.
I have seen issues with systems running out of lowmem memory. Maybe this has something to do with it
It could be.
I'm now in 32 bit installation (box is Athlon 64) and swap is not used.
One of reasons to switch to 32 bit was that 1 GB (-128 MB video) is tight with 64 bit running virtual machine with 400 MB and system was swapping and usage was growing fast. Now I don't see any, but I'm watching how it works.
1) ~> free total used free shared buffers cached Mem: 905804 884992 20812 0 18116 651424 -/+ buffers/cache: 215452 690352 Swap: 2104472 420 2104052
2) ~> free total used free shared buffers cached Mem: 905804 825180 80624 0 76312 523004 -/+ buffers/cache: 225864 679940 Swap: 2104472 86576 2017896
System was not used between 2 outputs for hours, though, it was probably running daily cron jobs.
Currently (after another few hours) when system was used:
3) ~> free total used free shared buffers cached Mem: 905804 893640 12164 0 77552 540008 -/+ buffers/cache: 276080 629724 Swap: 2104472 85800 2018672
The /proc/meminfo shows different for 1) and 2) MemFree:71220 kB Some 400 MB was used by virtual machine, so 'free' report seems way out of line. It is OK for the rest of the system, but I can't see this 400 MB as used.
The interesting part from meminfo is LowFree. If that gets very very low, the memory pressure will be high on the machine regardless of how much highmem is free Anders -- Madness takes its toll -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Hi!
Do you have compiz enabled? Here is another defect for the same problem. https://bugzilla.novell.com/show_bug.cgi?id=335862
And I have something s2ram-related reported, too... Are you sure userspace is not responsible? Reproducing it w/o X would be nice... Pavel
Lenz Grimmer wrote, On 11/20/2007 01:54 AM:
Hi,
I am desperately looking for clues on a very strange behaviour that I now have experienced with openSUSE 10.3 for several times now (actually, every 2-3 days on my Laptop, which is my primary workstation and up 24x7).
The system works fine for 1-2 days, then all the sudden it becomes less and less responsive due to massive swap activity. Looking at the output of vmstat and free, I observe that the VM decided to swap out large amounts of memory pages, while there still was plenty of RAM freely available (not used for buffers or cache). So the system ends up with over half of the RAM being marked as free, while the swap has filled up quite significantly. This makes the system become *very* sluggish.
It happened to me several times now that I returned to my desk in the morning and the system was completely unresponsive and the disk was rattling as mad. Only a hard reboot helped to regain control (Since I can't always wait ~1 hour for the Alt+SysRq+E to take effect).
I filed a bug report about this a few days ago, it would be great if some kernel expert could investigate this and let me know, what other information I need to provide.
https://bugzilla.novell.com/show_bug.cgi?id=340143
I toyed around with various swappiness values, but they don't seem to make a difference. Interestingly, even setting /proc/sys/vm/swappiness to "0" does not completely prohibit the system from swapping, while there still is plenty of free memory available:
lenz@metis:~> uname -a Linux metis 2.6.22.12-0.1-default #1 SMP 2007/11/06 23:05:18 UTC i686 i686 i386 GNU/Linux lenz@metis:~> uptime 8:59pm up 10:54, 7 users, load average: 0.58, 0.61, 0.63 lenz@metis:~> cat /proc/sys/vm/swappiness 0 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 1014084 21248 0 87160 197488 -/+ buffers/cache: 729436 305896 Swap: 1253028 131076 1121952 lenz@metis:~> cat /proc/vmstat nr_free_pages 5250 nr_inactive 26913 nr_active 214996 nr_anon_pages 169683 nr_mapped 29874 nr_file_pages 79046 nr_dirty 721 nr_writeback 0 nr_slab_reclaimable 2394 nr_slab_unreclaimable 3436 nr_page_table_pages 965 nr_unstable 0 nr_bounce 0 nr_vmscan_write 135771 pgpgin 25483536 pgpgout 10607861 pswpin 14777 pswpout 36166 pgalloc_dma 169320 pgalloc_normal 29403345 pgalloc_high 2876515 pgfree 32454506 pgactivate 1380976 pgdeactivate 1156011 pgfault 35317571 pgmajfault 13960 pgrefill_dma 571625 pgrefill_normal 29974997 pgrefill_high 5063004 pgsteal_dma 91237 pgsteal_normal 7634076 pgsteal_high 333204 pgscan_kswapd_dma 191877 pgscan_kswapd_normal 7868615 pgscan_kswapd_high 430492 pgscan_direct_dma 3991 pgscan_direct_normal 289792 pgscan_direct_high 19388 pginodesteal 6 slabs_scanned 4068352 kswapd_steal 7819731 kswapd_inodesteal 17211 pageoutrun 132841 allocstall 3593 pgrotated 115105
Is there anything else you need? Any help would be appreciated - I don't seem to be the only one experiencing this. I spotted one message on the general openSUSE List, and there was a thread on this list about "Resume take a long time to bring in the images to main memory" which sounded like it may suffer from the same problem.
Thanks!
Bye, LenZ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Pavel, thanks for your reply! Pavel Machek wrote:
Are you sure userspace is not responsible?
Of course I am not, all I can say is that this is a change in behaviour from 10.2 to 10.3, while I did not change my way of working with this system as a Desktop machine (e.g. which applications I keep running)...
Reproducing it w/o X would be nice...
Unfortunately this is something I am not able to test. I need this box
as my primary workstation and not being able to use X would severely
hamper this purpose :(
Would it help to also give you a list of all running processes and their
memory usage, next time it happens? Just let me know which output of
which command would be helpful.
Somebody informed me that he also noticed similar reports on the LKML. I
asked him if he could provide pointers, which I will add to the bug report.
Bye,
LenZ
- --
- ------------------------------------------------------------------
Lenz Grimmer
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Replying to myself) Lenz Grimmer wrote:
Of course I am not, all I can say is that this is a change in behaviour from 10.2 to 10.3, while I did not change my way of working with this system as a Desktop machine (e.g. which applications I keep running)...
It happened again this morning - returning to my desk which I left 6
hours before in a working state, the system again was very unresponsive
and the hard disk was rattling as mad.
I managed to capture some stats before rebooting it:
lenz@metis:~> date
Wed Nov 28 10:00:59 CET 2007
lenz@metis:~> uptime
10:01am up 2 days 0:11, 7 users, load average: 8.16, 6.54, 4.97
lenz@metis:~> free
total used free shared buffers cached
Mem: 1035332 664756 370576 0 25200 103252
- -/+ buffers/cache: 536304 499028
Swap: 1253028 777676 475352
lenz@metis:~> cat /proc/meminfo
MemTotal: 1035332 kB
MemFree: 514212 kB
Buffers: 18372 kB
Cached: 67852 kB
SwapCached: 100220 kB
Active: 433092 kB
Inactive: 39948 kB
HighTotal: 130368 kB
HighFree: 11964 kB
LowTotal: 904964 kB
LowFree: 502248 kB
SwapTotal: 1253028 kB
SwapFree: 476124 kB
Dirty: 25136 kB
Writeback: 620 kB
AnonPages: 380552 kB
Mapped: 44084 kB
Slab: 24052 kB
SReclaimable: 7976 kB
SUnreclaim: 16076 kB
PageTables: 4472 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1770692 kB
Committed_AS: 1731800 kB
VmallocTotal: 114680 kB
VmallocUsed: 24568 kB
VmallocChunk: 88568 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 4096 kB
lenz@metis:~> cat /proc/vmstat
nr_free_pages 128481
nr_inactive 9272
nr_active 109029
nr_anon_pages 95354
nr_mapped 11057
nr_file_pages 46490
nr_dirty 5699
nr_writeback 159
nr_slab_reclaimable 1994
nr_slab_unreclaimable 4009
nr_page_table_pages 1118
nr_unstable 0
nr_bounce 0
nr_vmscan_write 1888108
pgpgin 96418508
pgpgout 45637844
pswpin 2774246
pswpout 1125860
pgalloc_dma 851875
pgalloc_normal 87607894
pgalloc_high 11340630
pgfree 99929041
pgactivate 17272483
pgdeactivate 16874995
pgfault 108709328
pgmajfault 1059067
pgrefill_dma 2592619
pgrefill_normal 380007871
pgrefill_high 128076759
pgsteal_dma 354571
pgsteal_normal 28255238
pgsteal_high 2627230
pgscan_kswapd_dma 706840
pgscan_kswapd_normal 38108000
pgscan_kswapd_high 4184311
pgscan_direct_dma 50950
pgscan_direct_normal 2046397
pgscan_direct_high 131039
pginodesteal 1971
slabs_scanned 16685952
kswapd_steal 29692461
kswapd_inodesteal 231462
pageoutrun 468658
allocstall 22220
pgrotated 1566095
lenz@metis:~> cat /proc/sys/vm/swappiness
0
At this point I terminated a number of running apps (openoffice,
firefox, thunderbird), trying to regain control. However, the swapping
activity remained high:
lenz@metis:~> uptime
10:13am up 2 days 0:23, 4 users, load average: 8.65, 7.53, 6.21
lenz@metis:~> free
total used free shared buffers cached
Mem: 1035332 352576 682756 0 6376 76444
- -/+ buffers/cache: 269756 765576
Swap: 1253028 350440 902588
lenz@metis:~> cat /proc/meminfo
MemTotal: 1035332 kB
MemFree: 681840 kB
Buffers: 8584 kB
Cached: 72404 kB
SwapCached: 61036 kB
Active: 292924 kB
Inactive: 14960 kB
HighTotal: 130368 kB
HighFree: 260 kB
LowTotal: 904964 kB
LowFree: 681580 kB
SwapTotal: 1253028 kB
SwapFree: 902624 kB
Dirty: 2628 kB
Writeback: 448 kB
AnonPages: 225440 kB
Mapped: 54392 kB
Slab: 23200 kB
SReclaimable: 7364 kB
SUnreclaim: 15836 kB
PageTables: 3692 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1770692 kB
Committed_AS: 1088488 kB
VmallocTotal: 114680 kB
VmallocUsed: 24568 kB
VmallocChunk: 88568 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 4096 kB
lenz@metis:~> cat /proc/vmstat
nr_free_pages 171792
nr_inactive 3542
nr_active 72030
nr_anon_pages 55568
nr_mapped 13418
nr_file_pages 35014
nr_dirty 432
nr_writeback 75
nr_slab_reclaimable 1841
nr_slab_unreclaimable 3964
nr_page_table_pages 915
nr_unstable 0
nr_bounce 0
nr_vmscan_write 1935933
pgpgin 97455380
pgpgout 45834820
pswpin 2850118
pswpout 1128092
pgalloc_dma 851875
pgalloc_normal 87812685
pgalloc_high 11548807
pgfree 100385240
pgactivate 17871993
pgdeactivate 17518347
pgfault 108993634
pgmajfault 1088574
pgrefill_dma 2593158
pgrefill_normal 403564383
pgrefill_high 136467021
pgsteal_dma 354572
pgsteal_normal 28388247
pgsteal_high 2763581
pgscan_kswapd_dma 707914
pgscan_kswapd_normal 38687360
pgscan_kswapd_high 4502615
pgscan_direct_dma 50950
pgscan_direct_normal 2046397
pgscan_direct_high 131039
pginodesteal 1971
slabs_scanned 16719872
kswapd_steal 29961822
kswapd_inodesteal 231462
pageoutrun 472969
allocstall 22220
pgrotated 1598360
lenz@metis:~> vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system--
- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 7 350404 688560 7412 68808 60 24 60 15 112 219 9 2
76 13
3 4 350404 687660 7760 69812 180 0 1237 679 371 875 5 5
0 89
1 6 350404 688420 7792 68956 119 0 1168 587 345 835 13 5
0 82
0 9 350404 689644 6860 69556 364 0 1372 159 351 988 6 4
0 91
0 6 350404 689148 6448 69876 271 0 1424 127 368 684 7 4
0 90
2 7 350404 689328 6336 70584 145 0 1275 39 313 651 5 4
0 91
0 4 350404 690612 6136 68820 457 0 1368 164 394 756 8 3
0 89
0 5 350404 689452 5872 70292 204 0 1599 35 331 590 7 2
0 91
2 4 350404 688496 6116 70448 296 0 1391 79 394 1780 11 4
0 86
2 4 350404 685608 8216 70716 309 0 2181 41 404 917 6 4
0 90
1 6 350404 683164 8868 71568 463 0 1389 567 377 1065 10 4
0 86
1 6 350404 682280 7948 71764 663 0 1579 729 445 861 9 5
0 86
0 8 350404 685556 6116 70716 324 0 1353 736 418 872 9 5
0 87
1 5 350360 684536 6124 71276 525 0 1273 97 367 772 11 2
0 87
0 8 350360 683780 5908 71736 499 0 1331 77 376 681 10 5
0 85
0 4 350300 682348 6052 73180 175 0 1652 67 335 617 6 5
0 89
0 5 350300 681396 5896 73252 517 0 1828 43 386 810 7 6
0 87
0 5 350268 681272 5992 72996 380 0 1423 136 349 634 7 3
0 90
1 4 342204 688196 5816 73300 207 0 1109 60 327 611 6 3
0 91
0 3 342204 690740 6024 70100 433 0 813 141 334 764 9 2
0 89
I am clueless and highly annoyed. What's wrong here?
Bye,
LenZ
- --
- ------------------------------------------------------------------
Lenz Grimmer
On Wed, Nov 28, 2007 at 02:21:39PM +0100, Lenz Grimmer wrote:
lenz@metis:~> free total used free shared buffers cached Mem: 1035332 664756 370576 0 25200 103252 - -/+ buffers/cache: 536304 499028 Swap: 1253028 777676 475352
370M free (so no real mem pressure). System isn't responsive because 700m is in swap and swapins are required to bring it in ram.
LowFree: 502248 kB
now free ram increase to 500M. So you must have closed some app taking >100M.
At this point I terminated a number of running apps (openoffice, firefox, thunderbird), trying to regain control. However, the swapping activity remained high:
lenz@metis:~> uptime 10:13am up 2 days 0:23, 4 users, load average: 8.65, 7.53, 6.21 lenz@metis:~> free total used free shared buffers cached Mem: 1035332 352576 682756 0 6376 76444 - -/+ buffers/cache: 269756 765576 Swap: 1253028 350440 902588
free ram now increases to 680M and swap is reduced from 777 to 350M. so OO/firefox/thunderbid were taking around half a gig. You've still about 600M allocated, which seems too much for an empty desktop and only GUI + daemons running.
lenz@metis:~> vmstat 3 procs -----------memory---------- ---swap-- -----io---- -system-- - ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 7 350404 688560 7412 68808 60 24 60 15 112 219 9 2 76 13 3 4 350404 687660 7760 69812 180 0 1237 679 371 875 5 5 0 89 1 6 350404 688420 7792 68956 119 0 1168 587 345 835 13 5 0 82 0 9 350404 689644 6860 69556 364 0 1372 159 351 988 6 4 0 91 0 6 350404 689148 6448 69876 271 0 1424 127 368 684 7 4 0 90 2 7 350404 689328 6336 70584 145 0 1275 39 313 651 5 4 0 91 0 4 350404 690612 6136 68820 457 0 1368 164 394 756 8 3 0 89 0 5 350404 689452 5872 70292 204 0 1599 35 331 590 7 2 0 91 2 4 350404 688496 6116 70448 296 0 1391 79 394 1780 11 4 0 86 2 4 350404 685608 8216 70716 309 0 2181 41 404 917 6 4 0 90 1 6 350404 683164 8868 71568 463 0 1389 567 377 1065 10 4 0 86 1 6 350404 682280 7948 71764 663 0 1579 729 445 861 9 5 0 86 0 8 350404 685556 6116 70716 324 0 1353 736 418 872 9 5 0 87 1 5 350360 684536 6124 71276 525 0 1273 97 367 772 11 2 0 87 0 8 350360 683780 5908 71736 499 0 1331 77 376 681 10 5 0 85 0 4 350300 682348 6052 73180 175 0 1652 67 335 617 6 5 0 89 0 5 350300 681396 5896 73252 517 0 1828 43 386 810 7 6 0 87 0 5 350268 681272 5992 72996 380 0 1423 136 349 634 7 3 0 90 1 4 342204 688196 5816 73300 207 0 1109 60 327 611 6 3 0 91 0 3 342204 690740 6024 70100 433 0 813 141 334 764 9 2 0 89
see tons of ram free, only swapins. They're actually quite slow swapins, hope you're not swapping on a fragmented file in the filesystem but this may be normal.
I am clueless and highly annoyed. What's wrong here?
ps xau will tell you which is the app that is leaking memory, look for both RSS and VSZ going up (RSS will peak at some point when system starts swapping while VSZ will keep growing). firefox/thunderbird/OO cumulatively taking half a gig is probably expected, so they're probably not guilty. If nothing showup in "ps xau" look for `df /dev/shm` and `ipcs`. To verify it's not kernel related SYSRQ+E then after a few seconds SYSRQ+I will do the trick, then you can login in the console and verify that after that `free` tells ~950M free and ~0 swap. Hope this helps. Andrea -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (6)
-
Anders Johansson -
Andrea Arcangeli -
Jaimon Jose -
Lenz Grimmer -
Pavel Machek -
Rajko M.