[opensuse] Question about the correct way to monitor memory and advise
Guys, I have a question about memory, where I work we have been having heated debates about how to monitor memory in Linux. I am tasked to write a bash script that will alert me when our server are pegged using memory. I have been doing this to take a look at how much memory has been used. free -o -m total used free shared buffers cached Mem: 1134 1094 39 0 299 325 Swap: 2288 0 2288 I have been taking the total - cached = total memory used. I have also been looking at /proc/meminfo to make sure it matches, but from what I understand that's not the correct way to do it. I know that the three tools to use are free, top, vmstat. What is the best way to see what memory is being used. I found this script but every time it check it sends an alert that memory is over 90%, and that not good to get a lot of false alarms. !/bin/sh free -m | grep Mem | while read output; do mem=$(echo $output | awk '{print $2}' ) used=$(echo $output | awk '{ print $3 }' ) freed=$(echo $output | awk '{ print $4 }' ) echo "Swap : $mem" echo "Used : $used" echo "Free : $freed" usep=`expr $used \* 100 / $mem` echo $usep if [ $usep -ge 70 ]; then echo "Memory Usage Alert Total Memory: \"$mem\" Used: \"$used ($usep%)\" Free: \"$freed\" on $(hostname -f) as on $(date)" | mail -s "Alert: Memory Usage space $usep%" terrorpup@gmail.com fi done Does any one have a good script I can use, or can advise me on what is best practise for monitor memory. Thanks, Payne -- ---------------------------------------- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- Robert Heinlein -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Saturday 28 March 2009 22:48:53 Chuck Payne wrote:
Guys,
I have a question about memory, where I work we have been having heated debates about how to monitor memory in Linux. I am tasked to write a bash script that will alert me when our server are pegged using memory.
I have been doing this to take a look at how much memory has been used.
free -o -m total used free shared buffers cached Mem: 1134 1094 39 0 299 325 Swap: 2288 0 2288
I have been taking the total - cached = total memory used.
Almost. It's total - cached - buffers - free = memory used (or a close approximation of it). If you don't use the -o switch, you get it pre-calculated
free total used free shared buffers cached Mem: 6115016 6082736 32280 0 98812 4404988 -/+ buffers/cache: 1578936 4536080 Swap: 2096472 140 2096332
So on this machine I have used about 1.5GB (first number on second line) out of a total of 6 (first number on first line), and the free memory is roughly 4.5GB (second number on second line)
I found this script but every time it check it sends an alert that memory is over 90%, and that not good to get a lot of false alarms.
Yes, the script doesn't take into account the fact that linux uses unused memory for buffers and cache. This memory is not included in the "free" value, because it technically isn't (it's used internally by the kernel), but it is available, because as soon as an application needs it, the buffers get flushed and caches dropped - at least enough to accommodate the application.
Does any one have a good script I can use, or can advise me on what is best practise for monitor memory.
There are tools for monitoring systems that were written by people who understand linux memory management. If you want to write your own script, you can just monitor for when the "free" value on the "-/+ buffers/cache" line gets too low, but I would recommend using one of the existing monitoring tools. For example, have a look at nagios Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 28 Mar, 2009 at 23:11:52 +0100, Anders Johansson wrote:
On Saturday 28 March 2009 22:48:53 Chuck Payne wrote:
Does any one have a good script I can use, or can advise me on what is best practise for monitor memory.
In general I'd say that 'free' vs. 'used' memory is not nearly as important as whether or not the system is swapping. I'm certainly no expert on linux memory management, but AFAIK the overall strategy concerning RAM usage is that: Memory is there to be used. Since it's always a possibility that 'something' in memory is going to be needed 'next', there's no reason to evacuate it until such a time when 'something else' needs the space. So what it comes down to is; If there's enough physical RAM to keep all relevent 'somethings' in memory, then everything is fine regardless of how many percent it takes up. If not, then the system has to swap between physical memory and swapspace on disk, which carries a performance penalty.
There are tools for monitoring systems that were written by people who understand linux memory management. If you want to write your own script, you can just monitor for when the "free" value on the "-/+ buffers/cache" line gets too low, but I would recommend using one of the existing monitoring tools. For example, have a look at nagios
While Nagios can be somewhat intimidating when you first lay eyes on it, it's a very good monitoring system, and definitely worth looking in to. It may be overkill for the imediate task at hand, but once you get the hang of it you might want to monitor other stuff too. In any event you might save yourself some scripting/research time by installing nagios-plugins and just call the pertinent plugins directly: jon@zd8k110:~> /usr/lib/nagios/plugins/check_swap -w 95 -c 90 SWAP OK - 100% free (1027 MB out of 1027 MB) |swap=1027MB;0;0;0;1027 /jon -- YMMV -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sunday 29 March 2009 10:34:41 Jon Clausen wrote:
On Sat, 28 Mar, 2009 at 23:11:52 +0100, Anders Johansson wrote:
On Saturday 28 March 2009 22:48:53 Chuck Payne wrote:
Does any one have a good script I can use, or can advise me on what is best practise for monitor memory.
In general I'd say that 'free' vs. 'used' memory is not nearly as important as whether or not the system is swapping.
Good point. Just remember that what you're looking for is not whether or not swap is used at all, since linux will swap out areas of memory that haven't been used for a long time, to make room for more buffers and cache. This is normally a good thing which leads to improved performance. It's only a problem if the system starts running out of physical RAM so it needs to swap out areas of memory which actually are used actively. This is the first step on the road to thrashing, which can kill your system. This only happens when the "free" value runs low though, which is why it is a good idea to monitor it. Especially since you want to be warned before the system runs into trouble, not when it's already too late.
If there's enough physical RAM to keep all relevent 'somethings' in memory, then everything is fine regardless of how many percent it takes up. If not, then the system has to swap between physical memory and swapspace on disk, which carries a performance penalty.
Actually, I'd say that if you have X amount of RAM in the system, and you have applications running that take up X amount of RAM, then you wouldn't have a performant system. Sure, it wouldn't be swapping, but there would be absolutely no disk caching or buffering, and this would have a pretty big impact, I'd bet. Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sun, Mar 29, 2009 at 5:15 AM, Anders Johansson
On Sunday 29 March 2009 10:34:41 Jon Clausen wrote:
On Sat, 28 Mar, 2009 at 23:11:52 +0100, Anders Johansson wrote:
On Saturday 28 March 2009 22:48:53 Chuck Payne wrote:
Does any one have a good script I can use, or can advise me on what is best practise for monitor memory.
In general I'd say that 'free' vs. 'used' memory is not nearly as important as whether or not the system is swapping.
Good point.
Just remember that what you're looking for is not whether or not swap is used at all, since linux will swap out areas of memory that haven't been used for a long time, to make room for more buffers and cache. This is normally a good thing which leads to improved performance.
It's only a problem if the system starts running out of physical RAM so it needs to swap out areas of memory which actually are used actively. This is the first step on the road to thrashing, which can kill your system.
This only happens when the "free" value runs low though, which is why it is a good idea to monitor it. Especially since you want to be warned before the system runs into trouble, not when it's already too late.
If there's enough physical RAM to keep all relevent 'somethings' in memory, then everything is fine regardless of how many percent it takes up. If not, then the system has to swap between physical memory and swapspace on disk, which carries a performance penalty.
Actually, I'd say that if you have X amount of RAM in the system, and you have applications running that take up X amount of RAM, then you wouldn't have a performant system. Sure, it wouldn't be swapping, but there would be absolutely no disk caching or buffering, and this would have a pretty big impact, I'd bet.
Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Thanks guys, these are all good points. The reason I was asking we use a program from HP call SiteScope, not as good as Nagios in my opinion. SiteScope is set up out of the box to check swap, like you guys have talked about. Well we had a meeting, to talk about memory usage. When I had brought that how I measure memory use, an my three tools, top, vmstat, and free. I was told but my co-workers ( both think they know linux because they have worked on AIX and Solaris) that I was wrong. The funny thing, the windows admin who has so clue about Unix in general was backing me, because the statement I has posted early made sense. The reason that I has ask about script to measure memory. The AIX guy strip out the script that measure if a box had gone to swap and put into place a script like the one I posted, and now our blackberries are going off every time SiteScope does a memory test. I have nagios on my home box, I am looking at the the scripts on there. I have also found a couple more on the web that I am taking a look at well. Payne -- ---------------------------------------- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- Robert Heinlein -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (3)
-
Anders Johansson -
Chuck Payne -
Jon Clausen