[opensuse] Strange error with disc space
Guys, I have a question. Have you ever seen disc space just disappear and you were unable to find what is using? One of our DB came to me the other day pointed out that one of his rac boxes has filled up 2.0G 1.7G 214M 89% / 2.0G 88M 1.8G 5% /tmp 1023M 298M 673M 31% /opt 2.0G 203M 1.7G 11% /home 2.0G 297M 1.6G 16% /archive 2.0G 456M 1.4G 25% /var 4.9G 2.7G 2.0G 59% /usr 2.0G 68M 1.8G 4% /usr/local 40G 12G 26G 32% /u00 122M 32M 84M 28% /boot 7.9G 0 7.9G 0% /dev/shm 4.0G 118M 3.9G 3% /crs_db so I started looking into, but I can't find any file or dir on the root partition that eat space. I have looked at dmesg and don't see anything. Oh, we have rebooted the box and the space didn't clear up. Just an fyi, here are things I did look, please feel free to let me know what other things I need to check. alias ducks='du -chs * | sort -rn | head -11' find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -nrk 2,2 lsof -a +L1 -- ---------------------------------------- Old utilities do not die they just slowly fade away! ----------------------------------------- OpenSUSE -- http://en.opensuse.org/User:Terrorpup OpenSuSE Ambassador OpenSuSE Member Skype -- terrorpup twitter -- terrorpup -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Chuck Payne wrote:
alias ducks='du -chs * | sort -rn | head -11'
That won't give you a good list because the -h means the numbers aren't in numerical order (because they have k, M suffixes). It will also list other filesystems. Try du -xs * | sort -rn Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Chuck Payne a écrit :
Guys,
I have a question. Have you ever seen disc space just disappear and you were unable to find what is using?
look at mount points. May be you had a failure in mouting and so some files are hidden under the mount point also kde gives room for trash and thumbnails on "." files jdd -- http://www.dodin.net http://valerie.dodin.org http://news.opensuse.org/2009/04/13/people-of-opensuse-jean-daniel-dodin/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, Jul 30, 2009 at 9:14 AM, jdd (kim2)<jdd@dodin.org> wrote:
Chuck Payne a écrit :
Guys,
I have a question. Have you ever seen disc space just disappear and you were unable to find what is using?
look at mount points. May be you had a failure in mouting and so some files are hidden under the mount point
also kde gives room for trash and thumbnails on "." files
jdd
-- http://www.dodin.net http://valerie.dodin.org http://news.opensuse.org/2009/04/13/people-of-opensuse-jean-daniel-dodin/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sorry I hit send before finishing...but it only shows / using 474M. We don't have X install on the boxes because they are servers, they are just CLI. I am starting to think is a Oracle RAC issue. Because now we have another box doing this, and both boxes are Node 1 of the RAC Build. -- ---------------------------------------- Old utilities do not die they just slowly fade away! ----------------------------------------- OpenSUSE -- http://en.opensuse.org/User:Terrorpup OpenSuSE Ambassador OpenSuSE Member Skype -- terrorpup twitter -- terrorpup -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 30 July 2009 08:37:44 am Chuck Payne wrote:
On Thu, Jul 30, 2009 at 9:14 AM, jdd (kim2)<jdd@dodin.org> wrote:
Chuck Payne a écrit :
Guys,
I have a question. Have you ever seen disc space just disappear and you were unable to find what is using?
look at mount points. May be you had a failure in mouting and so some files are hidden under the mount point
also kde gives room for trash and thumbnails on "." files
jdd
Chuck, Be sure to double-check what jdd said. If in the past there was a failed mount on one of your mount points and then you did a backup or something that wrote files to the unmounted mount point you may very well have filled up the partition that contains the mount point --and-- now after a successful mount on that mount point you can't tell that there may be another bizillion files there -- and -- since something is now mounted on the mount point, all you are getting is the information about what the current mount holds, not what may exist in the unmounted state on the mount point itself. (clear as mud yet?) I had about 8 Gigs of stuff do that once when I had a failed cifs mount of /mnt/office. I then backed up all of the files from another machine to /mnt/office (recursively). The conflict preventing the mount was fixed in another unrelated credential file fix, and when the next backup ran with /mnt/office properly mounted I ended up with *double* the space being used. It isn't immediately apparent that a problem has occurred because when you look at the files on the mount point, you see what you think should be there. The problem is that you may have *DOUBLE* the information you think you have. (a full set of files on the unmounted mount point *and* another duplicate set of information on the share or partitions that gets mounted on mount point) The original files are now suppressed under what actually mounted there. The only real way to check is to umount each of the shares or partitions and then check for any files present on each of the mount points in the unmounted state. Confirm what is or isn't mounted with mtab when you are testing. A simple ls or ducks will do for discovering any suppressed files once everything is unmounted. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
In <4A719CC3.7050201@dodin.org>, jdd (kim2) wrote:
Chuck Payne a écrit :
I have a question. Have you ever seen disc space just disappear and you were unable to find what is using?
look at mount points. May be you had a failure in mouting and so some files are hidden under the mount point
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt Then, you can use whatever tools you'd normally use to look for files taking up space, such as: du -ax /mnt | sort -rn /tmp/fs_report && head -n $((LINES - 2)) /tmp/fs_report | cut -c "-$COLUMNS" Finally, you may want to use lsof etc. to look for "phantom" files. When an open file is deleted from a Linux/UNIX file system, it continues to take up space in the file system and can even grow but won't be visible in directory listings until the last process using it has closed it, at which point all the space will be released back to the file system. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
On Thu, Jul 30, 2009 at 11:33 AM, Boyd Stephen Smith Jr.<bss@iguanasuicide.net> wrote:
In <4A719CC3.7050201@dodin.org>, jdd (kim2) wrote:
Chuck Payne a écrit :
I have a question. Have you ever seen disc space just disappear and you were unable to find what is using?
look at mount points. May be you had a failure in mouting and so some files are hidden under the mount point
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt
Then, you can use whatever tools you'd normally use to look for files taking up space, such as: du -ax /mnt | sort -rn /tmp/fs_report && head -n $((LINES - 2)) /tmp/fs_report | cut -c "-$COLUMNS"
Finally, you may want to use lsof etc. to look for "phantom" files. When an open file is deleted from a Linux/UNIX file system, it continues to take up space in the file system and can even grow but won't be visible in directory listings until the last process using it has closed it, at which point all the space will be released back to the file system. -- Boyd Stephen Smith Jr. ,= ,-_-. =.
Cool trick. I did not know that one. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt
what is the use of this bind mount? jdd -- http://www.dodin.net http://valerie.dodin.org http://news.opensuse.org/2009/04/13/people-of-opensuse-jean-daniel-dodin/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Fri, Jul 31, 2009 at 1:48 AM, jdd (kim2)<jdd@dodin.org> wrote:
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt
what is the use of this bind mount?
jdd
jdd, Did you try it? It makes the root filesystem available as /mnt But none of the mounts are carried over, so on my desktop machine, /mnt/home is an empty directly, whereas /home on my machine is a mounted filesystem. So if I had hidden files in /home, they are hidden by the mount and it can be a pain to look underneath the mount. But after the above mount I can look at all of the underlying directories with no issues. A very cool diagnostic approach to this particular issue. Greg -- Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer Preservation and Forensic processing of Exchange Repositories White Paper - <http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html> The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Greg Freemyer a écrit :
On Fri, Jul 31, 2009 at 1:48 AM, jdd (kim2)<jdd@dodin.org> wrote:
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt what is the use of this bind mount?
jdd
jdd,
Did you try it?
no, my config is not the same as the OP
It makes the root filesystem available as /mnt
this I see
But after the above mount I can look at all of the underlying directories with no issues.
you mean the above mount shows all the mount point unmounted? very interesting indeed!! test done, it's even better: /proc, /dev and so on (presumably, not tested all) are not filled. This should also be a must for backup of /!! I will write this down... thanks jdd -- http://www.dodin.net http://valerie.dodin.org http://news.opensuse.org/2009/04/13/people-of-opensuse-jean-daniel-dodin/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Fri, Jul 31, 2009 at 11:06 AM, jdd (kim2)<jdd@dodin.org> wrote:
Greg Freemyer a écrit :
On Fri, Jul 31, 2009 at 1:48 AM, jdd (kim2)<jdd@dodin.org> wrote:
You can to this simply by bind mounting root on a temporary mount point: mount -o bind / /mnt what is the use of this bind mount?
jdd
jdd,
Did you try it?
no, my config is not the same as the OP
It makes the root filesystem available as /mnt
this I see
But after the above mount I can look at all of the underlying directories with no issues.
you mean the above mount shows all the mount point unmounted?
very interesting indeed!!
test done, it's even better: /proc, /dev and so on (presumably, not tested all) are not filled. This should also be a must for backup of /!!
Your right, that is very cool as well. Strangely I do still have directories: proc/bus/usb sys/kernel/debug I wonder if that is a bug in bind mounts. If it is, it is not much of one. I like this feature so much I think I'm going to update all of my /etc/fstab's to mount / as /pure_root (or something). (ie. not automount) That way anytime I want to work with just the root directory, I can just "mount /pure_root" and I have it. Greg -- Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer Preservation and Forensic processing of Exchange Repositories White Paper - <http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html> The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Boyd Stephen Smith Jr. -
Chuck Payne -
Dave Howorth -
David C. Rankin -
Greg Freemyer -
jdd (kim2)