[opensuse] Odd results from df
Hi ListMates, I'm not sure if this issue has been covered before or if anyone has come across this before. I recently did a fresh install of OpenSuse 12.3_x64 on one of my user's HP UEFI 64bit machines - the install went smoothly, the 1Tb disk was setup as GPT. I now, however, have some strange results from the "df" command!! rsgpfp0002:~ # uname -a Linux rsgpfp0002.loc 3.7.10-1.11-default #1 SMP Thu May 16 20:27:27 UTC 2013 (adf31bb) x86_64 x86_64 x86_64 GNU/Linux rsgpfp0002:~ # df -vh Filesystem Size Used Avail Use% Mounted on devtmpfs 2.9G 32K 2.9G 1% /dev tmpfs 2.9G 0 2.9G 0% /dev/shm tmpfs 2.9G 18M 2.9G 1% /run /dev/sda3 25G 4.5G 19G 19% / tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup tmpfs 2.9G 18M 2.9G 1% /var/lock tmpfs 2.9G 18M 2.9G 1% /var/run /dev/sda1 156M 128K 156M 1% /boot/efi /dev/sda4 891G 759G 87G 90% /public rsgpfp0002:~ # df -v Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2983260 32 2983228 1% /dev tmpfs 2998284 0 2998284 0% /dev/shm tmpfs 2998284 17764 2980520 1% /run /dev/sda3 25806716 4626136 19869656 19% / tmpfs 2998284 0 2998284 0% /sys/fs/cgroup tmpfs 2998284 17764 2980520 1% /var/lock tmpfs 2998284 17764 2980520 1% /var/run /dev/sda1 159564 128 159436 1% /boot/efi /dev/sda4 933396656 795685260 90297588 90% /public rsgpfp0002:~ # If I look at /dev/sd3 (25G - 4.5G does not equal 19G - I expect to get 20.5G!!!) or /dev/sda4 (891G - 759G does not equal 87G - I expect to get 132G!!!!), the numbers are way off!!!!! Has anyone seen this before, and where has all my disk space gone to?? Thanks for any insight into this!! Best regards. Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On June 14, 2013 at 8:53 AM Otto Rodusek <otto@applied.com.sg> wrote: rsgpfp0002:~ # df -vh Filesystem Size Used Avail Use% Mounted on [...] /dev/sda3 25G 4.5G 19G 19% / [...] /dev/sda4 891G 759G 87G 90% /public
rsgpfp0002:~ # df -v Filesystem 1K-blocks Used Available Use% Mounted on [...] /dev/sda3 25806716 4626136 19869656 19% / [...] /dev/sda4 933396656 795685260 90297588 90% /public
If I look at /dev/sd3 (25G - 4.5G does not equal 19G - I expect to get 20.5G!!!) or /dev/sda4 (891G - 759G does not equal 87G - I expect to get 132G!!!!), the numbers are way off!!!!!
Has anyone seen this before, and where has all my disk space gone to??
Df's -h option is short for --human-readable and equivalent to --block-size=human-readable. According to the Texinfo manual ("info coreutils 'df invocation'"), each non-integer number is rounded up to the next higher unit. AFAIR the same question arrived the upstream coreutils mailing list last year, and the answer was that this is mandated by POSIX [1]: ... all quantities ... shall be rounded up to the next higher unit. You get the weidest results when using astronomic block sizes, e.g.: $ df -BE / Filesystem 1E-blocks Used Available Use% Mounted on /dev/sda3 1E 1E 1E 29% / [1] http://pubs.opengroup.org/onlinepubs/007904875/utilities/df.html Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi Berny, Thanks for the reply. I'm not sure I understand tho. I know what you mean by "human-readable" and "each non-integer number is rounded up to the next higher unit." but the numbers are so way off. For example - df --block-size=1K Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2983260 32 2983228 1% /dev tmpfs 2998284 0 2998284 0% /dev/shm tmpfs 2998284 17844 2980440 1% /run /dev/sda3 25806716 4626964 19868828 19% / tmpfs 2998284 0 2998284 0% /sys/fs/cgroup tmpfs 2998284 17844 2980440 1% /var/lock tmpfs 2998284 17844 2980440 1% /var/run /dev/sda1 159564 128 159436 1% /boot/efi /dev/sda4 933396656 795692408 90290440 90% /public Let's look at /dev/sda4 ( if I take 933,396,565 - 795,692,408 I get and expect to get 137,704,248 and not the reported 90,290,440 a difference of 47GB!!!!). If that is the case with df, how does one get a "reasonable" size of used and available space?? This is rather confusing as the numbers are so far apart!! Otto. On 14/6/13 3:41 PM, Bernhard Voelker wrote:
On June 14, 2013 at 8:53 AM Otto Rodusek <otto@applied.com.sg> wrote: rsgpfp0002:~ # df -vh Filesystem Size Used Avail Use% Mounted on [...] /dev/sda3 25G 4.5G 19G 19% / [...] /dev/sda4 891G 759G 87G 90% /public
rsgpfp0002:~ # df -v Filesystem 1K-blocks Used Available Use% Mounted on [...] /dev/sda3 25806716 4626136 19869656 19% / [...] /dev/sda4 933396656 795685260 90297588 90% /public
If I look at /dev/sd3 (25G - 4.5G does not equal 19G - I expect to get 20.5G!!!) or /dev/sda4 (891G - 759G does not equal 87G - I expect to get 132G!!!!), the numbers are way off!!!!!
Has anyone seen this before, and where has all my disk space gone to?? Df's -h option is short for --human-readable and equivalent to --block-size=human-readable. According to the Texinfo manual ("info coreutils 'df invocation'"), each non-integer number is rounded up to the next higher unit.
AFAIR the same question arrived the upstream coreutils mailing list last year, and the answer was that this is mandated by POSIX [1]:
... all quantities ... shall be rounded up to the next higher unit.
You get the weidest results when using astronomic block sizes, e.g.:
$ df -BE / Filesystem 1E-blocks Used Available Use% Mounted on /dev/sda3 1E 1E 1E 29% /
[1] http://pubs.opengroup.org/onlinepubs/007904875/utilities/df.html
Have a nice day, Berny
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Jun 14, 2013, at 10:04 , Otto Rodusek wrote:
Let's look at /dev/sda4 ( if I take 933,396,565 - 795,692,408 I get and expect to get 137,704,248 and not the reported 90,290,440 a difference of 47GB!!!!). If that is the case with df, how does one get a "reasonable" size of used and available space?? This is rather confusing as the numbers are so far apart!!
The space reported as "available" applies to non-privileged users. Space that is reserved for privileged users is contained in the "Size" column only, not in "Used" or "Avail". For ext2, the default for this quantity is 5%, so 47G seems reasonable. A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
On 14/6/13 4:11 PM, Esztermann, Ansgar wrote:
On Jun 14, 2013, at 10:04 , Otto Rodusek wrote:
Let's look at /dev/sda4 ( if I take 933,396,565 - 795,692,408 I get and expect to get 137,704,248 and not the reported 90,290,440 a difference of 47GB!!!!). If that is the case with df, how does one get a "reasonable" size of used and available space?? This is rather confusing as the numbers are so far apart!! The space reported as "available" applies to non-privileged users. Space that is reserved for privileged users is contained in the "Size" column only, not in "Used" or "Avail". For ext2, the default for this quantity is 5%, so 47G seems reasonable.
A. Hi Angsar,
I'm using ext4 (although I don't think that makes a difference!!). So, from your comment, if I have 933,396,565K and use up 795,692,408K, in reality I can only write a further 90,290,440K and not 137,704,248K - literally 47Gb is gone into limbo?? is this correct?? Ouch!! Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-06-14 10:25, Otto Rodusek wrote:
On 14/6/13 4:11 PM, Esztermann, Ansgar wrote:
I'm using ext4 (although I don't think that makes a difference!!). So, from your comment, if I have 933,396,565K and use up 795,692,408K, in reality I can only write a further 90,290,440K and not 137,704,248K - literally 47Gb is gone into limbo?? is this correct?? Ouch!!
You can adjust the percent to nil. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On 14/6/13 4:32 PM, Carlos E. R. wrote:
On 2013-06-14 10:25, Otto Rodusek wrote:
On 14/6/13 4:11 PM, Esztermann, Ansgar wrote: I'm using ext4 (although I don't think that makes a difference!!). So, from your comment, if I have 933,396,565K and use up 795,692,408K, in reality I can only write a further 90,290,440K and not 137,704,248K - literally 47Gb is gone into limbo?? is this correct?? Ouch!! You can adjust the percent to nil.
Hi Carlos, Yep, I just got my answer. From a previous thread and also from Angsar, "When a file system is created, 5% of all blocks are reserved for the super user (for writing logs etc.) If this disk is only for storing files (and e.g. doesn't contain /var/) you can set this to 0% (or you may keep 1% just in case):" $ tune2fs -m 0 /dev/sd???? Solves the problem. Thanks for all the answers and helpful hints. Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On June 14, 2013 at 10:42 AM Otto Rodusek <otto@applied.com.sg> wrote: Yep, I just got my answer. From a previous thread and also from Angsar, "When a file system is created, 5% of all blocks are reserved for the super user (for writing logs etc.) If this disk is only for storing files (and e.g. doesn't contain /var/) you can set this to 0% (or you may keep 1% just in case):"
$ tune2fs -m 0 /dev/sd???? Solves the problem.
I would only do that for file systems with user data like /home, but not for system file systems like / or /var. A normal user could probably fill up the partition, and the system probably would crash ... Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/14/2013 10:42 AM, Otto Rodusek wrote:
$ tune2fs -m 0 /dev/sd???? Solves the problem.
yes, it solves the problem of "the numbers don't match" but it creates a new problem which those reserved-for-root-use space laid in the solution _before_ it happened.. that is, with the reserve: the day you can't log in as user because your disk is full, you _could_ log in as root and dump/move some stuff so that the disk was no longer full.. but as you have it now, with zero reserve, when the disk is full: not even root can get in to dump (say) /tmp and then find/fix the disk filling problem (like a log filled with thousands and thousands of error messages from some broke do-dad) anyway, with a 5% reserve *all* of the disk is still there and all of it is usable...and, once you know why the numbers don't match there is no problem (never was actually, just a lack of knowledge/understanding).. do as you wish, but consider a small reserve as a good thing to have (i think 5% was chosen before disks got so large that that was a LOT of gigs of space) dd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-06-14 12:17, DenverD wrote:
On 06/14/2013 10:42 AM, Otto Rodusek wrote:
$ tune2fs -m 0 /dev/sd???? Solves the problem.
yes, it solves the problem of "the numbers don't match" but it creates a new problem which those reserved-for-root-use space laid in the solution _before_ it happened..
that is, with the reserve: the day you can't log in as user because your disk is full, you _could_ log in as root and dump/move some stuff so that the disk was no longer full..
It only really aplies to the system partitions, not to a disk reserved for data only.
but as you have it now, with zero reserve, when the disk is full: not even root can get in to dump (say) /tmp and then find/fix the disk filling problem (like a log filled with thousands and thousands of error messages from some broke do-dad)
The log will fill if the process that writes is owned by root, anyway ;-) Anyway, having a reserve can be good for other things: allocation of space, shuffling... -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Jun 14, 2013, at 12:26 , Carlos E. R. wrote:
Anyway, having a reserve can be good for other things: allocation of space, shuffling...
Oh yes. For example, XFS gets _really_ slow when it fills up, and it stays slow even after you've deleted enough files. A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
Esztermann, Ansgar said the following on 06/18/2013 03:29 AM:
On Jun 14, 2013, at 12:26 , Carlos E. R. wrote:
Anyway, having a reserve can be good for other things: allocation of space, shuffling...
Oh yes. For example, XFS gets _really_ slow when it fills up, and it stays slow even after you've deleted enough files.
Hmm. Might be a b-tree balance problem rather than simply a space allocation problem. I'm not seeing anything like that with BtrFS, but then BtrFS has many kernel threads and I suspect at least one of them has to do with keeping the trees in shape. ps -ef | grep btrfs root 202 2 0 06:30 ? 00:00:00 [btrfs-genwork-1] root 203 2 0 06:30 ? 00:00:00 [btrfs-submit-1] root 204 2 0 06:30 ? 00:00:00 [btrfs-delalloc-] root 205 2 0 06:30 ? 00:00:00 [btrfs-fixup-1] root 208 2 0 06:30 ? 00:00:00 [btrfs-endio-met] root 210 2 0 06:30 ? 00:00:00 [btrfs-freespace] root 211 2 0 06:30 ? 00:00:00 [btrfs-delayed-m] root 212 2 0 06:30 ? 00:00:00 [btrfs-cache-1] root 213 2 0 06:30 ? 00:00:00 [btrfs-readahead] root 214 2 0 06:30 ? 00:00:00 [btrfs-cleaner] root 215 2 0 06:30 ? 00:00:00 [btrfs-transacti] root 252 2 0 06:30 ? 00:00:00 [flush-btrfs-1] root 331 2 0 06:31 ? 00:00:01 [btrfs-endio-3] root 3381 2 0 06:39 ? 00:00:00 [btrfs-endio-wri] root 4129 2 0 06:45 ? 00:00:00 [btrfs-worker-2] root 4170 2 0 06:50 ? 00:00:00 [btrfs-endio-met] root 4254 2 0 07:00 ? 00:00:00 [btrfs-endio-met] This is 12.2 system with one gigantic BtrFS for the whole system. I've stress tested resiserFS rather more over the years but that has always been on multi-partition systems so might not show up so easily. -- How long did the whining go on when KDE2 went on KDE3? The only universal constant is change. If a species can not adapt it goes extinct. That's the law of the universe, adapt or die. -- Billie Walsh, May 18 2013 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 14/6/13 3:41 PM, Bernhard Voelker wrote:
On June 14, 2013 at 8:53 AM Otto Rodusek <otto@applied.com.sg> wrote: rsgpfp0002:~ # df -vh Filesystem Size Used Avail Use% Mounted on [...] /dev/sda3 25G 4.5G 19G 19% / [...] /dev/sda4 891G 759G 87G 90% /public
rsgpfp0002:~ # df -v Filesystem 1K-blocks Used Available Use% Mounted on [...] /dev/sda3 25806716 4626136 19869656 19% / [...] /dev/sda4 933396656 795685260 90297588 90% /public
If I look at /dev/sd3 (25G - 4.5G does not equal 19G - I expect to get 20.5G!!!) or /dev/sda4 (891G - 759G does not equal 87G - I expect to get 132G!!!!), the numbers are way off!!!!!
Has anyone seen this before, and where has all my disk space gone to?? Df's -h option is short for --human-readable and equivalent to --block-size=human-readable. According to the Texinfo manual ("info coreutils 'df invocation'"), each non-integer number is rounded up to the next higher unit.
AFAIR the same question arrived the upstream coreutils mailing list last year, and the answer was that this is mandated by POSIX [1]:
... all quantities ... shall be rounded up to the next higher unit.
You get the weidest results when using astronomic block sizes, e.g.:
$ df -BE / Filesystem 1E-blocks Used Available Use% Mounted on /dev/sda3 1E 1E 1E 29% /
[1] http://pubs.opengroup.org/onlinepubs/007904875/utilities/df.html
Have a nice day, Berny
***OOPS*** sorry for the previous top posting reply!! *** Hi Berny, Thanks for the reply. I'm not sure I understand tho. I know what you mean by "human-readable" and "each non-integer number is rounded up to the next higher unit." but the numbers are so way off. For example - df --block-size=1K Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 2983260 32 2983228 1% /dev tmpfs 2998284 0 2998284 0% /dev/shm tmpfs 2998284 17844 2980440 1% /run /dev/sda3 25806716 4626964 19868828 19% / tmpfs 2998284 0 2998284 0% /sys/fs/cgroup tmpfs 2998284 17844 2980440 1% /var/lock tmpfs 2998284 17844 2980440 1% /var/run /dev/sda1 159564 128 159436 1% /boot/efi /dev/sda4 933396656 795692408 90290440 90% /public Let's look at /dev/sda4 ( if I take 933,396,565 - 795,692,408 I get and expect to get 137,704,248 and not the reported 90,290,440 a difference of 47GB!!!!). If that is the case with df, how does one get a "reasonable" size of used and available space?? This is rather confusing as the numbers are so far apart!! Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker wrote:
On June 14, 2013 at 8:53 AM Otto Rodusek <otto@applied.com.sg> wrote: rsgpfp0002:~ # df -vh Filesystem Size Used Avail Use% Mounted on [...] /dev/sda3 25G 4.5G 19G 19% / [...] /dev/sda4 891G 759G 87G 90% /public
rsgpfp0002:~ # df -v Filesystem 1K-blocks Used Available Use% Mounted on [...] /dev/sda3 25806716 4626136 19869656 19% / [...] /dev/sda4 933396656 795685260 90297588 90% /public
... all quantities ... shall be rounded up to the next higher unit.
Wait--- 25G could be 24.0000001G? so that would leave 19.5G, why wouldn't that round up to 20G? posix has always been braindead and is getting worse. it should be sterilized. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-06-14 11:26, Linda Walsh wrote:
Wait--- 25G could be 24.0000001G? so that would leave 19.5G, why wouldn't that round up to 20G?
I would have to read the rules... Normally, 10.4 rounds to 10, and 10.5 to 11. If posix does different, I dunno. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Jun 14, 2013, at 11:26 , Linda Walsh wrote:
... all quantities ... shall be rounded up to the next higher unit. Wait--- 25G could be 24.0000001G? so that would leave 19.5G, why wouldn't that round up to 20G?
posix has always been braindead and is getting worse.
POSIX does not say anything about GB (at least according to the man page installed by Suse. It only speaks about units of 512 bytes and 1024 bytes. Rounding of other units (e.g. GB) is up to GNU. A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
What really fun is using df to tell how many inodes you have left. Sometimes you can have plenty of space available and be unable create more files 'cos you've run out of inodes. One reason I prefer ReiserFS to the ext family :-) -- Why do cats get fur balls and dogs don't? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
What really fun is using df to tell how many inodes you have left.
Sometimes you can have plenty of space available and be unable create more files 'cos you've run out of inodes.
One reason I prefer ReiserFS to the ext family :-)
XFS will allow an administrator to reset the amount of disk space able to be allocated for inodes. It prevents the standard DOS of a user being able to create unlimited inodes until disk space is filled up, as many quota systems either don't pay attention to inode space OR simply weren't designed to be strict enough for diskspace usage solely in the form of inodes. 1GB space would be enough for MANY inodes... Of course for a personal system, it would be rare to have a program run amok creating inodes...but in general, flood-breaks or similar do have a useful purpose provided they aren't a pain to move & extend. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Jun 14, 2013, at 22:14 , Linda Walsh wrote:
Of course for a personal system, it would be rare to have a program run amok creating inode
... but a mail loop can do it just fine. A. -- Ansgar Esztermann DV-Systemadministration Max-Planck-Institut für biophysikalische Chemie, Abteilung 105
participants (7)
-
Anton Aylward -
Bernhard Voelker -
Carlos E. R. -
DenverD -
Esztermann, Ansgar -
Linda Walsh -
Otto Rodusek