[opensuse-factory] zypper dup disk space requirement

older
[opensuse-factory] Leap 15.0: krb5...

Peter Suetterlin

6 Feb 2019 6 Feb '19

13:40

Hi List, I just did my 'zypper dup' to the fully recompiled system. As there had been some reports about out of disk space conditions I thought I watch it carefully, and report here. I started deleting all old snapshots. This left me with woodstock:~% df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 49G 20G 72% / Then I started the dup. It told me it would download 3576 packages, totaling to 3.something GB. Only one conflict with a python package. So I started. After update is finished, this is the usage: woodstock:~% df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.2G 91% / So, that's 13.8GB space needed for the update because all files are COWed during update, but stay undeleted in the snapshot :o Now that extent might be non-standard, as I have installed lots of development stuff and also TeX. But the keypoint is that such an update will require the full size of your current system (minus the logs of course) as free space. (The difference between the initial 49GB usage and those ~14GB is mostly in the /usr/local subvolume and external software in /opt). So for sure checking that there is enough space to download the packages in advance is not sufficient to make sure the upgrade succeeds. Maybe zypper should get some better logic like summing the installed sizes of the packages to be installed and make sure that is available? (At least when snapshots are enabled, that is). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Show replies by date

Roger Oberholtzer

6 Feb 6 Feb

14:08

On Wed, Feb 6, 2019 at 2:40 PM Peter Suetterlin wrote:

...

woodstock:~% df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 49G 20G 72% /

Then I started the dup. It told me it would download 3576 packages, totaling to 3.something GB. Only one conflict with a python package. So I started.

After update is finished, this is the usage:

woodstock:~% df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.2G 91% /

So, that's 13.8GB space needed for the update because all files are COWed during update, but stay undeleted in the snapshot :o

Now that extent might be non-standard, as I have installed lots of development stuff and also TeX. But the keypoint is that such an update will require the full size of your current system (minus the logs of course) as free space. (The difference between the initial 49GB usage and those ~14GB is mostly in the /usr/local subvolume and external software in /opt).

I see similar requirements. The check that the RPMs fit is a start. But it hardly can be used to say that the install will complete.

...

So for sure checking that there is enough space to download the packages in advance is not sufficient to make sure the upgrade succeeds. Maybe zypper should get some better logic like summing the installed sizes of the packages to be installed and make sure that is available? (At least when snapshots are enabled, that is).

+1 -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

14:50

On Wed, 6 Feb 2019 at 15:08, Roger Oberholtzer wrote:

...

...
woodstock:~% df -h /

...

...
woodstock:~% df -h /

df is not an accurate tool for providing any reliable information on the free space available on a btrfs filesystem Please use `btrfs fi df /` (aka `btrfs filesystem df /`) instead Or `btrfs fi us /` (aka `btrfs filesystem usage /`) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

15:05

Richard Brown wrote:

...

On Wed, 6 Feb 2019 at 15:08, Roger Oberholtzer wrote:

...
...
woodstock:~% df -h /

...
...
woodstock:~% df -h /

df is not an accurate tool for providing any reliable information on the free space available on a btrfs filesystem

Please use `btrfs fi df /` (aka `btrfs filesystem df /`) instead Or `btrfs fi us /` (aka `btrfs filesystem usage /`)

For this issue it is sufficient and needed, as it refers to the total amount available on the partition. The issue is 'make snapshot, replace all packages = twice the space' is independent of any tool/program used to measure it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

15:43

On Wed, 6 Feb 2019 at 16:06, Peter Suetterlin wrote:

...

For this issue it is sufficient and needed, as it refers to the total amount available on the partition.

The issue is 'make snapshot, replace all packages = twice the space' is independent of any tool/program used to measure it.

There is a different of multiple GB between the space used on every-single-one of my btrfs systems (and I have a lot) when looking at `df -h` compared to `btrfs fi X` On every single one of my systems, that difference is enough to come to a different conclusion regarding the needs of my systems than the one you pose in your original post But, I can't even consider whether your proposition is correct or not, because you have posted to this list data which is known to be untrustworthy So, how about, instead of arguing fruitlessly with me, you take my advice and actual post to this list the results of a command that is actually trustworthy on reporting the space in use on your systems? Because, you might want my help on this topic, I'm the last person who touched the partition sizing rules in Tumbleweed. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

16:03

Richard Brown wrote:

...

On Wed, 6 Feb 2019 at 16:06, Peter Suetterlin wrote:

...
For this issue it is sufficient and needed, as it refers to the total amount available on the partition.

The issue is 'make snapshot, replace all packages = twice the space' is independent of any tool/program used to measure it.

There is a different of multiple GB between the space used on every-single-one of my btrfs systems (and I have a lot) when looking at `df -h` compared to `btrfs fi X`

*sigh* woodstock:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.1G 91% / woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB GlobalReserve, single: total=125.97MiB, used=0.00B As you probably understand I cannot give you the similar numbers for the state before the update. So what's different now?

...

So, how about, instead of arguing fruitlessly with me, you take my advice and actual post to this list the results of a command that is actually trustworthy on reporting the space in use on your systems?

Maybe instead of fruitlessly discussing about numbers, what have you to say about the intrinsic issue itself? Do you claim that if I do a snapshot and then replace all the files in the snapshot with new versions, the used space will not double?

...

Because, you might want my help on this topic, I'm the last person who touched the partition sizing rules in Tumbleweed.

Well, I do not need any help. My updates go fine, because I check myself what is going to happen, and if there's enough space etc. I only put some (in your word useless) numbers to an issue that several people have reported here, in the hope to trigger a discussion whether some changes in zypper might be reasonable to save those people from their issues. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

16:12

On Wed, 6 Feb 2019 at 17:03, Peter Suetterlin wrote:

...

*sigh*

woodstock:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.1G 91% / woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB GlobalReserve, single: total=125.97MiB, used=0.00B

As you probably understand I cannot give you the similar numbers for the state before the update. So what's different now?

Now I can't dismiss your point of view as based on invalid assumptions. Which is a good thing to have cleared up - any developer is always going to look for an easy reason to ignore your ideas. Now we've together eliminated that threat to your ideas.

...

...
So, how about, instead of arguing fruitlessly with me, you take my advice and actual post to this list the results of a command that is actually trustworthy on reporting the space in use on your systems?

Maybe instead of fruitlessly discussing about numbers, what have you to say about the intrinsic issue itself? Do you claim that if I do a snapshot and then replace all the files in the snapshot with new versions, the used space will not double?

Well I don't see the issue..but I've been using the new partition logic which makes the root filesystem fill the device without excess partitioning..so for me, I don't see it as a major issue. But I didn't want to be dismissive.. now you've shared accurate data, I can see how it's a problem for you. You have my sympathy.. but that isn't going to change much for you.

...

Well, I do not need any help. My updates go fine, because I check myself what is going to happen, and if there's enough space etc. I only put some (in your word useless) numbers to an issue that several people have reported here, in the hope to trigger a discussion whether some changes in zypper might be reasonable to save those people from their issues.

Oh well in that case, might I suggest a better place for that discussion would be the zypper github issues tracker? https://github.com/openSUSE/zypper/issues or file a bug on bugzilla.opensuse.org, either should work equally well for reaching the right brains you wish to tickle. Either seems like a more sensible venue more directly interacting with the people who'd have to implement such a feature, rather than the much broader audience of this list. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Roger Oberholtzer

7 Feb 7 Feb

07:32

On Wed, Feb 6, 2019 at 4:43 PM Richard Brown wrote:

...

But, I can't even consider whether your proposition is correct or not, because you have posted to this list data which is known to be untrustworthy

The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update. So updates start and fail because of lack of disk space. All that was being suggested was that perhaps zypper could be improved to also consider the space needed for (1) the uncompressed version of the RPM (i.e., the space the RPM will need for installation), and (2) the fact that many files will require disk space for the old and new versions until the old version is no longer needed (which is usually not resolvable until the update is complete). I realize that this is perhaps a less than obvious calculation. But the current one is grossly inadequate. And as to untrustworthy, well, I think that many people with what appears to be adequate disk space as determined by zypper have had installations fail because of lack of disk space. The question, it seems to me, is not if this is happening. It is how to improve the situation so it is less likely to happen again. It that involves the use of df or a btrfs program is not the salient point. Whatever the best way is to determine actual space available should be used. And this is not a btrfs issue. It is just that the layout of btrfs seems to cause this in some systems where the root file system was perhaps not made big enough. I have a laptop in that category. Since it happily runs Tumbleweed, I don't really want to reinstall. This is the only issue I have. I have learned to live with it. That is, not to base the chance of a successful update on the zypper free space test. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

07:53

On 2/7/19 8:32 AM, Roger Oberholtzer wrote:

...

The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update.

It is about BTRFS being "different" from (most/all?) other file system types: a) It is by nature that BTRFS needs almost double space for such huge updates because it keeps the old data in a snapshot. b) It does not expose the correct disk usage statistics via the kernel interfaces like all other file systems do since ages: statfs(2). Thus, df(1) doesn't have a chance, and simply reports those not-much-useful numbers it gets from BTRFS. And no, there is no upstream activity to add such special support to GNU coreutils' df(1). On my home PC, I'm using a traditional EXT4, so I can easily live with 20G for "/" ... still I need the same size for regular backup space. The difference for me is that I can trust the numbers df(1) is getting from the kernel. Yet I'm wondering if BTRFS guys don't have a separate backup - which is then 4x? For my oS:TW VMs, I simply use the snapshots of the virtualization as backup before any updates. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

09:50

Bernhard Voelker wrote:

...

On 2/7/19 8:32 AM, Roger Oberholtzer wrote:

...
The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update.

It is about BTRFS being "different" from (most/all?) other file system types:

a) It is by nature that BTRFS needs almost double space for such huge updates because it keeps the old data in a snapshot.

It's not even btrfs itself. It is snapper. Without that, the update would create the new file, but as the old one no longer has an directory entry pointing to it, it will be freed quite fast(*). But if you use snapshots, space cannot be freed until you remove that snapshot. (*) Check that when you delete a snapshot. On my TW it takes 10-30s at most until it is freed.

...

b) It does not expose the correct disk usage statistics via the kernel interfaces like all other file systems do since ages: statfs(2). Thus, df(1) doesn't have a chance, and simply reports those not-much-useful numbers it gets from BTRFS. And no, there is no upstream activity to add such special support to GNU coreutils' df(1).

TBH, I've never seen huge differences between df and btrfs fi df. If that makes the difference between a succeeded and a failed update the filesystem is already too small anyhow ;^>

...

On my home PC, I'm using a traditional EXT4, so I can easily live with 20G for "/" ...

So you don't use snapper. Deactivate it on btrfs, and you would be able to go with 20G, too, I'd say.

...

still I need the same size for regular backup space.

That's the point.

...

Yet I'm wondering if BTRFS guys don't have a separate backup - which is then 4x?

My server runs on a BTRFS RAID1, that would count as 4x I guess. For normal machines I don't really care much regarding the system. A reinstall is almost as fast as (or even faster than?) using a backup... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

14:22

On 2/7/19 10:50 AM, Peter Suetterlin wrote:

...

Bernhard Voelker wrote:

...
On my home PC, I'm using a traditional EXT4, so I can easily live with 20G for "/" ...

So you don't use snapper. Deactivate it on btrfs, and you would be able to go with 20G, too, I'd say.

Yeah, that's the freedom of software we all love: use the software you like for any purpose you like - and sharing with others, of course! ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

09:44

On 07/02/2019 08.32, Roger Oberholtzer wrote:

...

On Wed, Feb 6, 2019 at 4:43 PM Richard Brown wrote:

...
But, I can't even consider whether your proposition is correct or not, because you have posted to this list data which is known to be untrustworthy

The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update. So updates start and fail because of lack of disk space.

You are both correct :-) The btrfs size report tool paints a worse picture than df: Data, single: total=60.43GiB, used=59.82GiB There is no space on the disk at all - if I read that correctly, which perhaps I don't, because I don't understand that command and btrfs. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)

Peter Suetterlin

09:57

Carlos E. R. wrote:

...

The btrfs size report tool paints a worse picture than df:

Data, single: total=60.43GiB, used=59.82GiB

There is no space on the disk at all - if I read that correctly, which perhaps I don't, because I don't understand that command and btrfs.

I think this is the difference between real data amount and occupied sectors times sector size. The partition itself is 68GB.... The difference between btrfs data use (60.4) and df partition use (61) is likely metadata and rounding... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

10:00

On Thu, 7 Feb 2019 at 10:45, Carlos E. R. wrote:

...

On 07/02/2019 08.32, Roger Oberholtzer wrote:

...
On Wed, Feb 6, 2019 at 4:43 PM Richard Brown wrote:

...
But, I can't even consider whether your proposition is correct or not, because you have posted to this list data which is known to be untrustworthy

The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update. So updates start and fail because of lack of disk space.

You are both correct :-)

The btrfs size report tool paints a worse picture than df:

Data, single: total=60.43GiB, used=59.82GiB

There is no space on the disk at all - if I read that correctly, which perhaps I don't, because I don't understand that command and btrfs.

You didn't read it correctly ;) "total=" relates to the 'total allocated space' for that class of data (Data) "used=" related to how much of that space is used btrfs filesystem usage goes into far more useful detail For example on my system I have a 444GiB sized device 198GiB is allocated 195GiB is allocated to Data 3GiB is allocated to Metadata 32MiB is allocated to System The remainder 246GiB is unallocated and can be chosen by btrfs to be used for Data, Metadata, or System as it needs it. Inside those 3 buckets I am using 192GiB in Data 1.45GiB in Metadata 48KiB in System And so, with a bit of math, it's relatively easy to calculate that out of my 444GiB disk, at least 248GiB is still available. Might be more, depending on the impact on de-duplication and such. Rich -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

10:14

On 07/02/2019 11.00, Richard Brown wrote:

...

On Thu, 7 Feb 2019 at 10:45, Carlos E. R. wrote:

...
On 07/02/2019 08.32, Roger Oberholtzer wrote:

...
On Wed, Feb 6, 2019 at 4:43 PM Richard Brown wrote:

...
But, I can't even consider whether your proposition is correct or not, because you have posted to this list data which is known to be untrustworthy

The problem is not df vs some better utility to determine available space. It is that the test by zypper does not account for all the space that may be needed to complete the update. So updates start and fail because of lack of disk space.

You are both correct :-)

The btrfs size report tool paints a worse picture than df:

Data, single: total=60.43GiB, used=59.82GiB

There is no space on the disk at all - if I read that correctly, which perhaps I don't, because I don't understand that command and btrfs.

You didn't read it correctly ;)

"total=" relates to the 'total allocated space' for that class of data (Data) "used=" related to how much of that space is used

btrfs filesystem usage goes into far more useful detail

For example on my system I have a 444GiB sized device 198GiB is allocated 195GiB is allocated to Data 3GiB is allocated to Metadata 32MiB is allocated to System The remainder 246GiB is unallocated and can be chosen by btrfs to be used for Data, Metadata, or System as it needs it.

Inside those 3 buckets I am using 192GiB in Data 1.45GiB in Metadata 48KiB in System

And so, with a bit of math, it's relatively easy to calculate that out of my 444GiB disk, at least 248GiB is still available.

Ok; then he said: woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB So there is (rough aprox): 1 GB of data space available 31 MB of system space available 1 GB of metadata space available whereas woodstock:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.1G 91% / So basically btrfs tools says 1 G free, df says 6. The situation is much worse than what df says, right? -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)

Joachim Wagner

10:26

...

woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB

So there is (rough aprox):

1 GB of data space available 31 MB of system space available 1 GB of metadata space available [...] So basically btrfs tools says 1 G free, df says 6. The situation is much worse than what df says, right?

No. The 6 GB are available to grow any of the 3 spaces as needed. Have a look what happens if you copy a large file onto it. Joachim -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

10:29

Joachim Wagner wrote:

...

...
So basically btrfs tools says 1 G free, df says 6. The situation is much worse than what df says, right?

...

No. The 6 GB are available to grow any of the 3 spaces as needed. Have a look what happens if you copy a large file onto it.

The main issue is that 'btrfs fi df /' neither reports the total size of the partition/available space, nor the part that is usable for new data, like df does... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

10:30

On Thu, 7 Feb 2019 at 11:15, Carlos E. R. wrote:

...

Ok; then he said:

woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB

So there is (rough aprox):

1 GB of data space available 31 MB of system space available 1 GB of metadata space available

whereas

woodstock:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.1G 91% /

So basically btrfs tools says 1 G free, df says 6. The situation is much worse than what df says, right?

You're mixing up different units The device is 68G, that is 63GiB in size df says 61G is used with 6.1G free (56GiB used, 5.6GiB free) btrfs fi df / says 60.735GiB is used, therefore just over 2GiB free So yes, the situation is 'worse' than what df says..or in other words, df spit out useless nonsense that is not suitable for measuring the available space, and really shouldn't be used as the basis for any meaningful discussions -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andreas Schwab

10:36

New subject: [opensuse-factory] Re: zypper dup disk space requirement

On Feb 07 2019, Richard Brown wrote:

...

df says 61G is used with 6.1G free (56GiB used, 5.6GiB free)

df -h uses IEC units, not SI units (use df --si to display the latter). Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

11:47

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2019-02-07 a las 11:30 +0100, Richard Brown escribió:

...

On Thu, 7 Feb 2019 at 11:15, Carlos E. R. wrote:

...
Ok; then he said:

woodstock:~ # btrfs fi df / Data, single: total=60.43GiB, used=59.82GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=2.03GiB, used=915.50MiB

So there is (rough aprox):

1 GB of data space available 31 MB of system space available 1 GB of metadata space available

whereas

woodstock:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda2 68G 61G 6.1G 91% /

So basically btrfs tools says 1 G free, df says 6. The situation is much worse than what df says, right?

You're mixing up different units

The device is 68G, that is 63GiB in size df says 61G is used with 6.1G free (56GiB used, 5.6GiB free) btrfs fi df / says 60.735GiB is used, therefore just over 2GiB free

Do you want a bug report about 'df' using GiB but saying G? Because it is a bug, the units used should be GiB or GB according the the IEEE and other institutions. If I see 'GiB' I know what it is, if it says 'G' I don't know, if it says 'GB' I have doubts whether they follow the standard or the deprecated standard. >:-) And yes, I wrote by mistake GB instead of GiB in my previous post.

...

So yes, the situation is 'worse' than what df says..or in other words, df spit out useless nonsense that is not suitable for measuring the available space, and really shouldn't be used as the basis for any meaningful discussions

OK! :-) - -- Cheers Carlos E. R. (from openSUSE 15.0 (Legolas)) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXFwa5Bwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfV5TcAnAnGR4hMNtDtpBaLShN9 1SjkKclzAJ9LojV6LnGw38akRcswfTBv5c7mlg== =TxzS -----END PGP SIGNATURE-----

Bernhard Voelker

14:39

On 2/7/19 12:47 PM, Carlos E. R. wrote:

...

El 2019-02-07 a las 11:30 +0100, Richard Brown escribió:

...
You're mixing up different units

...
The device is 68G, that is 63GiB in size df says 61G is used with 6.1G free (56GiB used, 5.6GiB free) btrfs fi df / says 60.735GiB is used, therefore just over 2GiB free

Do you want a bug report about 'df' using GiB but saying G? Because it is a bug, the units used should be GiB or GB according the the IEEE and other institutions. If I see 'GiB' I know what it is, if it says 'G' I don't know, if it says 'GB' I have doubts whether they follow the standard or the deprecated standard. >:-)

You asked df(1) to output with "-human-readable" size, and that's what it did: -h, --human-readable print sizes in powers of 1024 (e.g., 1023M) GNU coreutils follows POSIX here, so there's nothing wrong with the output you got: http://pubs.opengroup.org/onlinepubs/9699919799/ https://www.gnu.org/software/coreutils/manual/html_node/df-invocation.html https://www.gnu.org/software/coreutils/manual/html_node/Block-size.html For example, ‘1M’ and ‘1MiB’ are equivalent to ‘1048576’, whereas ‘1MB’ is equivalent to ‘1000000’.

...

And yes, I wrote by mistake GB instead of GiB in my previous post.

...
So yes, the situation is 'worse' than what df says..or in other words, df spit out useless nonsense that is not suitable for measuring the available space, and really shouldn't be used as the basis for any meaningful discussions

OK! :-)

I disagree: as already stated, BTRFS does not expose the right (or useful?) numbers to the statfs(2) system call, so df(1) just displays what it gets from BTRFS. There's nothing wrong with df(1). Analogy: if you tell 'echo' to print "pig", then it won't print "pork". Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andreas Schwab

14:55

New subject: [opensuse-factory] Re: zypper dup disk space requirement

On Feb 07 2019, Bernhard Voelker wrote:

...

You asked df(1) to output with "-human-readable" size, and that's what it did:

-h, --human-readable print sizes in powers of 1024 (e.g., 1023M)

GNU coreutils follows POSIX here, so there's nothing wrong with the output you got:

df -h is an extension, so POSIX doesn't really come into play here. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

18:07

New subject: [opensuse-factory] Re: zypper dup disk space requirement

On 2/7/19 3:55 PM, Andreas Schwab wrote:

...

On Feb 07 2019, Bernhard Voelker wrote:

...
You asked df(1) to output with "-human-readable" size, and that's what it did:

-h, --human-readable print sizes in powers of 1024 (e.g., 1023M)

GNU coreutils follows POSIX here, so there's nothing wrong with the output you got:

df -h is an extension, so POSIX doesn't really come into play here.

True. BTW: also the *BSDs have 'df -h' with the same units, so there's some compatibility to consider. And, "M" without suffix is a Mebibyte (short MiB, 2^20 = 1 048 576), while "MB" stands for Megabyte (10^6 = 1 000 000). Therefore, the use of K/M/G/T/P/E/Z/Y without the 'B' suffix are correct, no? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

18:39

New subject: [opensuse-factory] Re: zypper dup disk space requirement - mebi units

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/02/2019 19.07, Bernhard Voelker wrote:

...

On 2/7/19 3:55 PM, Andreas Schwab wrote:

...
On Feb 07 2019, Bernhard Voelker wrote:

...
You asked df(1) to output with "-human-readable" size, and that's what it did:

-h, --human-readable print sizes in powers of 1024 (e.g., 1023M)

GNU coreutils follows POSIX here, so there's nothing wrong with the output you got:

df -h is an extension, so POSIX doesn't really come into play here.

True. BTW: also the *BSDs have 'df -h' with the same units, so there's some compatibility to consider.

And, "M" without suffix is a Mebibyte (short MiB, 2^20 = 1 048 576), while "MB" stands for Megabyte (10^6 = 1 000 000). Therefore, the use of K/M/G/T/P/E/Z/Y without the 'B' suffix are correct, no?

I don't find a reference for that on the wikipedia. https://en.wikipedia.org/wiki/Mebibyte - -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.0 (Legolas)) -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXFx7eAAKCRC1MxgcbY1H 1fj1AJsEPcTZ9t59et2mt57bQUghL/Nc9ACfa7xeBjbLCz12qGiETbSZHUqgvLY= =QiMU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

8 Feb 8 Feb

00:16

New subject: [opensuse-factory] Re: zypper dup disk space requirement - mebi units

On 2/7/19 7:39 PM, Carlos E. R. wrote:

...

I don't find a reference for that on the wikipedia.

It's documented here: https://www.gnu.org/software/coreutils/manual/html_node/Block-size.html GNU coreutils tools like df, du, ls, etc. are taking that conversion from gnulib ... like other (GNU) packages, e.g. the findutils. Have a nice day, Berny

Jan Engelhardt

00:45

New subject: [opensuse-factory] Re: zypper dup disk space requirement - mebi units

On Friday 2019-02-08 01:16, Bernhard Voelker wrote:

...

On 2/7/19 7:39 PM, Carlos E. R. wrote:

...
I don't find a reference for that on the wikipedia.

It's documented here: https://www.gnu.org/software/coreutils/manual/html_node/Block-size.html

GNU coreutils tools like df, du, ls, etc. are taking that conversion from gnulib ... like other (GNU) packages, e.g. the findutils.

Horizontal space is a premium in text consoles - printing a G instead of GiB feels like an acceptable compromise. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

10:18

New subject: [opensuse-factory] Re: zypper dup disk space requirement - mebi units

On 08/02/2019 01.16, Bernhard Voelker wrote:

...

On 2/7/19 7:39 PM, Carlos E. R. wrote:

...
I don't find a reference for that on the wikipedia.

It's documented here: https://www.gnu.org/software/coreutils/manual/html_node/Block-size.html

GNU coreutils tools like df, du, ls, etc. are taking that conversion from gnulib ... like other (GNU) packages, e.g. the findutils.

No. I'm looking for a documentation that the standard allows usage of G instead of GiB. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)

Carlos E. R.

7 Feb 7 Feb

15:42

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2019-02-07 a las 15:39 +0100, Bernhard Voelker escribió:

...

On 2/7/19 12:47 PM, Carlos E. R. wrote:

...
El 2019-02-07 a las 11:30 +0100, Richard Brown escribió:

...
You're mixing up different units

...
The device is 68G, that is 63GiB in size df says 61G is used with 6.1G free (56GiB used, 5.6GiB free) btrfs fi df / says 60.735GiB is used, therefore just over 2GiB free

Do you want a bug report about 'df' using GiB but saying G? Because it is a bug, the units used should be GiB or GB according the the IEEE and other institutions. If I see 'GiB' I know what it is, if it says 'G' I don't know, if it says 'GB' I have doubts whether they follow the standard or the deprecated standard. >:-)

You asked df(1) to output with "-human-readable" size, and that's what it did:

Was not me :-)

...

-h, --human-readable print sizes in powers of 1024 (e.g., 1023M)

GNU coreutils follows POSIX here, so there's nothing wrong with the output you got:

Then maybe POSIX has to be changed ;-p

...

...
...
So yes, the situation is 'worse' than what df says..or in other words, df spit out useless nonsense that is not suitable for measuring the available space, and really shouldn't be used as the basis for any meaningful discussions

OK! :-)

I disagree: as already stated, BTRFS does not expose the right (or useful?) numbers to the statfs(2) system call, so df(1) just displays what it gets from BTRFS. There's nothing wrong with df(1). Analogy: if you tell 'echo' to print "pig", then it won't print "pork".

Ah... sigh :-( - -- Cheers Carlos E. R. (from openSUSE 15.0 (Legolas)) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXFxRzxwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVqVQAn224z3/BCGPNccVNTpkc 61o+X3nPAJwPBUA8+OvnzwvM3aUEnhXRMn8Jtg== =06K5 -----END PGP SIGNATURE-----

Andrei Borzenkov

17:51

07.02.2019 17:39, Bernhard Voelker пишет:

...

I disagree: as already stated, BTRFS does not expose the right (or useful?) numbers to the statfs(2) system call

Show kernel code where it does it and explain why the value it calculates is not right. This also presumes you know how to calculate it right - did you submit your patch to fix it? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

8 Feb 8 Feb

00:43

On 2/7/19 6:51 PM, Andrei Borzenkov wrote:

...

07.02.2019 17:39, Bernhard Voelker пишет:

...
I disagree: as already stated, BTRFS does not expose the right (or useful?) numbers to the statfs(2) system call

Show kernel code where it does it and explain why the value it calculates is not right. This also presumes you know how to calculate it right - did you submit your patch to fix it?

It has been mentioned repeatedly that a user should run 'btrfs filesystem usage /' to get the "correct" numbers (whatever they are), and: "df spit out useless nonsense". I wouldn't go that far: I don't think the numbers df(1) gets from statfs(2) are totally wrong, but it is obviously not transparent for the user how to interpret them in the context of a BTRFS file system, and incidentally such confusions mostly happen in critical situations near ENOSPC. I don't want to (and don't have time to) fix anything in the kernel - especially wrt/ btrfs - , but I'd be willing to discuss with the other coreutils maintainers to add some snippet to the df documentation or on the FAQ page how a user could interpret BTRFS numbers. What do these numbers actually tell for BTRFS / and what not? Or should it just link to https://btrfs.wiki.kernel.org/index.php/FAQ#Why_is_free_space_so_complicated... ? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andrei Borzenkov

04:26

08.02.2019 3:43, Bernhard Voelker пишет:

...

On 2/7/19 6:51 PM, Andrei Borzenkov wrote:

...
07.02.2019 17:39, Bernhard Voelker пишет:

...
I disagree: as already stated, BTRFS does not expose the right (or useful?) numbers to the statfs(2) system call

Show kernel code where it does it and explain why the value it calculates is not right. This also presumes you know how to calculate it right - did you submit your patch to fix it?

It has been mentioned repeatedly that a user should run 'btrfs filesystem usage /' to get the "correct" numbers (whatever they are), and: "df spit out useless nonsense".

So you do not actually know anything about it, you just repeat what others say.

...

I wouldn't go that far: I don't think the numbers df(1) gets from statfs(2) are totally wrong, but it is obviously not transparent for the user how to interpret them in the context of a BTRFS file system, and incidentally such confusions mostly happen in critical situations near ENOSPC.

I don't want to (and don't have time to) fix anything in the kernel - especially wrt/ btrfs - , but I'd be willing to discuss with the other coreutils maintainers to add some snippet to the df documentation or on the FAQ page how a user could interpret BTRFS numbers.

This discussion is about estimating free space on root filesystem. None of other "BTRFS numbers" will give you anything that plain "df" does not already. Those "BTRFS numbers" may help to answer the question "where did my space go" but that is not the point of this thread.

...

What do these numbers actually tell for BTRFS / and what not? Or should it just link to https://btrfs.wiki.kernel.org/index.php/FAQ#Why_is_free_space_so_complicated... ?

Show me how to create filesystem with different per-subvolume profiles using SUSE installer (or manually for that matter). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

07:11

On 2/8/19 5:26 AM, Andrei Borzenkov wrote:

...

08.02.2019 3:43, Bernhard Voelker пишет:

...
It has been mentioned repeatedly that a user should run 'btrfs filesystem usage /' to get the "correct" numbers (whatever they are), and: "df spit out useless nonsense".

So you do not actually know anything about it, [...]

Indeed, I don't know too much about BTRFS, but...

...

you just repeat what others say.

Au contraire: I'm the only one pointing out that df(1) is not the culprit, and instead it is just outputting the information is gets. If that information is misleading for the user, then either the source of that numbers has to be changed, or - if that is impossible because the df fields 'itotal', 'iused', 'iavail', 'ipcent', 'size', 'used', 'avail', 'pcent' do not make 100% sense in the context of BTRFS - I prefer to amend the documentation to describe those possible pitfalls for the user at least. I am not a BTRFS user nor a kernel hacker; my point of view is that of a GNU coreutils contributor (upstream+downstream). Therefore, I try to fight the over-simplifying "df spits out useless nonsense for BTRFS", and see that the best solution for the GNU coreutils users is to document the specialties for this file-system somewhere. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Joachim Wagner

08:51

...

Au contraire: I'm the only one pointing out that df(1) is not the culprit, and instead it is just outputting the information is gets. If that information is misleading for the user, then either the source of that numbers has to be changed, or - if that is impossible because the df fields 'itotal', 'iused', 'iavail', 'ipcent', 'size', 'used', 'avail', 'pcent' do not make 100% sense in the context of BTRFS - I prefer to amend the documentation to describe those possible pitfalls for the user at least.

I am not a BTRFS user nor a kernel hacker; my point of view is that of a GNU coreutils contributor (upstream+downstream). Therefore, I try to fight the over-simplifying "df spits out useless nonsense for BTRFS", and see that the best solution for the GNU coreutils users is to document the specialties for this file-system somewhere.

Putting "(estimated)" directly next to the value is likely to break user's scripts that extract information from df output. Maybe a footnote in the df output would be acceptable. However, I don't think it's a good idea to maintain code in df that tells it what filesystem can provide what information accurately (or where that field makes sense). The struct the kernel returns should have an indicator bit mask which fields are estimates / best effort replacement values. Furthermore, df and kernel documentation should tell whether itotal should be iavail + iused and size = used + avail or each value can be an independent best effort. Joachim -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

17:11

On 2/8/19 9:51 AM, Joachim Wagner wrote:

...

Furthermore, df and kernel documentation should tell whether itotal should be iavail + iused and size = used + avail or each value can be an independent best effort.

https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#df-Size-and-Us... but this doesn't mention BTRFS specifics ... Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

7 Feb 7 Feb

10:20

Richard Brown wrote:

...

And so, with a bit of math, it's relatively easy to calculate that out of my 444GiB disk, at least 248GiB is still available.

Which of course raises the question why I have to use df or fdisk to get the size of the device and do the computation myself :o ( ;^> Yes I know, file a bug report....) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Richard Brown

10:31

On Thu, 7 Feb 2019 at 11:21, Peter Suetterlin wrote:

...

Richard Brown wrote:

...
And so, with a bit of math, it's relatively easy to calculate that out of my 444GiB disk, at least 248GiB is still available.

Which of course raises the question why I have to use df or fdisk to get the size of the device and do the computation myself :o

( ;^> Yes I know, file a bug report....)

Or, use `btrfs filesystem usage /` like I told you... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Peter Suetterlin

11:06

Richard Brown wrote:

...

On Thu, 7 Feb 2019 at 11:21, Peter Suetterlin wrote:

...

...
Which of course raises the question why I have to use df or fdisk to get the size of the device and do the computation myself :o

( ;^> Yes I know, file a bug report....)

Or, use `btrfs filesystem usage /` like I told you...

Ah, now! Your first recommendation had been 'btrfs fi df', not 'us', that is why I used it. You should rather advertize only the later, and forget about the 'df', as it doesn't really give the information that people using the 'normal' df are looking for.... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Axel Braun

6 Feb 6 Feb

16:06

Am Mittwoch, 6. Februar 2019, 15:08:02 CET schrieb Roger Oberholtzer:

...

...
So for sure checking that there is enough space to download the packages in advance is not sufficient to make sure the upgrade succeeds. Maybe zypper should get some better logic like summing the installed sizes of the packages to be installed and make sure that is available? (At least when snapshots are enabled, that is).

+1

You should vote for https://bugzilla.opensuse.org/show_bug.cgi?id=1124282 (and add ideas to it) Ta Axel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

1910

Age (days ago)

1912

Last active (days ago)

List overview

Download

37 comments

10 participants

participants (10)

Andreas Schwab
Andrei Borzenkov
Axel Braun
Bernhard Voelker
Carlos E. R.
Jan Engelhardt
Joachim Wagner
Peter Suetterlin
Richard Brown
Roger Oberholtzer

[opensuse-factory] zypper dup disk space requirement

tags

participants (10)