On Saturday, April 30, 2016 10:38:53 PM CDT Patrick Shanahan wrote:

* Chan Ju Ping [04-30-16 18:17]:

...

I can't seem to activate any of my VPN connections since the massive update. Is anyone experiencing the same issue?

see thread here from yesterday, "lost ssh from 13.1 server" maybe same problem.

Never mind. I figured out the issue when X failed to launch. I ran out of disk space due to the huge update. After deleting all Snapshots prior to today with Snapper, and doing a full balance for the btrfs /, everything is now back to normal. This is the 3rd time this has happened though since I started using TW. Someone should figure out a reliable way to avoid it.

On 2016-05-01 05:39, Chan Ju Ping wrote:

On Saturday, April 30, 2016 10:38:53 PM CDT Patrick Shanahan wrote:

After deleting all Snapshots prior to today with Snapper, and doing a full balance for the btrfs /, everything is now back to normal.

This is the 3rd time this has happened though since I started using TW. Someone should figure out a reliable way to avoid it.

Probably your "/" filesystem is too small. What is its size? Also, TW gets frequent and biggish updates, so that snapshots fill faster and have less time to be phased out. I'm not that familiar with snapshots, but I suppose there is some kind of cronjob deleting old snapshots. maybe on TW the strategy should be more aggressive, or the disk bigger. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2016-05-01 19:29, Chan Ju Ping wrote:

On Sunday, May 1, 2016 2:56:30 PM CDT Carlos E. R. wrote:

...

Also, TW gets frequent and biggish updates, so that snapshots fill faster and have less time to be phased out. I'm not that familiar with snapshots, but I suppose there is some kind of cronjob deleting old snapshots. maybe on TW the strategy should be more aggressive, or the disk bigger.

My / partition is 40 GB in size, and once rebalanced typically has 13 GB of free space. Plenty enough I would presume.

No, not for btrfs with snapshots. My guess is that you need about the same free space as twice the space needed by the files without any snapshot. At least you need the same or more free space as the applied updates. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On Sunday, May 1, 2016 6:04:17 PM CDT Roman Bysh wrote:

On 05/01/2016 04:45 PM, Carlos E. R. wrote:

...

On 2016-05-01 19:29, Chan Ju Ping wrote:

...

On Sunday, May 1, 2016 2:56:30 PM CDT Carlos E. R. wrote:

...

Also, TW gets frequent and biggish updates, so that snapshots fill faster and have less time to be phased out. I'm not that familiar with snapshots, but I suppose there is some kind of cronjob deleting old snapshots. maybe on TW the strategy should be more aggressive, or the disk bigger.

My / partition is 40 GB in size, and once rebalanced typically has 13 GB of free space. Plenty enough I would presume.

No, not for btrfs with snapshots. My guess is that you need about the same free space as twice the space needed by the files without any snapshot. At least you need the same or more free space as the applied updates.

I'm using 90 GB for / with no problems.

Then perhaps the recommended settings for / at installation should be a lot larger than it is at present.

On Monday, May 2, 2016 10:50:49 AM CDT Simon Lees wrote:

I tend to agree here, I hit this as well with basically a default installation on a month old laptop, if no one tells me they've raised a bug about this in the next 24 hrs i'll create one.

Cheers

If you file the bug, do put the link here, so I can add myself to it. Basically, it is easier to completely reinstall the system rather than fudge around with resizing the encrypted lvm partitions I now have in place. It would have been more convenient if any of the initial setup screens had mentioned how much space btrfs should have been given.

On Tue, 3 May 2016 01:18, Matthias G. Eckermann wrote:

On 2016-05-02 T 16:06 -0500 Chan Ju Ping wrote:

...

On Monday, May 2, 2016 10:50:49 AM CDT Simon Lees wrote:

...

...

I tend to agree here, I hit this as well with basically a default installation on a month old laptop, if no one tells me they've raised a bug about this in the next 24 hrs i'll create one.

Cheers

If you file the bug, do put the link here, so I can add myself to it. Basically, it is easier to completely reinstall the system rather than fudge around with resizing the encrypted lvm partitions I now have in place. It would have been more convenient if any of the initial setup screens had mentioned how much space btrfs should have been given.

This question - how much space to reserve for snapshots - depends on multiple factors: - the size of your installation - the update behaviour of the distribution you are using (Tumbleweed and LEAP / SLE 12 have fundamentally different patterns) - the default settings for the number of snapshots - the cleanup policies, and in this context probably also the question, if the cleanup policies are executed at all (and not the system is sleeping or off)

In general, it is a good general advice, to 1. Execute the snapshot cleanup daily 2. Execute "btrfsmaintenance" daily (with a lightweight policy) 3. Reserve three times more space than the OS currently uses. This already has a safety buffer, and allows for a complete distribution update of the OS plus metadata.

Addenum to point 3: For TW the recommention of three times the space in use is a little short, but mostly useable for a non-snapshot fs. For a snapshot fs (here btrfs), do your self a big favor and do not go below 4 times the projected used space. Here is why so much more for snapshotting: A TW, fresh installed, no packages kept by zypper, takes for example 12-16 GB on disk (X11, XFCE / KDE / GNOME, Office pattern). Now on Btrfs a snapshot happens, due to cow and hardlinks, no additional space is taken for non-changed files. Next TW-upgrade comes: User calls zypper dup 1. zypper will snapshot (pre update), 2. zypper downloads all the packages (this takes extra space not needed later, if the packages are not kept), and installs them, 3. second snapshot (post update) happens. The second snapshot is big, as it contains all the changed / added files. AFAICT, the paired snapshots for a update will take up about the unpacked size of the updates each. Repeat that circle a few weeks (up to 4 TW Releases per week) and without manual intervention, you have a overflowing root-fs, with all the errors and trouble. A full rebuild as it happend last week is then just the cherry on top of the disaster waiting to happen. If a user wants to 'keep packages' in zypper, the problem beocmes worse, fast. Either zypper snapshotting becomes a added functionality to clean up / remove old 'zypper update' snapshots, or zypper dup should give a heavy warning if there is less than 2.5 times the size of the update free, before the download is started. IMHO, either we get the tools in shape, and give propper defaults for rootfs size ( e.g. min 60GB with btrfs ) or will continue to get reports of 'no space left' for rootfs, or more users will become disatisfied with the state of TW and / or (open)SUSE in general. - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 3 May 2016 at 18:50, Daniele wrote:

Il 03/05/2016 11:57, Yamaban ha scritto:

...

Next TW-upgrade comes: User calls zypper dup 1. zypper will snapshot (pre update), 2. zypper downloads all the packages (this takes extra space not needed later, if the packages are not kept), and installs them, 3. second snapshot (post update) happens.

The second snapshot is big, as it contains all the changed / added files.

Hi, is the second snapshot really needed ? I think that is useful only in some corner case..

Daniele.

As these are COW snapshots there is no additional space used from the second snapshot compared to the running system at that time. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 2016-05-03 T 19:02 +0200 Daniele wrote:

Il 03/05/2016 18:57, Richard Brown ha scritto:

...

On 3 May 2016 at 18:50, Daniele wrote:

...

Il 03/05/2016 11:57, Yamaban ha scritto:

...

Next TW-upgrade comes: User calls zypper dup 1. zypper will snapshot (pre update), 2. zypper downloads all the packages (this takes extra space not needed later, if the packages are not kept), and installs them, 3. second snapshot (post update) happens.

The second snapshot is big, as it contains all the changed / added files.

Hi, is the second snapshot really needed ? I think that is useful only in some corner case..

Daniele.

As these are COW snapshots there is no additional space used from the second snapshot compared to the running system at that time.

But is big compared to the previous one or not ? Anyway, what is for ?

As described above, the first snapshot is the status "BEFORE Update" also called "pre". The second snapshot determines the status directly AFTER the update, also called "post". Directly after this snapshot, the status of the updated system starts changing. The primary purpose of snapshots is, though, to ever have a point to "jump back to", also known as "well known state". If there would not be a (second) snapshot directly after the update, your only "well known state" would be _before_ the update, and this is not necessarily what you want to have. As Richard said, a snapshot, where there is no CoW action involved costs basically nothing; exactly it is 12 KiB (3 blocks), if I am not mistaken. Thus taking the second snapshot is a good choice. ---- Now, back to the topic of "free space" (or more precisely lack thereof): Reading the above, I wonder, if we may have a race condition in the ZYpp stack, where the downloaded packages are deleted _after_ the second snapshot. This indeed would explain: - Why the second snapshot _is_ huge - Why Tumbleweed users have a more pressing space issue than e.g. SLE 12 users (such as myself). So long - MgE -- Matthias G. Eckermann, Director Product Management SUSE Linux Enterprise SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hello Andrei, On 2016-05-04 T 12:46 +0300 Andrei Borzenkov wrote:

On Wed, May 4, 2016 at 2:07 AM, Matthias G. Eckermann wrote:

...

Reading the above, I wonder, if we may have a race condition in the ZYpp stack, where the downloaded packages are deleted _after_ the second snapshot. This indeed would explain: - Why the second snapshot _is_ huge - Why Tumbleweed users have a more pressing space issue than e.g. SLE 12 users (such as myself).

Is there any reason why /var/cache/zypp (or even /var/cache as a whole) is not subvolume?

you have a valid point here: We probably just missed this (/var/cache sounds correct). Let me add Thorsten Kukuk who as our architect looked into the challenges of snapshot-rollback quite a bit and might see issues with this approach which I might not see, currently. So long MgE -- Matthias G. Eckermann, Director Product Management SUSE Linux Enterprise SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 03.05.16 18:50 Daniele wrote:

Il 03/05/2016 11:57, Yamaban ha scritto:

...

Next TW-upgrade comes: User calls zypper dup 1. zypper will snapshot (pre update), 2. zypper downloads all the packages (this takes extra space not needed later, if the packages are not kept), and installs them, 3. second snapshot (post update) happens.

The second snapshot is big, as it contains all the changed / added files.

Hi, is the second snapshot really needed ? I think that is useful only in some corner case..

Don't get confused (as I was at first glance), the second snapshot is actually point 3. Which is useful, as this is the state directly after all the updates being installed. Point 2 is not creating a snapshot. Johannes

On 05/04/2016 02:55 AM, Chris Murphy wrote:

On Tue, May 3, 2016 at 3:57 AM, Yamaban wrote:

...

Here is why so much more for snapshotting:

A TW, fresh installed, no packages kept by zypper, takes for example 12-16 GB on disk (X11, XFCE / KDE / GNOME, Office pattern).

Kinda old now but openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20160307-Media.iso default partitioning and installation on a 1TiB device results in:

Filesystem Size Used Avail Use% Mounted on /dev/vda2 41G 3.7G 35G 10% /

Off hand this doesn't seem unreasonable. So I don't know where the problem is that some people are running into where they run out of space. I suspect it's some combination of snapper triggering on used% that's too high, and btrfsmaintenance-refresh.service being disabled by default.

However, looking at /usr/share/btrfsmaintenance.sh it clearly needs updating.

- snapshot aware defrag was pulled out of btrfs a while ago due to problems, so I question the value and appropriateness of btrfs-defrag.sh being run on a regular basis; - btrfs-trim.sh is obsoleted by fstrim.timer which is enabled by default, there's no good reason to run both of these; - btrfs-balance.sh uses filters -dusage=0 and -musage=0 which is now handled by the kernel, this should probably be something like -dusage=5 and -musage=15 to consolidate extents from minimally used chunks and then revert them to unallocated space.

That this service isn't up to date is probably a good reason why it's not enabled by default right now, but then there's nothing to keep Btrfs from hitting enospc prematurely (until that work is done in the kernel which of course will happen eventually).

...

Now on Btrfs a snapshot happens, due to cow and hardlinks, no additional space is taken for non-changed files.

Next TW-upgrade comes: User calls zypper dup 1. zypper will snapshot (pre update), 2. zypper downloads all the packages (this takes extra space not needed later, if the packages are not kept), and installs them, 3. second snapshot (post update) happens.

The second snapshot is big, as it contains all the changed / added files.

AFAICT, the paired snapshots for a update will take up about the unpacked size of the updates each.

Snapshots don't take up space. They prevent files from being deleted, or more correctly, they cause file extents to be retained. So when updates happen that cause old binaries to be deleted in one subvolume, their extents aren't actually freed because the file exists in another subvolume still.

...

Repeat that circle a few weeks (up to 4 TW Releases per week) and without manual intervention, you have a overflowing root-fs, with all the errors and trouble.

Something else is going on. It's not the snapshot itself that's the problem, nor the default root fs size. Those are contributing factors in the problem revealing itself sooner than later, so if all that's done is increase root fs by 4x, you'll postpone the same exact problem for later, rather than actually solve the problem.

...

IMHO, either we get the tools in shape, and give propper defaults for rootfs size ( e.g. min 60GB with btrfs ) or will continue to get reports of 'no space left' for rootfs, or more users will become disatisfied with the state of TW and / or (open)SUSE in general.

When users report no space left / enospc, they need to be asked to report 'btrfs fi us' or at least 'btrfs fi df' if they're using older user space tools. Only that way can we have some idea which type of enospc is being hit and maybe what to do about it.

But it's true that the default out of the box experience should not result in face planted file systems that have to be reinstalled.

Fortunately in my case I realised what was wrong and was still able to use snapper from the commandline to delete a couple of snapshots and continue the update so I didn't need a reinstall, but there must be a way we can detect its probably going to happen by looking at the size of the packages were installing and the amount of freespace and warn users they should tidy up there snapshots before updating. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B