Not sure how I can check this, but running this command: sudo ls /var/lib/docker/btrfs/subvolumes lists various subvolumes. I'm pretty sure the docker folder is running on the btrfs partition rather than one of the other various partitions. The worst case approach you just described is the solution I had taken the last time after I had given up. I woud prefer not to nuke my docker installation everytime I rollback as a lot of my docker images require a long bootstrap process to setup at work :/ Is there an inherent problem with the way docker has been setup on btrfs that is causing this issue? Is this something we can fix? Thanks for the reply! Mike On Mon, Sep 18, 2017 at 3:29 AM, Aleksa Sarai <asarai@suse.de> wrote:

...

After a recent rollback I've noticed my docker containers dont want to start. From what I can tell this is related to btrfs since all the messages are all along the lines of:

If this is a snapper rollback this might be caused by some weird btrfs issue. I believe we have a subvolume for /var/lib/docker, but if we don't then a rollback with snapper might have screwed up the layer store.

Worst case scenario you can do something like:

% btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/* % rm -rf /var/lib/docker

Which will delete everything that Docker had stored and you can start "fresh".

-- Aleksa Sarai Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/

-- Michael Aquilina -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

sudo ls /var/lib/docker/btrfs/subvolumes You'd want to do btrfs subvolume list (or something like that). You could try btrfs subvolume create /@/var/lib/docker, and then set up the auto-mounting in fstab, to solve this problem for your install.

I'm pretty sure the docker folder is running on the btrfs partition rather than one of the other various partitions.

Yes, but if it's not in a subvolume (as opposed to living in one of the snapshot subvolumes) then this problem might happen.

The worst case approach you just described is the solution I had taken the last time after I had given up. I woud prefer not to nuke my docker installation everytime I rollback as a lot of my docker images require a long bootstrap process to setup at work :/

Yes, this is something that shouldn't be necessary -- I agree. But if you want to save your images for the migration you could just use docker-load and docker-save for all of your images. You could even use this technique to migrate to a different storage driver.

Is there an inherent problem with the way docker has been setup on btrfs that is causing this issue? Is this something we can fix?

On paper, creating a subvolume for /var/lib/docker should solve the problem. We have this for Kubic but I'm not sure whether we have this for the main distribution. Unfortunately it's non-trivial to migrate to this sort of setup (it might be possible to do that with btrfs-send and btrfs-receive, but I haven't personally done that sort of migration). You could try with docker-save and docker-load though. I've submitted patches to Docker upstream for them to use a subvolume for /var/lib/docker, but those patches have been rejected on the basis of "the distribution/administrator should do the right thing here". Unfortunately due to how our distribution's subvolume policy works (it needs to be set at install time in yast-installation) changing this is non-trivial in my experience. -- Aleksa Sarai Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hello, only a side-note FYI: On Sep 18 10:18 Michael Aquilina wrote (excerpt):

Not sure how I can check this, but running this command: sudo ls /var/lib/docker/btrfs/subvolumes lists various subvolumes.

At least on first glance this looks similar to https://github.com/rear/rear/issues/1496 Simply put: When applications "just create" btrfs subvolumes on a more complicated btrfs subvolume structure (in particular from within a mounted btrfs subvolume) then - usually - things go wrong, see also https://lists.opensuse.org/opensuse-factory/2016-10/msg00397.html Kind Regards Johannes Meixner -- SUSE LINUX GmbH - GF: Felix Imendoerffer, Jane Smithard, Graham Norton - HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org