[Bug 959686] New: btrfs:snapper: cannot delete subvolume messed up by docker
http://bugzilla.suse.com/show_bug.cgi?id=959686 Bug ID: 959686 Summary: btrfs:snapper: cannot delete subvolume messed up by docker Classification: openSUSE Product: openSUSE Distribution Version: Leap 42.1 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: bnc-team-screening@forge.provo.novell.com Reporter: zren@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- After coming across this issue, I've discussed with Arvin Schnell. Finally, Arvin found it's related with docker, yeah I truely tried docker things on leap42.1, and he suggested opening a bug. I will paste each email as a comment here, if more information needed, I'd like to provide further. Hi Arvin, First, thanks for your brilliant tool - snapper. It really saved my system days ago. Now, my system is on good condition, so I want to clear useless snapshots. I did the following, and am wondering why can't I delete "single" type snapshots? If so, I think it's better to mention this in "man snapper". I don't know if I'm asking a stupid question ;-) 1. section of manual ... delete (remove|rm) number | number1-number2 Delete a snapshot or a range of snapshots. ... 2. delete snapshots eric@laptop:~> sudo snapper list -a root's password: Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+-----+-------+---------------------------------+------+---------+-------------+--------- single | 0 | | | root | | current | single | 101 | | Mon 24 Aug 2015 09:42:27 PM CST | root | | | single | 182 | | Fri 06 Nov 2015 10:06:15 PM CST | root | | | single | 183 | | Fri 06 Nov 2015 10:06:18 PM CST | root | | | eric@laptop:~> sudo snapper delete 101 Deleting snapshot failed. PS. deleting other type of snapshots work well. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c1
--- Comment #1 from zhen ren
Hi Arvin,
First, thanks for your brilliant tool - snapper. It really saved my system days ago.
Now, my system is on good condition, so I want to clear useless snapshots. I did the following, and am wondering why can't I delete "single" type snapshots? If so, I think it's better to mention this in "man snapper". I don't know if I'm asking a stupid question ;-)
1. section of manual ... delete (remove|rm) number | number1-number2 Delete a snapshot or a range of snapshots. ...
2. delete snapshots
eric@laptop:~> sudo snapper list -a root's password: Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+-----+-------+---------------------------------+------+---------+-------------+--------- single | 0 | | | root | | current | single | 101 | | Mon 24 Aug 2015 09:42:27 PM CST | root | | | single | 182 | | Fri 06 Nov 2015 10:06:15 PM CST | root | | | single | 183 | | Fri 06 Nov 2015 10:06:18 PM CST | root | | | eric@laptop:~> sudo snapper delete 101 Deleting snapshot failed.
Maybe you made a rollback and now your system is in snapshot 101. Try 'btrfs subvolume get-default /'. Otherwise snapper.log in /var/log might have more information. Regards, Arvin -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c2
--- Comment #2 from zhen ren
eric@laptop:~> sudo snapper list -a root's password: Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+-----+-------+---------------------------------+------+---------+-------------+--------- single | 0 | | | root | | current | single | 101 | | Mon 24 Aug 2015 09:42:27 PM CST | root | | | single | 182 | | Fri 06 Nov 2015 10:06:15 PM CST | root | | | single | 183 | | Fri 06 Nov 2015 10:06:18 PM CST | root | | |
eric@laptop:~> sudo snapper delete 101 Deleting snapshot failed.
Maybe you made a rollback and now your system is in snapshot 101.
Try 'btrfs subvolume get-default /'. `btrfs subvolume get-default /` get me this: ID 491 gen 112548 top level 276 path .snapshots/183/snapshot
Otherwise snapper.log in /var/log might have more information.
eric@laptop:~> sudo snapper delete 101 Deleting snapshot failed. eric@laptop:~> sudo tail /var/log/snapper.log 2015-12-16 22:29:32 MIL libsnapper(3269) AsciiFile.cc(getValue):235 - key:SUBVOLUME value:/ 2015-12-16 22:29:32 MIL libsnapper(3269) AsciiFile.cc(getValue):235 - key:FSTYPE value:btrfs 2015-12-16 22:29:32 MIL libsnapper(3269) AsciiFile.cc(getValue):235 - key:SYNC_ACL value:no 2015-12-16 22:29:32 MIL libsnapper(3269) Snapper.cc(Snapper):103 - subvolume:/ filesystem:btrfs 2015-12-16 22:29:32 MIL libsnapper(3269) Snapper.cc(loadIgnorePatterns):152 - number of ignore patterns:8 2015-12-16 22:29:32 ERR libsnapper(3269) Snapshot.cc(read):217 - loading 175 failed 2015-12-16 22:29:32 ERR libsnapper(3269) Snapshot.cc(read):217 - loading 176 failed 2015-12-16 22:29:32 MIL libsnapper(3269) Snapshot.cc(read):223 - found 12 snapshots 2015-12-16 22:29:32 ERR libsnapper(3269) Btrfs.cc(deleteSnapshot):366 - delete snapshot failed, ioctl(BTRFS_IOC_SNAP_DESTROY) failed, errno:39 (Directory not empty) 2015-12-16 22:29:32 WAR libsnapper(3269) Client.cc(dispatch):1518 - CAUGHT: delete snapshot failed Thanks, Eric -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c3
--- Comment #3 from zhen ren
2015-12-16 22:29:32 ERR libsnapper(3269) Btrfs.cc(deleteSnapshot):366 - delete snapshot failed, ioctl(BTRFS_IOC_SNAP_DESTROY) failed, errno:39 (Directory not empty)
So the kernel cannot delete the snapshot since it is not empty. Likely you created a subvolume/snapshot inside the snapshot. Try to list the btrfs subvolumes along with parent information to figure out the problem. Regards, Arvin -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c4
--- Comment #4 from zhen ren
On Wed, Dec 16, 2015 at 10:38:41PM +0800, Eric Ren wrote:
2015-12-16 22:29:32 ERR libsnapper(3269) Btrfs.cc(deleteSnapshot):366 - delete snapshot failed, ioctl(BTRFS_IOC_SNAP_DESTROY) failed, errno:39 (Directory not empty)
So the kernel cannot delete the snapshot since it is not empty. Likely you created a subvolume/snapshot inside the By empty, what does it exactly mean? snapshot. Yeah, I did boot up into a readonly snapshot and run "snapper rollback" which get me a writable system. All of this like magic to me, hah...
Try to list the btrfs subvolumes along with parent information to figure out the problem.
Not sure if this information helps: --- eric@laptop:~> sudo snapper list -a Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+-----+-------+---------------------------------+------+---------+-------------------+------------- single | 0 | | | root | | current | single | 101 | | Mon 24 Aug 2015 09:42:27 PM CST | root | | | single | 183 | | Fri 06 Nov 2015 10:06:18 PM CST | root | | | pre | 184 | | Mon 14 Dec 2015 03:59:11 PM CST | root | number | zypp(zypper) | important=no post | 185 | 184 | Mon 14 Dec 2015 04:00:21 PM CST | root | number | | important=no pre | 186 | | Mon 14 Dec 2015 04:01:00 PM CST | root | number | zypp(zypper) | important=no post | 187 | 186 | Mon 14 Dec 2015 04:01:03 PM CST | root | number | | important=no pre | 188 | | Mon 14 Dec 2015 11:30:39 PM CST | root | number | zypp(packagekitd) | important=no post | 189 | 188 | Mon 14 Dec 2015 11:32:07 PM CST | root | number | | important=no pre | 190 | | Tue 15 Dec 2015 03:25:14 PM CST | root | number | zypp(zypper) | important=no post | 191 | 190 | Tue 15 Dec 2015 03:25:19 PM CST | root | number | | important=no eric@laptop:~> sudo btrfs subvolume list -p / ID 257 gen 108975 parent 5 top level 5 path boot/grub2/i386-pc ID 258 gen 108975 parent 5 top level 5 path boot/grub2/x86_64-efi ID 259 gen 108975 parent 5 top level 5 path home ID 260 gen 112730 parent 5 top level 5 path opt ID 261 gen 110054 parent 5 top level 5 path srv ID 262 gen 112731 parent 5 top level 5 path tmp ID 263 gen 112730 parent 5 top level 5 path usr/local ID 264 gen 108975 parent 5 top level 5 path var/crash ID 265 gen 108975 parent 5 top level 5 path var/lib/mailman ID 266 gen 108975 parent 5 top level 5 path var/lib/named ID 267 gen 108975 parent 5 top level 5 path var/lib/pgsql ID 268 gen 112731 parent 5 top level 5 path var/log ID 269 gen 108975 parent 5 top level 5 path var/opt ID 270 gen 112730 parent 5 top level 5 path var/spool ID 271 gen 112730 parent 5 top level 5 path var/tmp ID 276 gen 112567 parent 5 top level 5 path .snapshots ID 385 gen 86877 parent 276 top level 276 path .snapshots/101/snapshot ID 453 gen 108975 parent 385 top level 385 path .snapshots/101/snapshot/var/lib/docker/btrfs/subvolumes/5b8dd08af89b84564b09fc0dfe96cfdfee0ff650d50cf4f3683c6367a1295317 ID 455 gen 80459 parent 385 top level 385 path .snapshots/101/snapshot/var/lib/docker/btrfs/subvolumes/085130dff90935a5727d4e4a6b3c7f2cd6996fc3e053003d2a1bce92ab36408d ID 483 gen 86756 parent 276 top level 276 path .snapshots/175/snapshot ID 484 gen 86757 parent 276 top level 276 path .snapshots/176/snapshot ID 491 gen 112731 parent 276 top level 276 path .snapshots/183/snapshot\ ID 570 gen 110052 parent 276 top level 276 path .snapshots/184/snapshot ID 571 gen 110056 parent 276 top level 276 path .snapshots/185/snapshot ID 572 gen 110058 parent 276 top level 276 path .snapshots/186/snapshot ID 573 gen 110059 parent 276 top level 276 path .snapshots/187/snapshot ID 574 gen 110280 parent 276 top level 276 path .snapshots/188/snapshot ID 575 gen 110283 parent 276 top level 276 path .snapshots/189/snapshot ID 576 gen 110685 parent 276 top level 276 path .snapshots/190/snapshot ID 577 gen 110686 parent 276 top level 276 path .snapshots/191/snapshot Thanks, Eric -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c5
--- Comment #5 from zhen ren
eric@laptop:~> sudo btrfs subvolume list -p / ID 257 gen 108975 parent 5 top level 5 path boot/grub2/i386-pc ID 258 gen 108975 parent 5 top level 5 path boot/grub2/x86_64-efi ID 259 gen 108975 parent 5 top level 5 path home ID 260 gen 112730 parent 5 top level 5 path opt ID 261 gen 110054 parent 5 top level 5 path srv ID 262 gen 112731 parent 5 top level 5 path tmp ID 263 gen 112730 parent 5 top level 5 path usr/local ID 264 gen 108975 parent 5 top level 5 path var/crash ID 265 gen 108975 parent 5 top level 5 path var/lib/mailman ID 266 gen 108975 parent 5 top level 5 path var/lib/named ID 267 gen 108975 parent 5 top level 5 path var/lib/pgsql ID 268 gen 112731 parent 5 top level 5 path var/log ID 269 gen 108975 parent 5 top level 5 path var/opt ID 270 gen 112730 parent 5 top level 5 path var/spool ID 271 gen 112730 parent 5 top level 5 path var/tmp ID 276 gen 112567 parent 5 top level 5 path .snapshots ID 385 gen 86877 parent 276 top level 276 path .snapshots/101/snapshot ID 453 gen 108975 parent 385 top level 385 path .snapshots/101/snapshot/var/lib/docker/btrfs/subvolumes/5b8dd08af89b84564b09fc0dfe96cfdfee0ff650d50cf4f3683c6367a1295317 ID 455 gen 80459 parent 385 top level 385 path .snapshots/101/snapshot/var/lib/docker/btrfs/subvolumes/085130dff90935a5727d4e4a6b3c7f2cd6996fc3e053003d2a1bce92ab36408d
Apparently docker created subvolumes in a way that does not work on SUSE. I suggest to make a bug report against docker. Regards, Arvin -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=959686
Arvin Schnell
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c6
Chenzi Cao
http://bugzilla.suse.com/show_bug.cgi?id=959686
http://bugzilla.suse.com/show_bug.cgi?id=959686#c8
zhen ren
So it looks like Docker creates snapshots inside "snapshot 1" of snapper on my system (I'm on Tumbleweed). Can you tell me what version of Docker you are using?
Hi Aleksa, eric@laptop:~> docker --version Docker version 1.9.1, build a34a1d5 eric@laptop:~> cat /etc/issue Welcome to openSUSE Leap 42.1 - Kernel \r (\l). Sorry for reply very lately. I'd like to close this bug since you cannot reproduce on your side. We can reopen if it happens again. Thanks a lot! -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com