[Bug 1144782] New: File system filling up? (usr/lib/YaST2/bin/cut-messages: cpio: rename failed - No space left on device)
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782 Bug ID: 1144782 Summary: File system filling up? (usr/lib/YaST2/bin/cut-messages: cpio: rename failed - No space left on device) Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.1 Hardware: Other OS: Other Status: NEW Severity: Major Priority: P5 - None Component: Basesystem Assignee: bnc-team-screening@forge.provo.novell.com Reporter: Ulrich.Windl@rz.uni-regensburg.de QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- SHortly after having upgraded from openSUSE Leap 42.3 to 15.1 I'm seeing message sthat my filesystem is full. Actually it is getting filled for a reason unknown yet, but it seems I cannot install any more packages because of an error like this: error: unpacking of archive failed on file /usr/lib/YaST2/bin/cut-messages: cpio: rename failed - No space left on device Why does "rename" need space on the device? A BtrFS feature? # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1971948 8 1971940 1% /dev tmpfs 2014916 17952 1996964 1% /dev/shm tmpfs 2014916 1820 2013096 1% /run tmpfs 2014916 0 2014916 0% /sys/fs/cgroup /dev/mapper/sys-root 34603008 33858648 558520 99% / /dev/mapper/sys-root 34603008 33858648 558520 99% /srv /dev/mapper/sys-root 34603008 33858648 558520 99% /.snapshots /dev/mapper/sys-root 34603008 33858648 558520 99% /usr/local /dev/mapper/sys-root 34603008 33858648 558520 99% /tmp /dev/mapper/sys-root 34603008 33858648 558520 99% /opt /dev/sdb1 982840 86572 791412 10% /boot /dev/mapper/sys-var 8388608 2336708 5216572 31% /var /dev/mapper/sys-var 8388608 2336708 5216572 31% /var/lib/machines /dev/mapper/cr_home 262013956 242132632 19881324 93% /home tmpfs 402980 8 402972 1% /run/user/483 tmpfs 402980 48 402932 1% /run/user/1000 The package in question is yast2-network-4.1.51-lp151.2.3.1.noarch.rpm which should be less than 1MB in size. Error details (from rpm -vvU ...): D: Plugin: calling hook fsm_file_prepare in ima plugin D: Plugin: calling hook fsm_file_prepare in selinux plugin fdio: 5 reads, 670 total bytes in 0.000084 secs error: unpacking of archive failed on file /usr/lib/YaST2/bin/cut-messages: cpio: rename failed - No space left on device D: Plugin: calling hook psm_post in syslog plugin ufdio: 6 reads, 54752 total bytes in 0.000021 secs error: yast2-network-4.1.51-lp151.2.3.1.noarch: install failed -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c1
--- Comment #1 from Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c3
--- Comment #3 from Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c4
--- Comment #4 from Ulrich Windl
Being rather desperate I had made a fatal mistake: Trying to clean up space I removed snapshot "1" (which I thought it was the oldest). However after doing so, essential executables from /usr had vanished!
Accidentally I found out that on one SLES15 system it seems that snapshot #1 was mounted as root filesystem; if that was the case here also, that would explain what had happened: /dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot) ... /dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/.snapshots) ... /dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/root) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c6
Ulrich Windl
That seems to be OK, on my system I have
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot)
[...] Why should it be OK if snapshot #1 is mounted as / (root)? On my fixed system I have: # cat /proc/mounts |grep root /dev/mapper/sys-root / btrfs rw,relatime,space_cache,subvolid=256,subvol=/@ 0 0 /dev/mapper/sys-root /opt btrfs rw,relatime,space_cache,subvolid=258,subvol=/@/opt 0 0 /dev/mapper/sys-root /srv btrfs rw,relatime,space_cache,subvolid=261,subvol=/@/srv 0 0 /dev/mapper/sys-root /usr/local btrfs rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local 0 0 /dev/mapper/sys-root /tmp btrfs rw,relatime,space_cache,subvolid=260,subvol=/@/tmp 0 0 /dev/mapper/sys-root /.snapshots btrfs rw,relatime,space_cache,subvolid=274,subvol=/@/.snapshots 0 0 This looks correct, not what I had before! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c12
--- Comment #12 from Ulrich Windl
This is the default config when using when installing Leap15.1 using snapper snapshots / rollback:
linux-wfhj:~ # cat /etc/fstab UUID=e637ab3e-e230-47ea-9f43-66f02385aaf7 / btrfs defaults 0 0 [...] linux-wfhj:~ # btrfs subvolume get-default / ID 268 gen 174 top level 267 path @/.snapshots/1/snapshot
Here I have (the latest snapshot has #67, however) ID 256 gen 2418 top level 5 path @
linux-wfhj:~ # btrfs subvolume delete /.snapshots/1/snapshot Delete subvolume (no-commit): '/.snapshots/1/snapshot' ERROR: Could not destroy subvolume/snapshot: Operation not permitted
linux-wfhj:~ # mount | grep snapshots /dev/vda2 on / type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot) /dev/vda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots) linux-wfhj:~ #
I have /dev/mapper/sys-root on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=274,subvol=/@/.snapshots) which is different.
So the root-fs is mounted in the default subvolume which could be any subvolume and it not specified explicitly in the fstab entry but just the defaults are used.
SO, not sure how you removed the default snapshot, but seems if you removed it was not the default at that point somehow.
I was using "rm -r".
At this point I'm not sure what the request is about, or what we should modify / fix at all.
Basically the request was that the filesystem filled up without finding put why, or what to do if it does. As I was unable to perform any useful work with the computer, I tried to fix things (which was the wrong way to do it.
BTW: I have involved Arvin as he is the expert in the area.
(In reply to Ulrich Windl from comment #3)
Anyway the filesystem after my recovery was only 44% filled, and currently (with about 15 snapshots) the filesystem is 55% filled.
Since then the filesystem filled to 66%, or about 5% (or almost 2GB) per month. Snapshots are (no idea where the gaps come from): # du -sh /.snapshots/[0-9]* 11G /.snapshots/1 11G /.snapshots/2 12G /.snapshots/3 11G /.snapshots/11 11G /.snapshots/12 11G /.snapshots/47 12G /.snapshots/48 12G /.snapshots/56 12G /.snapshots/57 12G /.snapshots/58 12G /.snapshots/59 12G /.snapshots/60 12G /.snapshots/61 12G /.snapshots/62 12G /.snapshots/63 12G /.snapshots/64 12G /.snapshots/65 12G /.snapshots/66 12G /.snapshots/67 These are the dates: # ll -d /.snapshots/[0-9]* drwxr-xr-x 1 root root 32 Aug 20 12:50 /.snapshots/1 drwxr-xr-x 1 root root 32 Aug 20 12:56 /.snapshots/2 drwxr-xr-x 1 root root 60 Aug 20 17:27 /.snapshots/3 drwxr-xr-x 1 root root 32 Aug 20 17:48 /.snapshots/11 drwxr-xr-x 1 root root 62 Aug 20 17:50 /.snapshots/12 drwxr-xr-x 1 root root 32 Oct 7 09:45 /.snapshots/47 drwxr-xr-x 1 root root 62 Oct 7 10:00 /.snapshots/48 drwxr-xr-x 1 root root 32 Oct 10 08:57 /.snapshots/56 drwxr-xr-x 1 root root 32 Oct 10 09:03 /.snapshots/57 drwxr-xr-x 1 root root 62 Oct 10 09:03 /.snapshots/58 drwxr-xr-x 1 root root 62 Oct 10 09:04 /.snapshots/59 drwxr-xr-x 1 root root 32 Oct 14 14:49 /.snapshots/60 drwxr-xr-x 1 root root 32 Oct 14 14:56 /.snapshots/61 drwxr-xr-x 1 root root 62 Oct 14 14:57 /.snapshots/62 drwxr-xr-x 1 root root 62 Oct 14 14:57 /.snapshots/63 drwxr-xr-x 1 root root 32 Oct 16 15:29 /.snapshots/64 drwxr-xr-x 1 root root 32 Oct 16 15:30 /.snapshots/65 drwxr-xr-x 1 root root 62 Oct 16 15:30 /.snapshots/66 drwxr-xr-x 1 root root 62 Oct 16 15:30 /.snapshots/67 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c13
--- Comment #13 from Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c15
--- Comment #15 from Ulrich Windl
I suggest to read about snapshots in btrfs, esp. the space used by snapshots (and how to measure the space using qgroups). Using 'du' is not suitable here.
One resource is http://snapper.io/2018/10/04/used-space.html.
I was aware that df is not the best tool to calculate the size being used for snapshots, but using "snapper --iso list" doesn't show the space used either (in contrast to the examples shown in that URL): # snapper --iso list # | Type | Pre # | Date | User | Cleanup | Description | Userdata ----+--------+-------+---------------------+------+---------+--------------------+-------------- 0 | single | | | root | | current | 1 | pre | | 2019-08-20 12:50:23 | root | number | before update | important=yes 2 | pre | | 2019-08-20 12:56:27 | root | number | before update | important=yes 3 | post | 2 | 2019-08-20 13:14:40 | root | number | after update | important=yes 11 | pre | | 2019-08-20 17:37:07 | root | number | zypp(ruby.ruby2.5) | important=yes 12 | post | 11 | 2019-08-20 17:48:13 | root | number | | important=yes 47 | pre | | 2019-10-07 09:37:24 | root | number | zypp(ruby.ruby2.5) | important=yes 48 | post | 47 | 2019-10-07 09:45:48 | root | number | | important=yes 56 | pre | | 2019-10-10 08:57:00 | root | number | yast online_update | 57 | pre | | 2019-10-10 09:00:04 | root | number | zypp(ruby.ruby2.5) | important=no 58 | post | 57 | 2019-10-10 09:03:39 | root | number | | important=no 59 | post | 56 | 2019-10-10 09:03:46 | root | number | | 60 | pre | | 2019-10-14 14:49:18 | root | number | yast online_update | 61 | pre | | 2019-10-14 14:51:05 | root | number | zypp(ruby.ruby2.5) | important=yes 62 | post | 61 | 2019-10-14 14:56:48 | root | number | | important=yes 63 | post | 60 | 2019-10-14 14:57:04 | root | number | | 64 | pre | | 2019-10-16 15:29:17 | root | number | yast online_update | 65 | pre | | 2019-10-16 15:29:48 | root | number | zypp(ruby.ruby2.5) | important=no 66 | post | 65 | 2019-10-16 15:30:32 | root | number | | important=no 67 | post | 64 | 2019-10-16 15:30:38 | root | number | | 68 | pre | | 2019-10-22 13:49:50 | root | number | yast online_update | 69 | pre | | 2019-10-22 13:50:38 | root | number | zypp(ruby.ruby2.5) | important=no 70 | post | 69 | 2019-10-22 13:53:59 | root | number | | important=no 71 | post | 68 | 2019-10-22 13:54:06 | root | number | | And the "Description" leaves room for improvements. I also don't understand the comment referring to "qgroups": # btrfs qgroup show / ERROR: can't list qgroups: quotas not enabled -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c18
--- Comment #18 from Ulrich Windl
Yep, and basically because you mount the / explicitly in the /@ subvol, then not in the btrfs default subvolume as you commented previously:
--- # cat /proc/mounts |grep root /dev/mapper/sys-root / btrfs rw,relatime,space_cache,subvolid=256,subvol=/@ 0 ----
Is there any thing wrong with that?
I was using "rm -r".
So not used snapper for that.
It could be that I had been thinking those snapshots are basically links to the originals that are COW'd (copied on write) if the original changes (the way "normal" snapshots work today). So rm on the snapshot would not affect the original in a negative way. [...]
I still see it as an invalid bug and an unfortunate misuse of btrfs subvolumes deleting important ones :(
When considering snapshots to be non-essential backups of an original filesystem, I think (as it turned out during this issue) mounting any snapshot as original is a brain-dead solution. Despite of that the filesystem is still filling up at a rate that worries me. Maybe that's, because too many snapshots are being created. Fr example today I had an update with separate zypper update, making six snapshots altogether: # snapper --iso list # | Type | Pre # | Date | User | Cleanup | Description | Userdata ----+--------+-------+---------------------+------+---------+--------------------+-------------- [...] 76 | pre | | 2019-10-28 08:28:20 | root | number | yast online_update | 77 | pre | | 2019-10-28 08:31:01 | root | number | zypp(ruby.ruby2.5) | important=no 78 | post | 77 | 2019-10-28 08:32:00 | root | number | | important=no 79 | pre | | 2019-10-28 08:33:55 | root | number | zypp(ruby.ruby2.5) | important=yes 80 | post | 79 | 2019-10-28 08:37:07 | root | number | | important=yes 81 | post | 76 | 2019-10-28 08:37:14 | root | number | | Is any user really expected to recover between these snapshots? And the "Description" isn't really helpful (as obvious from comment 15 already). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c23
--- Comment #23 from Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c27
--- Comment #27 from Ulrich Windl
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c28
--- Comment #28 from Ulrich Windl
(In reply to Ulrich Windl from comment #23)
"zypp(ruby.ruby2.5)" (not really helpful) "zypp(zypper)" (not really helpful)
These are created by the snapper-zypp-plugin, "zypp" means it was created by the libzypp library, the name in the parenthesis is the process name. That "ruby.ruby2.5" likely belongs to YaST.
You completely missed the most obvious: Those texts are constants, while the snapshots are not: Shouldn't the description for a snapshot be related to the snapshot being made (i.e.. the data it contains, instead of writing the version of the software that created the snapshot)? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c34
--- Comment #34 from Ulrich Windl
I got lost in this discussion. It moved from disk space to too many snapshots to naming the snapshots.
Easy: The filesystem is filling up more and more while snapshots that are created for no obvious reason accumulate. Now you have it all in one sentence. User's expectation are that old snapshots are deleted automatically before the filesystem is so full that you cannot even delete anything. Another expectation is that there exists documentation for such questionably features; I mean: Documentation that is easy to find. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782
http://bugzilla.opensuse.org/show_bug.cgi?id=1144782#c38
--- Comment #38 from Ulrich Windl
Cleanup Algorithms ... number ... timeline ... empty-pre-post ...
Soit seems an algorithm "freespace" is missing; that algorithm should make sure that at least X units of free space is available; otherwise delete oldest snapshots. Complain loudly if it's not possible to create enough free space. (In reply to Arvin Schnell from comment #37) I'm usually using yast inline_update to install updates, not zypper directly. As the machine also has the Nvidia driver that is renowned for (still!) doing excessive initrd rebuilds, this may also cause many snapshots to be created (I'm not sure, but then we are at the item "useful comments for a snapshot"). To me it seems the currect snapshot "description" is based on WHO called snapper instead of WHY it called snapper. I could image better descriptions like "kernel X.Y.Z", ".../important-module.ko", "/bin/essential-command" (being installed) The comments I see at the moment are these: "current", "zypp(ruby.ruby2.5)", "" (empty/nothing), "yast online_update", and "zypp(zypper)": # snapper --iso list | awk -F'|' 'NF == 8 && $7 !~ "Description" { print $7 }' | sort | uniq -c 11 1 current 3 yast online_update 7 zypp(ruby.ruby2.5) 1 zypp(zypper) # snapper --iso list | awk -F'|' 'NF == 8 && $7 !~ "Description" { print $7, "(" $2 ")" }' | sort | uniq -c 1 current ( single ) 11 ( post ) 3 yast online_update ( pre ) 7 zypp(ruby.ruby2.5) ( pre ) 1 zypp(zypper) ( pre ) -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com