http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c1 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com Flags| |needinfo?(fbui@suse.com) --- Comment #1 from Fabian Vogt --- Created attachment 823824 --> http://bugzilla.suse.com/attachment.cgi?id=823824&action=edit journalctl What's in /var/log/journal and /run/log/journal? I think I can reproduce this here. The issue appears to be that systemd-journal-flush.service runs, but does somehow not cause a flush. Snippet from the journal during bootup: Oct 31 07:32:11 localhost systemd[1]: Starting Flush Journal to Persistent Storage... Oct 31 07:32:11 localhost systemd-journald[521]: Runtime Journal (/run/log/journal/88c7cea6795b4428909767e3e81c15bd) is 5.9M, max 47.2M, 41.3M free. Oct 31 07:32:11 localhost kernel: audit: type=1400 audit(1572507131.859:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="klogd" pid=584 comm="apparmor_parser" Oct 31 07:32:11 localhost systemd[1]: Started Flush Journal to Persistent Storage. That apparently didn't work, it still writing to /run/log/journal. Snipped from the journal after restarting systemd-journal-flush.service manually: Oct 31 07:44:53 localhost systemd[1]: Starting Flush Journal to Persistent Storage... Oct 31 07:44:53 localhost systemd-journald[521]: Time spent on flushing to /var is 108.202ms for 876 entries. Oct 31 07:44:53 localhost systemd-journald[521]: System Journal (/var/log/journal/88c7cea6795b4428909767e3e81c15bd) is 24.0M, max 1.4G, 1.4G free. Oct 31 07:44:53 localhost tallow[960]: Journal was rotated, resetting Oct 31 07:44:53 localhost systemd[1]: Started Flush Journal to Persistent Storage. Here it switched to /var successfully. So the question is why it does not happen (reliably?) when running during boot? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c3 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fbui@suse.com) | --- Comment #3 from Franck Bui --- (In reply to Fabian Vogt from comment #1)

Here it switched to /var successfully. So the question is why it does not happen (reliably?) when running during boot?

Hmm... the only reason I can see is that /var is still a RO partition at the time the journal is flushed and therefore the system journal couldn't be created in /var. But you would get an explicit error on the next reboot since this time the journal file was already created when you flushed the journal manually and journald would fail at opening it RW. Can you see such error ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c5 --- Comment #5 from Franck Bui --- (In reply to Fabian Vogt from comment #1)

I think I can reproduce this here. The issue appears to be that systemd-journal-flush.service runs, but does somehow not cause a flush.

Any chance I can reproduce it locally, maybe with qemu ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c7 --- Comment #7 from Thorsten Kukuk --- (In reply to Fabian Vogt from comment #6)

(In reply to Franck Bui from comment #3)

...

Hmm... the only reason I can see is that /var is still a RO partition at the time the journal is flushed and therefore the system journal couldn't be created in /var.

/var is mounted read-write in the initrd already. So unless something re-mounts it multiple times during boot, I don't think that's the case.

systemd remounts that at first after leaving the initrd, at least it did so when we debugged similar problems with apparmor. There we had the same problems, even if the filesystem was mounted read-write in the initrd, systemd remounted it to read-only and later to read-write, and loading the apparmor profiles failed due to read-only /var. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c9 --- Comment #9 from Fabian Vogt --- Created attachment 824579 --> http://bugzilla.suse.com/attachment.cgi?id=824579&action=edit strace of journald during systemd-journal-flush.service failing I added an strace call to systemd-journal-flush.service and rebooted and luckily it still broke. The issue seems to be that indeed /var is still read-only: writev(6, [{iov_base="<46>", iov_len=4}, {iov_base="systemd-journald", iov_len=16}, {iov_base="[815]: ", iov_len=7}, {iov_base="Received client request to flush"..., iov_len=49}, {iov_base="\n", iov_len=1}], 5) = 77 mkdirat(AT_FDCWD, "/var/log/journal/ec14ed36308c4af2b46995c50601f402", 0755) = -1 EROFS (Read-only file system) openat(AT_FDCWD, "/var/log/journal/ec14ed36308c4af2b46995c50601f402/system.journal", O_RDWR|O_CREAT|O_NONBLOCK|O_CLOEXEC, 0640) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/run/log/journal/ec14ed36308c4af2b46995c50601f402", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 24 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c11 --- Comment #11 from Franck Bui --- Fabian, can you please provide the debug logs again but with also "printk.devkmsg=on" option so the kernel stop dropping messages ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c13 --- Comment #13 from Franck Bui --- Thanks. As the logs are broken due to the unreliable clock on your system (the debug logs with printk.devkmsg=on showed that the system switched to rootfs twice !), I'm not 100% sure but I can't see any hints on systemd unmounting or remounting /var RO after leaving initrd. According to our private chat /var is mounted RW just before switching to rootfs. So either something in userspace unmount it or remount it RO. Can you replace /usr/bin/mount and /usr/bin/umount with wrappers that log any attempts to mount/remount/umount /var before doing the actual work in order to make sure that the issue leaves in userspace ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c15 --- Comment #15 from Fabian Vogt --- Progress! The culprit seems to be "mount -o remount /". Looks harmless, but actually isn't - it mounts everything on the same btrfs partition as read-only, for some reason. I suspect it's because "rw" isn't explicitly set as option, but that doesn't explain why: a) It's initially mounted rw b) It's mounted rw again later -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dsterba@suse.com, | |rgoldwyn@suse.com, | |tiwai@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c18 Miroslav Beneš changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mbenes@suse.com --- Comment #18 from Miroslav Beneš --- David, Goldwyn, do you consider the behaviour Fabian described in comment 16 as step 3 a bug in the kernel? If not, we should reassign? Fabian, I assume the issue still exists with the newer kernel. Is that correct? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c20 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fvogt@suse.com --- Comment #20 from Fabian Vogt --- For some reason I'm not in CC here, so I didn't get a notification mail... (In reply to Miroslav Beneš from comment #18)

David, Goldwyn, do you consider the behaviour Fabian described in comment 16 as step 3 a bug in the kernel? If not, we should reassign?

Fabian, I assume the issue still exists with the newer kernel. Is that correct?

I don't know, but I would assume that it's the case too. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1156421 http://bugzilla.suse.com/show_bug.cgi?id=1156421#c22 --- Comment #22 from Goldwyn Rodrigues --- Created attachment 835970 --> http://bugzilla.suse.com/attachment.cgi?id=835970&action=edit Proposed patch: Fix RO remount for subvols This is a proposed fix. However, there are problems by removing sb->s_flags setting code from fs/super.c. Filesystems depend of fs/super.c to set sb_flags for them, so it is not expected to be accepted upstream. I need to dig deeper with respect to fs_context -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 https://bugzilla.suse.com/show_bug.cgi?id=1156421#c25 --- Comment #25 from Goldwyn Rodrigues --- We had a discussion in the btrfs mailing list and it was concluded that these changes would deviate from the "default" case [1]. Currently sys admins are setting any subvolume read-only to set the root filesystem read-only. While this is not ideal, this behavior has already been exploited. [1] https://lore.kernel.org/linux-btrfs/20220211164422.GA12643@twin.jikos.cz/T/#... -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 https://bugzilla.suse.com/show_bug.cgi?id=1156421#c27 --- Comment #27 from Goldwyn Rodrigues --- (In reply to Fabian Vogt from comment #26)

(In reply to Goldwyn Rodrigues from comment #25)

...

We had a discussion in the btrfs mailing list and it was concluded that these changes would deviate from the "default" case [1]. Currently sys admins are setting any subvolume read-only to set the root filesystem read-only. While this is not ideal, this behavior has already been exploited.

[1] https://lore.kernel.org/linux-btrfs/20220211164422.GA12643@twin.jikos.cz/T/#...

Ah, good ol' spacebar heating (https://xkcd.com/1172/).

Maybe the example doesn't really show the severity of the issue, as it focuses on the "root" mountpoint. This has more independent mounts of subvolumes:

Yes, this is a problem with any subvolume. *Any* subvolume setting to remount read-only renders all subvolumes currently mounted of the entire filesystem read-only. This is what was pointed out by Graham Cobb in the discussion.

dd if=/dev/zero of=btrfsfs seek=240 count=0 bs=1M mkfs.btrfs btrfsfs mkdir mnt mount btrfsfs mnt btrfs subvol create mnt/sv0 btrfs subvol create mnt/sv1 umount mnt

mkdir sv{0,1}mnt mount -o subvol=/sv0 btrfsfs sv0mnt mount -o subvol=/sv1 btrfsfs sv1mnt findmnt sv0mnt # RW findmnt sv1mnt # RW mount -o remount,ro sv0mnt findmnt sv0mnt # RO findmnt sv1mnt # RO! mount -o remount,rw sv1mnt findmnt sv0mnt # RW! findmnt sv1mnt # RW umount sv*mnt

Do we have any filesystems we can compare the behaviour with?

Unfortunately, btrfs is pretty unique in this front. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 https://bugzilla.suse.com/show_bug.cgi?id=1156421#c34 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |felix.niederwanger@suse.com --- Comment #34 from Fabian Vogt --- *** Bug 1202276 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 Matthias Eckermann changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|WONTFIX |--- -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 https://bugzilla.suse.com/show_bug.cgi?id=1156421#c38 --- Comment #38 from Jeff Mahoney --- Does mount --bind -oro,remount <mnt> work for you? -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1156421 https://bugzilla.suse.com/show_bug.cgi?id=1156421#c43 Jeff Mahoney changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jeffm@suse.com) | --- Comment #43 from Jeff Mahoney --- (In reply to Thorsten Kukuk from comment #41)

(In reply to Jeff Mahoney from comment #38)

...

Does mount --bind -oro,remount <mnt> work for you?

If I call this from the commandline, yes, works. If I add "bind" to /etc/fstab (ro,bind), no, this doesn't seem to work.

But we need something for /etc/fstab for the boot process.

Beside that, what would be the side effects of using this option?

Using ro or rw without bind in the fstab has the same effect. If the initial mount is read-only, it will mark the entire file system read-only. If a another subvolume mount wants a read-write mount, it will leave the first mountpoint read-only, remount the file system internally read-write, and mark the new mount point read-write. Writes to the first mount will continue to be refused and will be allowed on the second mount. For context, the read-only flag can be set in: - Filesystem-wide superblock flags - Mountpoint flags - Subvolume flags If the superblock flag or the subvolume flag is set, the subvolume will be read-only no matter where it's mounted and regardless whether the mountpoint is read-write. If the mountpoint flag is set, the subvolume will be read-only only at that mountpoint and could be writable at another location if the subvolume was mounted more than once. If you use mount -oremount,ro on a subvolume, that overrides that internal remount to rw and you end up with the situation where the entire fs is read-only. If you use mount -oremount,ro --bind on a subvolume _mountpoint_ it only applies to the mountpoint. If you do that on a subvolume that isn't explicitly mounted, it'll complain about it not being a mountpoint and fail with an error. All of this happens with flags that the VFS layer understands, using the superblock and mountpoints. The problem is that btrfs subvolumes don't have to be represented by a mountpoint. We do that explicitly to give the appearance of a coherent file system. As a result, there's no internal linkage between mount points and subvolumes. The mount point flags and the subvolume flags aren't connected. This has some annoying UX effects. 1) You can set the mountpoint readonly status and it'll be enforced at that mountpoint, but not subsequent mounts of the same subvolume unless those are also mounted with the same read-only status. This would either be mount --bind -oremount,r[ow] or by just specifying the ro/rw flags in /etc/fstab. 2) You can set the btrfs subvolume readonly status and it'll be enforced everywhere, but the mountpoint flags are all that's shown to the user via 'mount.' To get the subvolume flags you'll need to use btrfs tools. This would be btrfs property set path read-only [true|false] The subvolume flags are persistent, while the mountpoint flags are not. Unfortunately, mount points essentially don't exist to the underlying file system. The mount routine for a file system returns a dentry and _then_ the mountpoint is created. You will see a mount point used in the btrfs mount code internally but it's only temporary and is released before the mount completes. So there's no way to have the mount point automatically reflect the flags of the subvolume. A long-term idea of mine has been to leverage the automount behavior in the VFS to handle subvolumes. The automount callback _does_ have access to the vfsmount and so it could set those flags before returning it to the vfs. I haven't had the time to do this and we've never made it an official feature request. So, I'm not sure this helps but should at least explain why it's so confusing. -- You are receiving this mail because: You are on the CC list for the bug.