[Bug 1121276] New: x-initrd.mount in fstab does not work with overlayfs and x-systemd.requires-mounts-for
http://bugzilla.suse.com/show_bug.cgi?id=1121276 Bug ID: 1121276 Summary: x-initrd.mount in fstab does not work with overlayfs and x-systemd.requires-mounts-for Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: fbui@suse.com Reporter: fvogt@suse.com QA Contact: qa-bugs@suse.de CC: iforster@suse.com, kukuk@suse.com Found By: --- Blocker: --- Having entries with "x-initrd.mount" in /etc/fstab means that those will be mounted before switching root, i.e. in the initrd's context. systemd-fstab-generator correctly prefixes the destination path with /sysroot in that case, but forgets to do it for "x-systemd.requires-mounts-for" and overlayfs specific parameters such as "workdir", "upperdir" and "lowerdir". Example: overlay /etc overlay defaults,lowerdir=/etc,upperdir=/var/lib/overlay/etc,workdir=/var/lib/overlay/work-etc,x-systemd.requires-mounts-for=/var,x-initrd.mount 0 0 Does not work, but overlay /etc overlay defaults,lowerdir=/sysroot/etc,upperdir=/sysroot/var/lib/overlay/etc,workdir=/sysroot/var/lib/overlay/work-etc,x-systemd.requires-mounts-for=/sysroot/var,x-initrd.mount 0 does. There is no workaround available, as adding the /sysroot prefix would break mounting outside of the initrd and also once this bug is fixed. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 Fabian Vogt <fvogt@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fbui@suse.com |systemd-maintainers@suse.de -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c1 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS CC| |fbui@suse.com --- Comment #1 from Franck Bui <fbui@suse.com> --- (In reply to Fabian Vogt from comment #0)
Having entries with "x-initrd.mount" in /etc/fstab means that those will be mounted before switching root, i.e. in the initrd's context.
systemd-fstab-generator correctly prefixes the destination path with /sysroot in that case, but forgets to do it for "x-systemd.requires-mounts-for" and overlayfs specific parameters such as "workdir", "upperdir" and "lowerdir".
Well I'm really not sure that systemd is supposed to parse and understand filesystem specific options... BTW what's the purpose of mounting /sysroot/etc from inside initrd ? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c2 --- Comment #2 from Thorsten Kukuk <kukuk@suse.com> --- (In reply to Franck Bui from comment #1)
BTW what's the purpose of mounting /sysroot/etc from inside initrd ?
Mainly workaround for systemd design bug: systemd starts at first systemd-journald systemd-journald requires /etc/machine-id /etc/machine-id is created by systemd during FirstBoot if not installed with YaST. root filesystem is read-only at this point in time, as mounting the root filesystem is done by systemd only after systemd-journald is running. So systemd cannot create /etc/machine-id and fails to boot. Solution: mount /etc in initrd so that systemd can create /etc/machine-id during first boot. Other problems: there are services (like ignition), which need to write into /sysroot/etc during boot phase, too. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c3 --- Comment #3 from Franck Bui <fbui@suse.com> --- (In reply to Thorsten Kukuk from comment #2)
Mainly workaround for systemd design bug: systemd starts at first systemd-journald systemd-journald requires /etc/machine-id
/etc/machine-id is created by systemd during FirstBoot if not installed with YaST.
That's not correct: /etc/machine-id is created as an empty file during systemd package installation so systemd is able to init machine-id even on a RO filesystem and in this case machine-id is ephemeral.
Solution: mount /etc in initrd so that systemd can create /etc/machine-id during first boot.
That's indeed an workaround/hack as the main purpose of initramfs is to mount the rootfs and the rest is supposed to be done by the final system. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c5 --- Comment #5 from Thorsten Kukuk <kukuk@suse.com> --- (In reply to Franck Bui from comment #3)
(In reply to Thorsten Kukuk from comment #2)
Mainly workaround for systemd design bug: systemd starts at first systemd-journald systemd-journald requires /etc/machine-id
/etc/machine-id is created by systemd during FirstBoot if not installed with YaST.
That's not correct: /etc/machine-id is created as an empty file during systemd package installation so systemd is able to init machine-id even on a RO filesystem and in this case machine-id is ephemeral.
An empty file is useless, it only has a meaning for the firstboot behavior, systemd still needs to write content into this file. And that's not possible with a read-only root filesystem. So my comment is correct.
Solution: mount /etc in initrd so that systemd can create /etc/machine-id during first boot.
That's indeed an workaround/hack as the main purpose of initramfs is to mount the rootfs and the rest is supposed to be done by the final system.
No, this was the outcome of several discussions with Werner, Thomas and some RedHat/Fedora people at the systemd conference 2,5 years ago. (In reply to Franck Bui from comment #4)
(In reply to Thorsten Kukuk from comment #2)
Mainly workaround for systemd design bug:
May I suggest to try addressing design shortcomings first before going for workarounds as they're doomed to fail soon or later anyway.
Also design issues should be discussed with upstream for obvious reasons.
That's what we did 2.5 year ago in person at the systemd conference and this implementation is the result of the discussion. Between, the official systemd point of view for this is, that the root filesystem has to be mounted read-write by the kernel from the beginning and that the default kernel behavior ("ro") will not work. Exactly because of the /etc/machine-id problem, that systemd else cannot create it or write content into it. That's why the Fedora people changed now the default kernel commandline option from "ro" to "rw" for booting. That's consistent with our discussion and mounting it writeable in the initramfs. The Fedora solution is no option for a read-only root filesystem. CoreOS did switch the root filesystem from ro to rw in initramfs for that reason. Don't remember anymore the systemd issue number, but the topic was about that the firstboot option for services did not work. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c6 --- Comment #6 from Franck Bui <fbui@suse.com> --- (In reply to Thorsten Kukuk from comment #5)
An empty file is useless, it only has a meaning for the firstboot behavior, systemd still needs to write content into this file. And that's not possible with a read-only root filesystem. So my comment is correct.
Sorry but at that point I still don't know what precisely your use case is. It sounds like you want to use overlayfs inside initramfs for making /etc writable so systemd can setup a machine-id on a RO system... If so that's not correct as systemd is supposed to deal with RO systems by bind mounting a temporary file (containing a machine-id) over (empty) /etc/machine-id. If that doesn't work then it's a bug that should be fixed. Of course no such issue happens inside initrd since the ramdisk is RW. So to sump: - in initramfs, / is RW so machine-id will be created if it's missing - in rootfs and after the system installation, /etc/machine-id is empty so systemd can either populate it on RW systems or bind mounting a temp file over it on RO systems. Currently I don't see why your use case can't work with the current scheme therefore can you please describe it accurately. PS: The logic for initializing /etc/machine-id is fully described in man machine-id(5). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c7 --- Comment #7 from Fabian Vogt <fvogt@suse.com> ---
Currently I don't see why your use case can't work with the current scheme therefore can you please describe it accurately.
I just set up my running kubic system to boot without fstab.sys and x-initrd.mount and an empty /etc/machine-id - it boots fine, but /etc/machine-id is not written back into /etc when the overlay is mounted. This is because /etc gets writable by mounting a filesystem on top of it - so the bind-mounted /etc/machine-id is not reachable anymore. So ConditionPathIsMountPoint=/etc/machine-id fails. One other reason I can see for mounting /etc before switch-root is to have edited /etc/fstab entries available, so that / can be mounted with the right flags and other mount points are also handled properly. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c8 --- Comment #8 from Franck Bui <fbui@suse.com> --- (In reply to Fabian Vogt from comment #7)
Currently I don't see why your use case can't work with the current scheme therefore can you please describe it accurately.
I just set up my running kubic system to boot without fstab.sys and x-initrd.mount and an empty /etc/machine-id - it boots fine, but /etc/machine-id is not written back into /etc when the overlay is mounted.
Well is there still a point to have /etc as a partition separate from the root one ? (and yes your use case is still unknown to me...) And I really have doubt that /etc splitted out from / is supported at all...
One other reason I can see for mounting /etc before switch-root is to have edited /etc/fstab entries available, so that / can be mounted with the right flags and other mount points are also handled properly.
Erf and what about mounting /etc with the correct flags in initrd ? Honestly I don't see the point. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c9 --- Comment #9 from Thorsten Kukuk <kukuk@suse.com> --- (In reply to Franck Bui from comment #8)
Well is there still a point to have /etc as a partition separate from the root one ? (and yes your use case is still unknown to me...)
No, of course you don't want that, as you cannot create snapshots across partition boundaries. And the usecase is simple: SLES 15 SP1 transactional server SUSE CaaS Platform openSUSE Leap 15.0 transactional server openSUSE Tumbleweed transactional server openSUSE Kubic openSUSE MicroOS list will be extended in the future, more products are already in planning. You have a read-only root filesystem, /etc/ needs to be writeable. Inside of the initrd (e.g. for ignition) and in the running system.
One other reason I can see for mounting /etc before switch-root is to have edited /etc/fstab entries available, so that / can be mounted with the right flags and other mount points are also handled properly.
Erf and what about mounting /etc with the correct flags in initrd ?
That's what we want to archive: mount /etc with the correct flags initrd. That's what this bug is about. But /etc/fstab needs the correct flags, too, else you get errors when switching from initrd to the real system, and systemd is not able to unmount the filesystem during shutdown (as the paths don't match), what our customers don't like. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c10 --- Comment #10 from Franck Bui <fbui@suse.com> --- (In reply to Thorsten Kukuk from comment #9)
(In reply to Franck Bui from comment #8)
Well is there still a point to have /etc as a partition separate from the root one ? (and yes your use case is still unknown to me...)
No, of course you don't want that, as you cannot create snapshots across partition boundaries. And the usecase is simple:
The list of the use case is maybe simple but explaining why mounting / and /etc differently seems harder apparently. The only explanation you gave me was something about machine-id that couldn't be written by systemd which made journald failing and this appeared to be wrong.
You have a read-only root filesystem, /etc/ needs to be writeable. Inside of the initrd (e.g. for ignition) and in the running system.
Again *why* ? Why / should be left RO and /etc should be RW ? Why can't both be RW like it's done for live-cd systems for instance ? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c11 --- Comment #11 from Fabian Vogt <fvogt@suse.com> --- (In reply to Franck Bui from comment #10)
(In reply to Thorsten Kukuk from comment #9)
(In reply to Franck Bui from comment #8)
Well is there still a point to have /etc as a partition separate from the root one ? (and yes your use case is still unknown to me...)
No, of course you don't want that, as you cannot create snapshots across partition boundaries. And the usecase is simple:
The list of the use case is maybe simple but explaining why mounting / and /etc differently seems harder apparently.
The only explanation you gave me was something about machine-id that couldn't be written by systemd which made journald failing and this appeared to be wrong.
Which I demonstrated doesn't work for this at least.
You have a read-only root filesystem, /etc/ needs to be writeable. Inside of the initrd (e.g. for ignition) and in the running system.
Again *why* ?
Why / should be left RO and /etc should be RW ?
Why can't both be RW like it's done for live-cd systems for instance ?
Because that violates the design of transactional updates in MicroOS. That / is RO and /etc is RW through an overlay mount is a given and will/can not change. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1121276 http://bugzilla.suse.com/show_bug.cgi?id=1121276#c12 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |UPSTREAM --- Comment #12 from Franck Bui <fbui@suse.com> --- (In reply to Fabian Vogt from comment #11)
Which I demonstrated doesn't work for this at least.
Well... I think mounting / and /etc differently is not supported currently and it seems that you're requesting a design enhancement. Therefore, please open an issue against upstream with a (better) description of your problem so we can find a proper solution that will be acceptable by both side. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com