Comment # 12 on bug 1175779 from Franck Bui

(In reply to Franck Bui from comment #11)
> For affected users, the problem arose during a reboot after this second
> update. Either the system failed to boot completely because / was left RO or
> /tmp wasn't mounted at all.

Now for those interested more on this.

It wasn't clear to me why some users reported that the system couldn't boot
anymore because / was left RO whereas I would have expected the system to be
able to boot even though tmp.mount was dangling (because local-fs.target only
has "Wants=" dependency type on /tmp) and /tmp wasn't mounted.

Then I realized that some users might have not followed the guide to switch to
tmpfs completely and forgot to comment /tmp entry in fstab as it was
recommended. I did gave it a try and booted with /tmp in fstab _and_ the
dangling symlink present in /etc/systemd/system. And the magic happened, on the
next reboot / was left RO and my system failed to boot.

Now that I could reproduce this problem easily, I found out that this incorrect
behavior is due to a nasty bug in systemd that I reported there:
https://github.com/systemd/systemd/issues/16909

An interesting part of the problem is that for those who forgot to comment /tmp
out in fstab then a dependency of type "Requires=local-fs.target" was added to
local-fs.target [1] (instead of "Wants=tmp.mount") [2].

And in this specific case, the dependencies of local-fs.target are enqueued
until tmp.mount is processed. Since it's a dangling symlink, PID1 fails to
enqueue the relevant job and abort the handling of local-fs.target deps hence
skipping the not yet processed deps - systemd-remount-fs.servce, which is
supposed to remount / RW, was part of them.

Finally since sysvinit.target only 'Wants=local-fs.target' (weak dep), PID1
still started the transaction with dependencies of local-fs.target partially
enqueued.

[1] I'm wondering if that's correct as we should be able to mask tmp.mount but
masking /tmp is probably forbidden when btrfs is used hence the installer
didn't use the "nofail" option.

[2] This is a bit weird because it means that if a user decides to override a
mount unit that is generated by fstab, local-fs.target will still use the
"Requires=" dep type specified in fstab.