[opensuse-factory] Unable to boot after upgrading to 20200824
Hi all! I have a tumbleweed transactional server that is unable to boot after automatically updating to snapshot 20200824. The error seems to occur during volume mounting. Related logs: etc.mount: Directory /etc to mount over is not empty, mounting anyway overlayfs: failed to resolve '/sysroot/var/lib/overlay/395/etc': -2 ignored apparmor.service errors etc.mount: Mount process exited, code=exited, status=32/n/a etc.mount: Failed with result 'exit-code'. Failed to mount /etc. bunch of dependency errors and related mounting errors This warning occurs towards the start of the boot but I'm not sure that it is related: "sysroot-var.mount: Directory /sysroot/var to mount over is not empty, mounting anyway." Regards -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 30.08.20 um 18:45 Uhr schrieb Viktor Sævarsson:
Hi all!
I have a tumbleweed transactional server that is unable to boot after automatically updating to snapshot 20200824. The error seems to occur during volume mounting.
Related logs:
etc.mount: Directory /etc to mount over is not empty, mounting anyway overlayfs: failed to resolve '/sysroot/var/lib/overlay/395/etc': -2 ignored apparmor.service errors etc.mount: Mount process exited, code=exited, status=32/n/a etc.mount: Failed with result 'exit-code'. Failed to mount /etc. bunch of dependency errors and related mounting errors
Oh dear, that's not good: I assume /var/lib/overlay is empty? If so then I've seen that once before, but wasn't able to find out what caused it. A recent health-checker update (in case you are using it) should prevent such a broken system from booting and switch to emergency mode instead to prevent any other jobs doing any more harm like in this case. What you can do now: If you haven't changed anything in /etc lately, then you can try to recreate the directories listed in /etc/fstab in the hopes that there haven't been any new configuration files from package updates. I'll add more consistency checks into transactional-update to prevent such a case in the future, and also change the overlay mechanism to make this more robust, i.e. always sync the /etc status of the currently running system to make a rollback in case of a loss of the overlays easier. Ignaz
This warning occurs towards the start of the boot but I'm not sure that it is related: "sysroot-var.mount: Directory /sysroot/var to mount over is not empty, mounting anyway."
Regards
-- Ignaz Forster <iforster@suse.com> Research Engineer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Aug 31, 2020 at 9:12 AM Ignaz Forster <iforster@suse.de> wrote:
Oh dear, that's not good: I assume /var/lib/overlay is empty?
It is actually full of directories. A lot of numbered ones and then etc and work-etc. Thanks for the response. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 01.09.20 um 19:40 Uhr schrieb Viktor Sævarsson:
On Mon, Aug 31, 2020 at 9:12 AM Ignaz Forster <iforster@suse.de> wrote:
Oh dear, that's not good: I assume /var/lib/overlay is empty?
It is actually full of directories. A lot of numbered ones and then etc and work-etc.
OK, that's good. Is /var/lib/overlay/395/etc also in there? If so then we may have a (very rare) race condition in the initrd. If it isn't, then you can just do a rollback to the previous snapshot and update the system again. (But then I'd be curious to know how the directory got deleted...) Cheers, Ignaz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Sep 2, 2020 at 8:44 AM Ignaz Forster <iforster@suse.de> wrote:
OK, that's good. Is /var/lib/overlay/395/etc also in there? If so then we may have a (very rare) race condition in the initrd. If it isn't, then you can just do a rollback to the previous snapshot and update the system again. (But then I'd be curious to know how the directory got deleted...)
/var/lib/overlay/395/etc is there and does have some files and directories. I would try to rollback but the only available snapshots are from february this year. I think that's an issue that happened when I updated from leap 15.1 to tumbleweed. Might that be related? Regards, Viktor -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
So after quite a bit of digging I found that there were newer snapshots they were just not showing up in the grub menu. So in the emergency mode I edited the /.snapshots/grub-snapshot.cfg and manually added the missing boot options. Then I booted into a previous snapshot and from there I could successfully rollback. No clue what caused the original issue though. But the system is working now and has been successfully updated to the latest version. Cheers On Wed, Sep 2, 2020 at 6:49 PM Viktor Sævarsson <viktor.saevars@gmail.com> wrote:
On Wed, Sep 2, 2020 at 8:44 AM Ignaz Forster <iforster@suse.de> wrote:
OK, that's good. Is /var/lib/overlay/395/etc also in there? If so then we may have a (very rare) race condition in the initrd. If it isn't, then you can just do a rollback to the previous snapshot and update the system again. (But then I'd be curious to know how the directory got deleted...)
/var/lib/overlay/395/etc is there and does have some files and directories.
I would try to rollback but the only available snapshots are from february this year. I think that's an issue that happened when I updated from leap 15.1 to tumbleweed. Might that be related?
Regards, Viktor
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Thanks a lot for the solution! Am 03.09.20 um 03:40 Uhr schrieb Viktor Sævarsson:
So after quite a bit of digging I found that there were newer snapshots they were just not showing up in the grub menu. This makes me a bit suspicious, as that happens when the root file system is not marked as read-only (see https://bugzilla.opensuse.org/show_bug.cgi?id=1174285). Do both btrfs property get / ro and findmnt / indicate that the current snapshot is a read-only snapshot?
No clue what caused the original issue though. But the system is working now and has been successfully updated to the latest version.
Excellent, good to hear! Cheers, Ignaz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Sep 3, 2020 at 7:40 AM Ignaz Forster <iforster@suse.de> wrote:
So after quite a bit of digging I found that there were newer snapshots they were just not showing up in the grub menu. This makes me a bit suspicious, as that happens when the root file system is not marked as read-only (see https://bugzilla.opensuse.org/show_bug.cgi?id=1174285). Do both btrfs property get / ro and findmnt / indicate that the current snapshot is a read-only snapshot?
"btrfs property get / ro" indicates that it is not read-only but "findmnt /" indicates that it is read-only. Should I run "btrfs property set / ro true"?
No clue what caused the original issue though. But the system is working now and has been successfully updated to the latest version.
Excellent, good to hear!
Spoke too soon apparently. System auto updated again tonight and then failed to boot with the same message as originally. As a quick and dirty fix I have disabled auto updating for now. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Sep 03, Viktor Sævarsson wrote:
On Thu, Sep 3, 2020 at 7:40 AM Ignaz Forster <iforster@suse.de> wrote:
So after quite a bit of digging I found that there were newer snapshots they were just not showing up in the grub menu. This makes me a bit suspicious, as that happens when the root file system is not marked as read-only (see https://bugzilla.opensuse.org/show_bug.cgi?id=1174285). Do both btrfs property get / ro and findmnt / indicate that the current snapshot is a read-only snapshot?
"btrfs property get / ro" indicates that it is not read-only but
Ok, so the btrfs subvolume is not read-only, which means you system has to break. Did you install the transactional-server with YaST, or did you add the packages later? Or did you play with btrfs property on the subvolume?
"findmnt /" indicates that it is read-only. Should I run "btrfs property set / ro true"?
You can try it. But since you did run with a read-write btrfs subvolume, I'm not sure the system will recover from it.
No clue what caused the original issue though. But the system is working now and has been successfully updated to the latest version.
Excellent, good to hear!
Spoke too soon apparently. System auto updated again tonight and then failed to boot with the same message as originally.
That's expected if your btrfs subvolume is not ro. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On fimmtudagur, 3. september 2020 13:06:49 GMT Thorsten Kukuk wrote:
"btrfs property get / ro" indicates that it is not read-only but
Ok, so the btrfs subvolume is not read-only, which means you system has to break. Did you install the transactional-server with YaST, or did you add the packages later? Or did you play with btrfs property on the subvolume?
I chose the transactional server option during the installation. However this was originally a leap 15.0 server. But in february I updated it from 15.0 -> 15.1 -> Tumbleweed and there have been no problems until now. I've not been playing around with the btrfs properties.
"findmnt /" indicates that it is read-only. Should I run "btrfs property set / ro true"?
You can try it. But since you did run with a read-write btrfs subvolume, I'm not sure the system will recover from it.
Well I tried and got: ERROR: Could not set subvolume flags: Read-only file system Regards -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (3)
-
Ignaz Forster -
Thorsten Kukuk -
Viktor Sævarsson