[Bug 1188110] New: pkcon system-upgrade leads to unbootable system with kernel upgrade
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110 Bug ID: 1188110 Summary: pkcon system-upgrade leads to unbootable system with kernel upgrade Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Major Priority: P5 - None Component: MicroOS Assignee: kubic-bugs@opensuse.org Reporter: alois1@gmx-topmail.de QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- I recently installed MicroOS Desktop KDE. I attempted to upgrade this system from 20210704 to 20210706 using pkcon, which included a kenel upgrade to 5.13. This upgrade lead to an unbootable system with the following symptoms: * In the GRUB menu, it is attempted to load the wrong kernel version (5.12.13). * When manually editing the kernel and initrd version to 5.13.0, the initrd enters emergency mode after some delay. * Further investigation shows that initrd is incomplete and lacks at least crypttab, so the partitions on the encrypted disks cannot be mounted and caused the timeout. The issue is reproducible, after rolling back to the last snapshot on 20210704 and attempting the upgrade again the symptoms are the same. I have for now upgraded the system using zypper dup inside tukit. This lead to a working upgraded system. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c1
--- Comment #1 from Alois Wohlschlager
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c2
--- Comment #2 from Neal Gompa
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c3
Giuseppe Fierro
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c4
--- Comment #4 from Dario Faggioli
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c5
--- Comment #5 from Neal Gompa
Yes, I have also experienced this. I'll add that doing:
sudo tukit --continue execute update-bootloader
after `pkcon update` / `pkcon upgrade-system` and before rebooting acts as a workaround for this problem, at least for me and for a couple of other users with which I was discussing this issue.
Of course, that's not acceptable as a solution and pkcon/PackageKit should handle kernel updates automatically. But maybe it's something useful to know, while we look for a proper fix
What I'm confused about is how we're supposed to know to run `update-bootloader`? What is supposed to trigger this? Currently, I expect something like this to be run as some kind of post-transaction trigger by the packages that would require this to be run... -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c7
--- Comment #7 from Neal Gompa
(In reply to Neal Gompa from comment #5)
Currently, I expect something like this to be run as some kind of post-transaction trigger by the packages that would require this to be run...
... is what should happen. And, in fact, it happens automatically when we upgrade with zypper inside `tukit` (or with the old `transactional-update`).
The point is why it does not happen when installing/updating the package with `pkcon`
Do we know what package actually does run update-bootloader itself so I can narrow down to a test case? -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c8
Richard Brown
(In reply to Dario Faggioli from comment #6)
(In reply to Neal Gompa from comment #5)
Currently, I expect something like this to be run as some kind of post-transaction trigger by the packages that would require this to be run...
... is what should happen. And, in fact, it happens automatically when we upgrade with zypper inside `tukit` (or with the old `transactional-update`).
The point is why it does not happen when installing/updating the package with `pkcon`
Do we know what package actually does run update-bootloader itself so I can narrow down to a test case?
Isn't that done by the kernel post script which is codified in suse-module-tools? https://build.opensuse.org/package/show/openSUSE:Factory/suse-module-tools https://github.com/openSUSE/suse-module-tools/tree/master/kernel-scriptlets -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c9
--- Comment #9 from Neal Gompa
(In reply to Neal Gompa from comment #7)
Do we know what package actually does run update-bootloader itself so I can narrow down to a test case?
Isn't that done by the kernel post script which is codified in suse-module-tools?
https://build.opensuse.org/package/show/openSUSE:Factory/suse-module-tools
https://github.com/openSUSE/suse-module-tools/tree/master/kernel-scriptlets
It doesn't appear to be the case. It looks like only dracut is called from the scripts in there, not update-bootloader. And I didn't find a direct reference to a script that calls update-bootloader in the kernel spec templates in kernel-source either. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c10
--- Comment #10 from Dario Faggioli
(In reply to Richard Brown from comment #8)
Isn't that done by the kernel post script which is codified in suse-module-tools?
https://build.opensuse.org/package/show/openSUSE:Factory/suse-module-tools
https://github.com/openSUSE/suse-module-tools/tree/master/kernel-scriptlets
It doesn't appear to be the case. It looks like only dracut is called from the scripts in there, not update-bootloader.
And I didn't find a direct reference to a script that calls update-bootloader in the kernel spec templates in kernel-source either.
Same. Could it be that it's not update-bootloader itself, but either grub2-mkconfig or some other wrapper? -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110 http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c12 Neal Gompachanged: What |Removed |Added ---------------------------------------------------------------------------- CC| |iforster@suse.com Flags| |needinfo?(iforster@suse.com | |) --- Comment #12 from Neal Gompa --- Created attachment 853724 --> http://bugzilla.opensuse.org/attachment.cgi?id=853724&action=edit System upgrade from MicroOS 20211008 to 20211111 using Micro DNF I installed Micro DNF to attempt to get a better idea of what's happening, and now I know a bit about what's wrong. I attempted an upgrade using Micro DNF from the October 8 snapshot to the November 11 snapshot, which includes a kernel upgrade, and I got some interesting output: ``` grub2-probe: error: cannot find a device for /boot (is /dev mounted?). update-bootloader: 2021-11-13 14:28:57 <3> update-bootloader-0815 run_command.294: '/usr/lib/bootloader/grub2/install' failed with exit code 1, output: <<<<<<<<<<<<<<<< target = i386-pc + /usr/sbin/grub2-install --target=i386-pc --force --skip-fs-probe /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0 Installing for i386-pc platform. /usr/sbin/grub2-install: error: cannot find a device for / (is /dev mounted?). >>>>>>>>>>>>>>>> update-bootloader: 2021-11-13 14:28:57 <3> update-bootloader-0815 run_command.294: '/usr/lib/bootloader/grub2/config' failed with exit code 1, output: <<<<<<<<<<<<<<<< + /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg /usr/sbin/grub2-probe: error: cannot find a device for / (is /dev mounted?). >>>>>>>>>>>>>>>> ``` This seems like something is wrong with how libtukit sets up the new root snapshot for the transaction. I've attached the full output log as well, so you can see for yourself. My reproducer: 1. Download and install the 20211008 snapshot 2. Install microdnf with pkcon: "pkcon install microdnf" 3. Reboot 4. Update with microdnf: "sudo microdnf update" -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c13
Ignaz Forster
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c14
--- Comment #14 from Neal Gompa
I see the problem now: Transaction->getRoot() returns the path to the directory of the root file system. This path is then set as the root path for DNF directly.
The problem is that /usr/sbin/grub2-install requires to have a dedicated mount point and not just a directory, so tukit is also creating a temporary bind mount and executes commands it receives on that bind mount instead of the root directory. However libdnf-plugin-txnupd is using the root path directly.
I guess the best solution would be to return the bind path for Transaction->getRoot() instead. If one needs the real root path one can use Snapshot->getRoot() then (the Snapshot class will become public with transactional-update 4.0 anyway).
That would make sense, as I'm expecting Transaction->getRoot() to be the prepared transaction area for package management to work in. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c15
--- Comment #15 from OBSbugzilla Bot
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110
http://bugzilla.opensuse.org/show_bug.cgi?id=1188110#c16
Ignaz Forster
participants (1)
-
bugzilla_noreply@suse.com