[Bug 1177900] New: openSUSE Tumbleweed: Boot from Live ISO - only emergency mode
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 Bug ID: 1177900 Summary: openSUSE Tumbleweed: Boot from Live ISO - only emergency mode Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Critical Priority: P5 - None Component: Live Medium Assignee: dimstar@opensuse.org Reporter: adrian.feliks@gmail.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0 Build Identifier: All Tumbleweed versions do not start when booting from the ISO image by Grub2 ("iso-scan/filename=$iso_path") Stuck in Emergency Mode. I tested on my laptop and in QEMU (EFI and bios). Last time it was ok. openSUSE Leap 15.2 boots correctly. openSUSE-Tumbleweed-*-Live-x86_64-Snapshot20201014-Media.iso openSUSE-Tumbleweed-Rescue-CD-x86_64-Snapshot20201014-Media.iso Grub config file: https://github.com/Mexit/MultiOS-USB/blob/master/config/openSUSE/openSUSE-Li... Reproducible: Always Actual Results: Stuck in Emergency Mode Expected Results: Starts to the desktop I attach a screenshot and rdsosreport.txt -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c1 --- Comment #1 from Adrian Feliks <adrian.feliks@gmail.com> --- Created attachment 842793 --> http://bugzilla.opensuse.org/attachment.cgi?id=842793&action=edit Screenshot -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c2 --- Comment #2 from Adrian Feliks <adrian.feliks@gmail.com> --- Created attachment 842794 --> http://bugzilla.opensuse.org/attachment.cgi?id=842794&action=edit rdsosreport.txt -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 Adrian Feliks <adrian.feliks@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adrian.feliks@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c3 Dominique Leuenberger <dimstar@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcassany@suse.com Assignee|dimstar@opensuse.org |ms@suse.com --- Comment #3 from Dominique Leuenberger <dimstar@opensuse.org> --- Assigning to the kiwi maintainer -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 David Cassany <dcassany@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|ms@suse.com |dcassany@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c4 David Cassany <dcassany@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #4 from David Cassany <dcassany@suse.com> --- I can reproduce this issue, seams to be related to some udev misbehavior. During boot when booting the ISO from grub2 by using the `iso-scan/filename=$iso_path` kernel command for some reason the devices on `/dev/disk/by-label` are not populated. KIWI dracut live module relies on finding the root device on `/dev/disk/by-label` thus at boot it gets stuck waiting for the expected device until dracut-initqueue timout is reached. This does not happen on a regular cdrom boot from the same ISO, so for now I am debugging the procedure around iso-scan usage. Curiously, once on the emergency shell I see the systemd-udev-trigger.service was executed successfully, restarting it populates the `/dev/disk/by-label` as expected. So I am wondering if there might be some race condition between udev trigger and the loopback setup of the iso in iso-scan scripts. I am having a deeper look now. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c5 David Cassany <dcassany@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jslaby@suse.com Flags| |needinfo?(jslaby@suse.com) --- Comment #5 from David Cassany <dcassany@suse.com> --- I could not find any reason of this failure in KIWI code. In fact seams to be an issue with the kernel update. The problem is that after running udev services at boot, udev does not react on registering the ISO loop device according to the persistent storage rules, it does not populate the `/dev/disk/by-label`. I tested the exact same image [1] with different versions of the Kernel and using any pre 5.8.0 kernel solves the issue, since 5.8.0 it fails. At boot dracut initqueue falls in a never ending loop waiting for a device that has not been detected. After the timeout it just drops in to a shell. From that given shell I can verify that the losetup over the ISO was executed as expected and I also verified that calling `udevadm trigger --type=devices --action=add` populates the expected `/dev/disk/by-label/CDROM` device. This udeadm call is part of the `systemd-udev-trigger.service` which was also properly executed during the boot. I am mostly clue less about what is actually causing this behaviour change, I also could not see any relevant difference on udev package and the related services and udev rules. Given that what raises the issue is a kernel update I adding a require info to the kernel maintainers. Reproducer: 1. Download a TW live image 2. 'Install' it on a Fat32 usb stick with `live-grub-stick` utility $ sudo live-grub-stick <isofile> /dev/sdXY 3. Boot the image on KVM $ sudo -E qemu-kvm -m 2048 -hda /dev/sdX [1] https://build.opensuse.org/project/show/home:dcassany:branches:Virtualizatio... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c6 David Cassany <dcassany@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ptesarik@suse.com Flags| |needinfo?(ptesarik@suse.com | |) --- Comment #6 from David Cassany <dcassany@suse.com> --- I did few more investigations on this issue and now I am quite convinced the problem is within some behavior change in the kernel. I noticed that during the initqueue stage at boot `losetup -f <isofile>` does not trigger any uevent (nothing gets detected by `udevadm monitor`), so udev does not have any event to react to while setting the loop device. However if the uevent is manually forced by ` echo "change" >> /sys/block/loop0/uevent` or `udevadm trigger --type=devices --action=add` then the device is properly managed by udev and the `/dev/disk/*` links are created as expected. If I boot the same ISO as `qemu-kvm -m 2049 -cdrom <isofile>` I can see how `losetup -f <isofile>` triggers the expected uevent on the booted system, thus it looks like the issue is only happening at early boot stages. I could not figure out why. Given the fact that kernel 5.7 works fine and since 5.8 it doesn't I checked the kernel release notes for 5.8 and saw there are some changes applied for loop devices in this commit [1], however I don't have enough knowledge on the kernel to trace if is actually related or not. The reported issue could be easily worked around by retriggering device events after the loop device setup on `/usr/lib/dracut/modules.d/90dmsquash-live/iso-scan.sh`. Just did a quick test and it worked smooth. This hacky workaround can be applied at image description level by including the following into the `config.sh` script: ```bash #====================================== # Patch isoscan script # See OSInside/kiwi#1586 at github.com #-------------------------------------- isoscan_script="/usr/lib/dracut/modules.d/90dmsquash-live/iso-scan.sh" if [ -f "${isoscan_script}" ]; then new_line="udevadm trigger --type=devices --action=add" sed -i "/losetup -f/a ${new_line}" "${isoscan_script}" fi ``` The patch above (use udevadm to trigger the required loop events) or something similar could be applied in isoscan.sh script. I believe it should not hurt and serve as a sanity check. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c9 David Cassany <dcassany@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(dcassany@suse.com | |) | --- Comment #9 from David Cassany <dcassany@suse.com> --- (In reply to Fabian Vogt from comment #7)
This kernel commit might fix it: https://github.com/torvalds/linux/commit/ c01a21b77722db0474bbcc4eafc8c4e0d8fed6d8
That's in 5.10-rc4, so you can get an RPM from https://build.opensuse.org/package/show/Kernel:HEAD/kernel-default.
Does it work with that?
Thanks much for finding it. Yes finally managed to get my OBS test to finalize the build and just verified that it works as expected. @Jiri thanks for pushing the fix to stable. Was the this issue referenced in the backport? Just wondering I am gonna get a notification when this becomes available in TW. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c11 Harish Karumuthil <harish2704@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |harish2704@gmail.com Resolution|FIXED |--- --- Comment #11 from Harish Karumuthil <harish2704@gmail.com> --- I am facing this same issue now . I tested it on following Live images. 1. openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20221210-Media.iso 2. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221210-Media.iso 3. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221209-Media.iso ( Ie, first day, I tested with KDE live image, then next day, I tested wit both KDE and Gnome Live image on multiple hardware & Virtualbox ) Boot fails because `/dev/disk/by-UUID` and/or `/dev/disk/by-label` directories does not exists at the time iso-scan hook -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c12 Fabian Vogt <fabian@ritter-vogt.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fabian@ritter-vogt.de --- Comment #12 from Fabian Vogt <fabian@ritter-vogt.de> --- (In reply to Harish Karumuthil from comment #11)
I am facing this same issue now . I tested it on following Live images.
1. openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20221210-Media.iso 2. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221210-Media.iso 3. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221209-Media.iso
( Ie, first day, I tested with KDE live image, then next day, I tested wit both KDE and Gnome Live image on multiple hardware & Virtualbox )
Boot fails because `/dev/disk/by-UUID` and/or `/dev/disk/by-label` directories does not exists at the time iso-scan hook
How did you boot the .isos? Note that you must not use ventoy, it breaks the image. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c13 --- Comment #13 from Harish Karumuthil <harish2704@gmail.com> --- (In reply to Fabian Vogt from comment #12)
(In reply to Harish Karumuthil from comment #11)
I am facing this same issue now . I tested it on following Live images.
1. openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20221210-Media.iso 2. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221210-Media.iso 3. openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221209-Media.iso
( Ie, first day, I tested with KDE live image, then next day, I tested wit both KDE and Gnome Live image on multiple hardware & Virtualbox )
Boot fails because `/dev/disk/by-UUID` and/or `/dev/disk/by-label` directories does not exists at the time iso-scan hook
How did you boot the .isos? Note that you must not use ventoy, it breaks the image.
Thanks for letting me know about Ventoy. I was not aware about it and was not using it. I used normal Grub to boot the Live image. my grub conf is given below. ``` menuentry "Select openSUSE Tumbleweed KDE Live" { set iso_path=/openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20221210-Media.iso set cdlabel=openSUSE_Tumbleweed_KDE_Live } menuentry "Select openSUSE Tumbleweed GNOME Live" { set iso_path=/openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20221210-Media.iso set cdlabel=openSUSE_Tumbleweed_GNOME_Live } menuentry "openSUSE Tumbleweed KDE Live" --class os --unrestricted { set isoboot=iso-scan/filename=${iso_path} search --file --set=root ${iso_path} loopback loop ${iso_path} #set root=(loop) #set gfxpayload=keep echo Loading kernel... $linux (loop)/boot/x86_64/loader/linux ${cmd_persistent} ${cmd_safegraphics} ${isoboot} root=live:CDLABEL=${cdlabel} rd.live.image systemd.show_status=yes echo Loading initrd... $initrd (loop)/boot/x86_64/loader/initrd } ``` There is a mistake in my previous comment. /dev/disk/by-UUID directory actually existed there. But it was not updating when new loopback device is added. During the boot, it dropped to an emergency shell. There I checked. `losetup -a` was showing successful attachment of iso image to loopback device. but, its link was not present in the `/dev/disk/by-label` directoy. Then I run `udevadm trigger` and everything was populated correctly. Next time, 1. I put break in initqueue `rd.break=initqueue` in kernel param and removed iso-scan/filename 2. Manually setup loop devices and run udevadm trigger 3. Exit the shell This way I was able to boot the live images. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c16 --- Comment #16 from Fabian Vogt <fabian@ritter-vogt.de> --- (In reply to Harish Karumuthil from comment #15)
(In reply to Fabian Vogt from comment #14)
I don't think booting .isos that way is supported by kiwi.
It is supported by `dracut` . Please see their documentation here https://github.com/dracutdevs/dracut/blob/master/man/dracut.cmdline.7. asc#iso-scanfilename This same method will work for Opensuse-Leap, Fedora & Ubuntu live cd images. ( ubuntu uses different system but kernel parameter is same )
The issue is that openSUSE Live images use kiwi's modules instead of dracut, which are quite different.
Also. it will work for Tumbleweed also. For that we need to do the following. 1. Boot the ISO with above config. 2. System will start booting, but it will drop into a emergency shell. 3. Then run the command `udevadm trigger` 4. Exit the emergency shell. 5. Simply exit all other upcoming emergency shell prompts.
You can see successfully running Live OS
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1177900 http://bugzilla.opensuse.org/show_bug.cgi?id=1177900#c17 --- Comment #17 from Harish Karumuthil <harish2704@gmail.com> --- (In reply to Fabian Vogt from comment #16)
(In reply to Harish Karumuthil from comment #15)
The issue is that openSUSE Live images use kiwi's modules instead of dracut, which are quite different.
I just unpacked the initrd file from openSUSE livecd image and checked. I was able to find following files in the initrd 1. https://github.com/dracutdevs/dracut/blob/master/modules.d/90dmsquash-live/i... ( @ /usr/sbin/iso-scan ) 2. https://github.com/dracutdevs/dracut/blob/master/modules.d/90dmsquash-live/p... ( @ /usr/lib/dracut/hooks/cmdline/31-parse-iso-scan.sh ) It means opensuse live cd images uses dracut modules to run LiveOS. isn't it ? -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@suse.com