[Bug 1190326] New: Update to kernel 5.14 makes disks disapear/system unbootable
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 Bug ID: 1190326 Summary: Update to kernel 5.14 makes disks disapear/system unbootable Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Major Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: dennis@glindhart.dk QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Updating to kernel 5.14 (Tumbleweed Sep 5 2021 snapshot) makes system unbootable when running on top of Xen-server. This seems like a general kernel 5.14 bug, not specific to openSuSE/Tumbleweed as I can reproduce on Fedora 34 (by updating to 5.14 testing kernel from koji). 5.13 works perfectly. On Tumbleweed, during boot it hangs in "starting dracut initqueue hook" and times out waiting for /dev/disk/by-uuid/xxxx It seems like kernel cannot identify the disks at all (working fine on 5.13) After entering the recovery-mode, follow is observed: - lsblk shows only the CD-rom drive - ls /dev/disk/ only has by-path and by-id. by-uuid directory is gone. I've reproduced the problem in cleanly installed Tumbleweed (both MicroOS/Ignition and install from ISO), and clean installed Fedora 34. Following kernel versions tested/are affected: 5.14.0 and 5.14.1 (5.14.1 tested on Fedora). Can be reproduced on Fedora with both BTRFS and XFS/LVM install, so not specific to BTRFS. Reproduced on two seperate XCP-ng 8.2 (xen-based) Hypervisors. From hypervisors: # yum info qemu Installed Packages Name : qemu Arch : x86_64 Epoch : 2 Version : 2.10.2 Release : 4.5.3.xcpng8.2 Size : 19 M Repo : installed From repo : install Summary : qemu-dm device model License : GPL Description : This package contains Qemu. # uname -a Linux hostname 4.19.0+1 #1 SMP Wed Dec 16 12:16:11 CET 2020 x86_64 x86_64 x86_64 GNU/Linux -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c1 --- Comment #1 from Dennis Glindhart <dennis@glindhart.dk> --- Reproduced on 5.14.2 too -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c2 Takashi Iwai <tiwai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hare@suse.com, | |tiwai@suse.com --- Comment #2 from Takashi Iwai <tiwai@suse.com> --- Likely something about storage: Hannes, please distribute in the team. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c3 Hannes Reinecke <hare@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dennis@glindhart.dk Flags| |needinfo?(dennis@glindhart. | |dk) --- Comment #3 from Hannes Reinecke <hare@suse.com> --- Sure, as soon as I got logfiles. Can you please attach the kernel message log here? -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c4 --- Comment #4 from Dennis Glindhart <dennis@glindhart.dk> --- Created attachment 852555 --> http://bugzilla.opensuse.org/attachment.cgi?id=852555&action=edit rdsosreport.txt -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c5 --- Comment #5 from Dennis Glindhart <dennis@glindhart.dk> --- Created attachment 852556 --> http://bugzilla.opensuse.org/attachment.cgi?id=852556&action=edit journalctl -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c6 Dennis Glindhart <dennis@glindhart.dk> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(dennis@glindhart. | |dk) | --- Comment #6 from Dennis Glindhart <dennis@glindhart.dk> --- rdososreport and journalctl output attached -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c7 --- Comment #7 from Dennis Glindhart <dennis@glindhart.dk> --- Seems the xen-blkfront module is somehow not available/loaded for the 5.14 kernel. Kernel 5.13: # lsmod | grep xen_ xen_netfront 45056 1 xen_blkfront 49152 6 # ls /lib/modules/5.13.13-1-default/kernel/drivers/block | grep xen xen-blkback xen-blkfront.ko.xz Kernel 5.14 (In emergency mode) # lsmod | grep xen_ xen_netfront 45056 1 # ls /lib/modules/5.14.0-1-default/kernel/drivers/block | grep xen (empty) -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c8 --- Comment #8 from Dennis Glindhart <dennis@glindhart.dk> --- Seems filesystem (/lib/modules/5.xxx-default) is different in emergency mode than what is on the actual filesystem on disk, so above comment might not mean anything. On the actual snapshot the module is present on filesystem (when booting into work snapshot and looking at the new (non-working snapshot) # ls /.snapshots/102/snapshot/lib/modules/5.14.2-1-default/kernel/drivers/block/ | grep xen xen-blkback xen-blkfront.ko.xz -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c9 --- Comment #9 from Dennis Glindhart <dennis@glindhart.dk> --- Making sure the xen-blkfront module is being added to the initrd seems to solve the problem: # echo 'add_drivers+="xen-blkfront"' > /etc/dracut.conf.d/1-xen.conf # transactional-update -c <NEWSNAPSHOTWITHKERNEL5.14> initrd Then system can reboot into the new kernel/snapshot. Any idea what change makes this necessary? (xen)-modules not being loaded automatically anymore? Is there a better/more sensible way of doing this? And is it possible to somehow make the upgrade path smoother not breaking peoples existing systems when updating kernel? (I will try to read/understand/experiment with how to combine this with Ignition/Bootstrapping on MicroOS) -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c10 Dennis Glindhart <dennis@glindhart.dk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ohering@suse.com --- Comment #10 from Dennis Glindhart <dennis@glindhart.dk> --- *** Bug 1190814 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1190326 http://bugzilla.opensuse.org/show_bug.cgi?id=1190326#c11 --- Comment #11 from Dennis Glindhart <dennis@glindhart.dk> --- Note that this, besides happening on "ordinary" Tumbleweed's, also happens on the MicroOS image for Qemu KVM & XEN. So Xen-drivers not being included in that distribution is definitely not to be expected. It can be worked around on existing systems with above workaround, but I haven't managed to bootstrap a new system from scratch (unless using old images with pre 5 sep image). -- You are receiving this mail because: You are the assignee for the bug.
participants (1)
-
bugzilla_noreply＠suse.com