https://bugzilla.suse.com/show_bug.cgi?id=1190411
Bug ID: 1190411 Summary: USB devices missing after kexec reboot with XHCI enabled Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: roger.whittaker@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: ---
Created attachment 852446 --> https://bugzilla.suse.com/attachment.cgi?id=852446&action=edit hwinfo, dmesg, usb-devices and lsusb with and without XHCI enabled in BIOS, normal boot and kexec boot
Hardware is an Intel NUC (D54250WYK).
With XHCI enabled in the UEFI BIOS, USB devices are not available after a kexec reboot. Hence the boot failed in the initrd when I had USB attached disks referenced in /etc/fstab. When I commented these out, the boot succeeded but the keyboard and mouse did not work after a kexec reboot.
With XHCI disabled in the BIOS, kexec works as expected.
On this hardware using Tumbleweed at some time in the past (but I'm afraid I don't know when) kexec worked with XHCI enabled.
Linux jabberwock 5.14.0-1-default #1 SMP Mon Aug 30 07:01:36 UTC 2021 (dc06e24) x86_64 x86_64 x86_64 GNU/Linux
NAME="openSUSE Tumbleweed" # VERSION="20210908" ID="opensuse-tumbleweed" ID_LIKE="opensuse suse" VERSION_ID="20210908" PRETTY_NAME="openSUSE Tumbleweed" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:tumbleweed:20210908" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org/" DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed" LOGO="distributor-logo"
https://bugzilla.suse.com/show_bug.cgi?id=1190411
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |roger.whittaker@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1190411
Takashi Iwai tiwai@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |oneukum@suse.com, | |tiwai@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c1
Oliver Neukum oneukum@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(roger.whittaker@s | |use.com)
--- Comment #1 from Oliver Neukum oneukum@suse.com --- Are hotplug events still working? What happens if with XHCI enabled after a kexec you
a) unplug and replug a device b) plug a device into a hitherto unused port
please attach dmesg of both tests
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c2
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(roger.whittaker@s | |use.com) |
--- Comment #2 from Roger Whittaker roger.whittaker@suse.com --- Nothing appears in dmesg if I unplug and replug or plug into an unused port.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c3
--- Comment #3 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 852502 --> https://bugzilla.suse.com/attachment.cgi?id=852502&action=edit dmesg
Attached dmesg as requested. Once the system was up I unplugged and replugged both keyboard and mouse (connected through a hub) and two USB disks (connected directly).
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c4
--- Comment #4 from Roger Whittaker roger.whittaker@suse.com --- To be clear, I plugged the disks into an unused port.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c5
--- Comment #5 from Oliver Neukum oneukum@suse.com --- Please test the kernel at https://build.opensuse.org/repositories/home:oneukum:bsc1190411_20210914a once it will have built
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c6
--- Comment #6 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 852522 --> https://bugzilla.suse.com/attachment.cgi?id=852522&action=edit tests with kernel-default-5.15~rc1-1.1.g8787773.x86_64.rpm
I'm afraid I'm seeing the same behaviour with the test kernel kernel-default-5.15~rc1-1.1.g8787773.x86_64.rpm as I saw before. After a kexec boot the USB devices are missing.
https://bugzilla.suse.com/show_bug.cgi?id=1190411
Nicola Di Marzo nicola.dimarzo@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |nicola.dimarzo@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c7
--- Comment #7 from Roger Whittaker roger.whittaker@suse.com --- After a kexec reboot I tried (over ssh) removing and modprobing the relevant modules:
rmmod xhci_pci xhci_hcd usbcore
modprobe xhci_pci xhci_hcd usbcore
The devices did not reappear, but this led to a long trace in dmesg which I'll attach.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c8
--- Comment #8 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 852820 --> https://bugzilla.suse.com/attachment.cgi?id=852820&action=edit dmesg mentioned in comment#7
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c9
--- Comment #9 from Roger Whittaker roger.whittaker@suse.com --- I'm sorry - that was wrong. I'll test again shortly.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c10
--- Comment #10 from Roger Whittaker roger.whittaker@suse.com --- Corrected test:
rmmod xhci_pci rmmod xhci_hcd rmmod xhci_pci_renesas
then
modprobe xhci_pci modprobe xhci_hcd modprobe xhci_pci_renesas
then unplug and replug mouse, keyboard and two USB disks.
dmesg after doing these:
[ 85.218000] xhci_hcd 0000:00:14.0: remove, state 1 [ 85.218010] usb usb2: USB disconnect, device number 1 [ 85.218482] xhci_hcd 0000:00:14.0: USB bus 2 deregistered [ 85.238848] xhci_hcd 0000:00:14.0: xHCI host controller not responding, assume dead [ 85.238856] xhci_hcd 0000:00:14.0: remove, state 1 [ 85.238864] usb usb1: USB disconnect, device number 1 [ 85.239378] xhci_hcd 0000:00:14.0: Host halt failed, -19 [ 85.239387] xhci_hcd 0000:00:14.0: Host not accessible, reset failed. [ 85.239586] xhci_hcd 0000:00:14.0: USB bus 1 deregistered [ 97.099036] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 97.099052] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1 [ 97.100102] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x000000000004b810 [ 97.100252] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.14 [ 97.100254] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 97.100256] usb usb1: Product: xHCI Host Controller [ 97.100257] usb usb1: Manufacturer: Linux 5.14.6-1-default xhci-hcd [ 97.100258] usb usb1: SerialNumber: 0000:00:14.0 [ 97.100505] hub 1-0:1.0: USB hub found [ 97.100524] hub 1-0:1.0: 9 ports detected [ 97.102777] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 97.102784] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 97.102790] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed [ 97.102842] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.14 [ 97.102847] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 97.102850] usb usb2: Product: xHCI Host Controller [ 97.102852] usb usb2: Manufacturer: Linux 5.14.6-1-default xhci-hcd [ 97.102854] usb usb2: SerialNumber: 0000:00:14.0 [ 97.103272] hub 2-0:1.0: USB hub found [ 97.103289] hub 2-0:1.0: 4 ports detected
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c11
Oliver Neukum oneukum@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(roger.whittaker@s | |use.com)
--- Comment #11 from Oliver Neukum oneukum@suse.com --- We have to make sure that the Renesas 64 bit quirk is applied properly. Could you rerun the test with dynamic debugging for the xhci_hcd module enabled with
echo "module xhci_pci +mfp" > /sys/kernel/debug/dynamic_debug/control echo "module xhci_hcd +mfp" > /sys/kernel/debug/dynamic_debug/control
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c12
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(roger.whittaker@s | |use.com) |
--- Comment #12 from Roger Whittaker roger.whittaker@suse.com --- I did a kexec reboot and then:
echo "module xhci_pci +mfp" > /sys/kernel/debug/dynamic_debug/control echo "module xhci_hcd +mfp" > /sys/kernel/debug/dynamic_debug/control
rmmod xhci_pci rmmod xhci_hcd rmmod xhci_pci_renesas
modprobe xhci_pci modprobe xhci_hcd modprobe xhci_pci_renesas
echo "module xhci_pci +mfp" > /sys/kernel/debug/dynamic_debug/control echo "module xhci_hcd +mfp" > /sys/kernel/debug/dynamic_debug/control
The relevant dmesg extract is:
[ 30.181244] xhci_hcd 0000:00:14.0: remove, state 1 [ 30.181252] usb usb2: USB disconnect, device number 1 [ 30.181407] xhci_hcd:xhci_check_args: xHCI xhci_drop_endpoint called for root hub [ 30.181410] xhci_hcd:xhci_check_args: xHCI xhci_check_bandwidth called for root hub [ 30.181496] xhci_hcd 0000:00:14.0: USB bus 2 deregistered [ 30.198306] xhci_hcd 0000:00:14.0: xHCI host controller not responding, assume dead [ 30.198314] xhci_hcd 0000:00:14.0: remove, state 1 [ 30.198322] usb usb1: USB disconnect, device number 1 [ 30.198623] xhci_hcd:xhci_check_args: xHCI xhci_drop_endpoint called for root hub [ 30.198629] xhci_hcd:xhci_check_args: xHCI xhci_check_bandwidth called for root hub [ 30.198863] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: // Halt the HC [ 30.198871] xhci_hcd 0000:00:14.0: Host halt failed, -19 [ 30.198876] xhci_hcd 0000:00:14.0: Host not accessible, reset failed. [ 30.198942] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: // Disabling event ring interrupts [ 30.198950] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: cleaning up memory [ 30.198958] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed event ring [ 30.198964] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed command ring [ 30.198972] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed segment pool [ 30.198977] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed device context pool [ 30.198981] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed small stream array pool [ 30.198985] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: Freed medium stream array pool [ 30.199091] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: xhci_stop completed - status = ffffffff [ 30.199094] xhci_hcd 0000:00:14.0: USB bus 1 deregistered [ 36.106382] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 36.106397] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1 [ 36.107444] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x000000000004b810 [ 36.107594] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.14 [ 36.107597] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 36.107598] usb usb1: Product: xHCI Host Controller [ 36.107599] usb usb1: Manufacturer: Linux 5.14.6-1-default xhci-hcd [ 36.107601] usb usb1: SerialNumber: 0000:00:14.0 [ 36.108112] hub 1-0:1.0: USB hub found [ 36.108134] hub 1-0:1.0: 9 ports detected [ 36.111654] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 36.111663] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 36.111669] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed [ 36.111725] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.14 [ 36.111729] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 36.111732] usb usb2: Product: xHCI Host Controller [ 36.111734] usb usb2: Manufacturer: Linux 5.14.6-1-default xhci-hcd [ 36.111736] usb usb2: SerialNumber: 0000:00:14.0 [ 36.111996] hub 2-0:1.0: USB hub found [ 36.112009] hub 2-0:1.0: 4 ports detected
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c13
--- Comment #13 from Roger Whittaker roger.whittaker@suse.com --- (And I also unplugged and replugged.)
If I do those debug commands on the normally booted system and then unplug and replug devices, I see a *lot* of debugging information in dmesg.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c14
--- Comment #14 from Oliver Neukum oneukum@suse.com --- For the record:
[ 30.181496] xhci_hcd 0000:00:14.0: USB bus 2 deregistered [ 30.198306] xhci_hcd 0000:00:14.0: xHCI host controller not responding, assume dead [ 30.198314] xhci_hcd 0000:00:14.0: remove, state 1 [ 30.198322] usb usb1: USB disconnect, device number 1 [ 30.198623] xhci_hcd:xhci_check_args: xHCI xhci_drop_endpoint called for root hub [ 30.198629] xhci_hcd:xhci_check_args: xHCI xhci_check_bandwidth called for root hub [ 30.198863] xhci_hcd:xhci_dbg_trace: xhci_hcd 0000:00:14.0: // Halt the HC [ 30.198871] xhci_hcd 0000:00:14.0: Host halt failed, -19 [ 30.198876] xhci_hcd 0000:00:14.0: Host not accessible, reset failed.
The HC has already failed at that point.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c15
Oliver Neukum oneukum@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED Flags| |needinfo?(roger.whittaker@s | |use.com)
--- Comment #15 from Oliver Neukum oneukum@suse.com --- Please test the kernel building at https://build.opensuse.org/project/monitor?project=home%3Aoneukum%3Absc11904...
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c16
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(roger.whittaker@s | |use.com) |
--- Comment #16 from Roger Whittaker roger.whittaker@suse.com --- I'm afraid I'm seeing the same behaviour. The attempt to mount the disks times out and there is no working keyboard.
After commenting out the USB disks in fstab and testing again, I went in over ssh and got the following:
# lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# usb-devices
T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 9 D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=01 MxPS=64 #Cfgs= 1 P: Vendor=1d6b ProdID=0002 Rev=05.16 S: Manufacturer=Linux 5.16.0-rc2-1.g696d453-default xhci-hcd S: Product=xHCI Host Controller S: SerialNumber=0000:00:14.0 C: #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=0mA I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
T: Bus=02 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=5000 MxCh= 4 D: Ver= 3.00 Cls=09(hub ) Sub=00 Prot=03 MxPS= 9 #Cfgs= 1 P: Vendor=1d6b ProdID=0003 Rev=05.16 S: Manufacturer=Linux 5.16.0-rc2-1.g696d453-default xhci-hcd S: Product=xHCI Host Controller S: SerialNumber=0000:00:14.0 C: #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=0mA I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub E: Ad=81(I) Atr=03(Int.) MxPS= 4 Ivl=256ms
dmesg output attached separately.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c17
--- Comment #17 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 853938 --> https://bugzilla.suse.com/attachment.cgi?id=853938&action=edit dmesg mentioned above
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c18
--- Comment #18 from Roger Whittaker roger.whittaker@suse.com --- Also replugging devices had no effect, and nothing seen in dmesg when doing so.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c19
Oliver Neukum oneukum@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(roger.whittaker@s | |use.com)
--- Comment #19 from Oliver Neukum oneukum@suse.com --- Please repeat the test from comment#10 with this test kernel and
pci=nomsi iommu=off
added to the kernel command line in both boots.
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c20
--- Comment #20 from Roger Whittaker roger.whittaker@suse.com --- Both boots:
# uname -r ; cat /proc/cmdline 5.16.0-rc2-1.g696d453-default BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc2-1.g696d453-default root=UUID=a81717ee-f292-40fc-8701-12e6e4591665 linux resume=/dev/disk/by-uuid/957eb50f-8f08-47e1-b805-aba1235a78fc plymouth.enable=0 pci=nomsi iommu=off
# rmmod xhci_pci ; rmmod xhci_hcd ; rmmod xhci_pci_renesas
# modprobe xhci_pci ; modprobe xhci_hcd ; modprobe xhci_pci_renesas
(unplug and replug mouse, keyboard and two disks)
# dmesg
dmesg from each will now be attached.
Would you like me to also do the debugging stuff as in comment#12?
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c21
--- Comment #21 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 853964 --> https://bugzilla.suse.com/attachment.cgi?id=853964&action=edit dmesg from normal boot
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c22
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(roger.whittaker@s | |use.com) |
--- Comment #22 from Roger Whittaker roger.whittaker@suse.com --- Created attachment 853965 --> https://bugzilla.suse.com/attachment.cgi?id=853965&action=edit dmesg from kexec boot
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c23
Jiri Slaby jslaby@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-bugs@opensuse.org |oneukum@suse.com Flags| |needinfo?(roger.whittaker@s | |use.com)
--- Comment #23 from Jiri Slaby jslaby@suse.com --- Oliver, any other ideas?
Roger, is it still defunct?
https://bugzilla.suse.com/show_bug.cgi?id=1190411 https://bugzilla.suse.com/show_bug.cgi?id=1190411#c24
Roger Whittaker roger.whittaker@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(roger.whittaker@s | |use.com) |
--- Comment #24 from Roger Whittaker roger.whittaker@suse.com --- Yes - just tested.
I'm seeing the same behaviour with Tumbleweed 20230123, 6.1.7-1-default.
-
bugzilla_noreply＠suse.com