https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c2 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jcarricksmith@gmail.com, | |tiwai@suse.com Flags| |needinfo?(jcarricksmith@gma | |il.com) --- Comment #2 from Takashi Iwai --- Yeah, the most important bits are missing in the log. Any chance to get the full stack traces? -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c4 --- Comment #4 from John Carrick Smith --- 1. Tried with intel_iommu=off on the boot command line and it made no difference. 2. How do I produce the full stack trace please? Thanks John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c6 John Carrick Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jcarricksmith@gma | |il.com) | --- Comment #6 from John Carrick Smith --- Created attachment 861609 --> https://bugzilla.suse.com/attachment.cgi?id=861609&action=edit Output from journalctl I tried using the alt-sysrq-? sequence and it didn't appear to make any difference. I tried a second time. 1. I left it in the 'hung' state for 3 minutes. 2. Pressed ctrl-alt-del, then ctrl-alt-ins, then ctrl-alt-bksp. 3. Then followed the alt-sysrq-? sequence. There is more data in the attached file than in the previous - I hope it is what you are looking for. John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c8 --- Comment #8 from Jan Kara --- For reference the softlockup is: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd-udevd:711] Modules linked in: idma64 thermal fjes(-) ac acpi_pad intel_hid(N) acpi_cpufreq(-) sparse_keymap fuse configfs ip_tables x_tables ext4 crc16 mbcache jbd2 hid_generic usbhid rtsx_pci_sdmmc i915 mmc_core sd_mod i2c_algo_bit ttm drm_kms_helper nvme crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt nvme_core fb_sys_fops cec ahci rtsx_pci rc_core libahci xhci_pci nvme_common xhci_pci_renesas ghash_clmulni_intel xhci_hcd aesni_intel drm crypto_simd cryptd libata usbcore serio_raw mfd_core t10_pi i2c_hid_acpi battery i2c_hid wmi video pinctrl_cannonlake button v4l2loopback(OEN) videodev mc sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod msr efivarfs Supported: No, Unsupported modules are loaded CPU: 0 PID: 711 Comm: systemd-udevd Tainted: G OE N 5.14.21-150400.24.21-default #1 SLE15-SP4 7550826c4c7e8c258239e300508e0c8b2a69bad2 Hardware name: Novatech PB50_70DFx,DDx /PB50_70DFx,DDx , BIOS 1.07.07TNO 09/25/2019 RIP: 0010:smp_call_function_many_cond+0x126/0x560 RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 RDX: 0000000000000011 RSI: 0000000000000000 RDI: ffff9416c04794c0 RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff9416c0434a40 R13: ffff9416c0434a40 R14: ffffffff9f38d650 R15: 00000000000394c0 FS: 00007f340f894b00(0000) GS:ffff9416c0400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000564868903ce8 CR3: 000000010a608002 CR4: 00000000007706f0 PKRU: 55555554 Call Trace: <TASK> ? __brelse+0x30/0x30 on_each_cpu_cond_mask+0x25/0x40 kill_bdev.isra.31+0x16/0x30 blkdev_flush_mapping+0x46/0xf0 blkdev_put_whole+0x3a/0x50 blkdev_put+0x57/0x180 blkdev_close+0x21/0x30 __fput+0x8f/0x250 ? __SCT__preempt_schedule_notrace+0x8/0x8 task_work_run+0x70/0xb0 exit_to_user_mode_prepare+0x228/0x230 syscall_exit_to_user_mode+0x18/0x40 do_syscall_64+0x67/0x80 ... So we are actually stuck in the smp_call_function_many_cond() call. The work queued on each CPU is actually pretty trivial (read per-cpu variable and decrease one refcount). So likely this is not related to any change in the block layer but rather due to some CPU getting stuck with interrupts disabled so we cannot execute work on it. Maybe this is somehow related to the USB devices that are probed by mtp-probe when the hang happens? -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c10 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(jcarricksmith@gma | |il.com) --- Comment #10 from Takashi Iwai --- What kind of device is this? I guess almost same slot should be assigned at the boot with the old working kernel. Please give the hwinfo output and the dmesg output from the working kernel. And, if it's an external USB device, could you try to boot without it once? -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c12 --- Comment #12 from Takashi Iwai --- (In reply to John Carrick Smith from comment #11)

I removed external USB devices one by one and even with none plugged in the boot hung in the usual place. I have the journals from those if required.

Yes, the log without USB devices is still helpful. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c14 --- Comment #14 from John Carrick Smith --- Created attachment 861652 --> https://bugzilla.suse.com/attachment.cgi?id=861652&action=edit Journalctl and mkinitrd output Rather than removing 69-libmtp.rules from /usr/lib/udev/rules.d/ I followed the idea in the man page for udev and linked /etc/udev/rules.d/69-libmtp.rules to /dev/null which disabled the file in /usr/lib/udev/rules.d/ . It made no difference. I included the output from /var/log/Yast2/mkinitrd.log in case there was anything in there that might help. The file jrnl_no_USB is the journal from a hang when there were no devices plugged in to the computer. John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c17 John Carrick Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jcarricksmith@gma | |il.com) | --- Comment #17 from John Carrick Smith --- Created attachment 861741 --> https://bugzilla.suse.com/attachment.cgi?id=861741&action=edit journal output from KASAN enabled kernel Hello Takashi, I downloaded the KASAN enabled kernel as installed as instructed. I needed to install a MOK. The kernel booted to a command prompt login. I have attached the journal to this report. This was not what I was expecting! I logged in as root successfully. Thank you for your help John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c19 --- Comment #19 from John Carrick Smith --- The problem still occurs with the intel driver. I used 'prime-select boot intel' as root and it still hung on boot. When I switched back to the working OS it didn't give me a log in, but ctrl-alt-F1 and a CLI login solved the problem. John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c21 --- Comment #21 from John Carrick Smith --- Would installation of a debug kernel help? I have no understanding but just wondered. John -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1203539 https://bugzilla.suse.com/show_bug.cgi?id=1203539#c26 --- Comment #26 from John Carrick Smith --- Now running with 5.14.21-150400.24.28-default for several days - all OK. Many thanks. John -- You are receiving this mail because: You are the assignee for the bug.