http://bugzilla.opensuse.org/show_bug.cgi?id=1177800 http://bugzilla.opensuse.org/show_bug.cgi?id=1177800#c1 --- Comment #1 from Andreas Färber --- (In reply to Andreas Färber from comment #0)

Running `getsysinfo` (via ssh) on Tumbleweed 20201011 with kernel-default 5.9.0 from Kernel:HEAD repository caused a kernel error, with ssh getting stuck and reconnections failing. Serial login still worked.

Correction: I got the login prompt on Enter key, but login timed out. After reset and reboot the /tmp tarball was gone. Sadly getsysinfo did not support any command line argument for output location. Re-trying, I was able to log in via serial, copy the file elsewhere, but during reboot got stuck in RCU errors and had to reset again. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1177800 http://bugzilla.opensuse.org/show_bug.cgi?id=1177800#c3 Andreas Färber changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jcheung@suse.com, | |lpechacek@suse.com, | |ptesarik@suse.com --- Comment #3 from Andreas Färber --- For comparison, on a MacchiatoBin with kernel 5.8.14 `getsysinfo` also produces one kernel error on serial console, but the machine remains usable: [ 1252.719947] BUG: Bad page state in process getsysinfo pfn:7fe40 mack:~ # getsysinfo /proc/bus/input /proc/cpuinfo /proc/device-tree /proc/devices /proc/fb /proc/filesystems /proc/interrupts /proc/iomem /proc/ioports /proc/meminfo /proc/modules /proc/net/dev /proc/partitions /proc/scsi /proc/tty /proc/version /sys /var/lib/hardware/udi /proc/mounts System data written to: /tmp/mack.tar.gz mack:~ # -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1177800 http://bugzilla.opensuse.org/show_bug.cgi?id=1177800#c6 --- Comment #6 from Mathew McBride --- I had a fresh look into this today and managed to find the cause of the problem! In summary the Layerscape PCIe controller generates a synchronous abort related to reading PCI config data for the PCIe switch/bridge. This read does not happen in normal operation but is triggered by getsysinfo archiving/enumerating the /sys tree, where one can read out the pci config register as a file. The synchronous abort problem exists in mainline kernels / non SUSE systems as well. The Ten64 retail (1064-0201C) board has a Diodes/Pericom PI7C9X2G304SV PCIe switch to split 1xPCIe lane to 2xPCIe 2.0 for the miniPCIe slots lspci -nn 0000:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10) 0001:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10) 0001:01:00.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01) 0001:02:01.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01) 0001:02:02.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:b304] (rev 01) 0001:03:00.0 Unclassified device [0002]: MEDIATEK Corp. MT7915E 802.11ax PCI Express Wireless Network Adapter [14c3:7915] 0001:04:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c] 0002:00:00.0 PCI bridge [0604]: Freescale Semiconductor Inc Device [1957:80c0] (rev 10) 0002:01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a] root@recovery000afa24295d:/tmp# lspci -tnn -+-[0002:00]---00.0-[01-ff]----00.0 +-[0001:00]---00.0-[01-ff]----00.0-[02-04]--+-01.0-[03]----00.0 | \-02.0-[04]----00.0 If the PCIe switch is hidden (disable it's upstream PCIe controller in the FDT blob) or missing (it's been removed from some Ten64 board variants), the problem does not occur and getsysinfo will not cause a panic. FreeBSD had a similar issue and the cause sounds very similar to what is happening here. "pci: Don't try to read cfg registers of non-existing devices Instead of returning 0xffs some controllers, such as Layerscape generate an external exception when someone attempts to read any register of config space of a non-existing device other than PCIR_VENDOR. This causes a kernel panic. Fix it by bailing during device enumeration if a device vendor register returns invalid value. (0xffff) Use this opportunity to replace some hardcoded values with a macro." From https://cgit.freebsd.org/src/commit/?id=68cbe189fdd3c572476f8af9219a5d335f05... I have been able to isolate it down to the 'config' sysfs file, here is a reduced testcase: for i in $(find /sys/devices/platform/soc/3500000.pcie -type f); do echo "Opening $i" echo "------------------------------------------" sleep 1 # allow time for console to flush cat $i echo "------------------------------------------" done .... ------------------------------------------ Opening /sys/devices/platform/soc/3500000.pcie/pci0001:00/0001:00:00.0/0001:01:00.0/0001:02:02.0/config ------------------------------------------ [ 150.192901] Internal error: synchronous external abort: 96000210 [#1] SMP I have verified the problem exists on non-SUSE systems so it's just a kernel bug (including 5.19.0-rc5) which getsysinfo triggers. Here is the trace from the latest Tumbleweed snapshot: openSUSE-Tumbleweed-ARM-JeOS-efi.aarch64-2022.07.01-Snapshot20220704.raw.xz Linux localhost.localdomain 5.18.6-1-default #1 SMP PREEMPT_DYNAMIC Thu Jun 23 05:46:18 UTC 2022 (5aa0763) aarch64 aarch64 aarch64 GNU/Linux [ 36.849750][ T2016] Internal error: synchronous external abort: 96000210 [#1] SMP [ 36.857252][ T2016] Modules linked in: af_packet mt7915e ath10k_pci ath10k_co re mt76_connac_lib mt76 ath mac80211 libarc4 fsl_dpaa2_eth pcs_lynx cfg80211 phy link rfkill i2c_mux_pca954x i2c_mux pci_endpoint_test tpm_i2c_atmel qoriq_therma l tee sfp uio_pdrv_genirq mdio_i2c leds_gpio uio qoriq_cpufreq nls_iso8859_1 nls _cp437 vfat fat fuse drm ip_tables x_tables xhci_plat_hcd xhci_hcd caam_jr crypt o_engine usbcore dpaa2_caam caamhash_desc caamalg_desc aes_ce_blk aes_ce_cipher crct10dif_ce ghash_ce gf128mul sha2_ce sha256_arm64 sha1_ce sp805_wdt fsl_mc_dpi o dpaa2_console authenc libdes caam nvme nvme_core error dwc3 sdhci_of_esdhc sdh ci_pltfm sdhci udc_core rtc_fsl_ftm_alarm roles mmc_core ulpi i2c_imx usb_common gpio_keys btrfs blake2b_generic xor xor_neon raid6_pq libcrc32c dm_mirror dm_re gion_hash dm_log dm_mod sg [ 36.929404][ T2016] CPU: 0 PID: 2016 Comm: cp Not tainted 5.18.6-1-default #1 openSUSE Tumbleweed a3ce01492e87efb4fa7f3baf169c992c0c69c4b7 [ 36.941846][ T2016] Hardware name: traverse ten64/ten64, BIOS 2020.07-rc1-ga9 4e0d21 03/15/2022 [ 36.950460][ T2016] pstate: 204000c5 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTY PE=--) [ 36.958119][ T2016] pc : pci_generic_config_read+0x44/0xcc [ 36.963613][ T2016] lr : pci_generic_config_read+0x30/0xcc [ 36.969099][ T2016] sp : ffff80000a31b9f0 [ 36.973105][ T2016] x29: ffff80000a31b9f0 x28: ffff08be45472400 x27: 00000000 00000400 [ 36.980941][ T2016] x26: 00000000000003ff x25: ffff08be45472000 x24: 00000000 00001000 [ 36.988779][ T2016] x23: 0000000000001000 x22: ffff80000a31bae4 x21: ffffbac4 1ea22fa0 [ 36.996616][ T2016] x20: ffff80000a31ba64 x19: 0000000000000004 x18: 00000000 00000000 [ 37.004453][ T2016] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000 00000000 [ 37.012289][ T2016] x14: 0000000000000000 x13: 0000000000000000 x12: 00000000 00000000 [ 37.020132][ T2016] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffbac4 1ca785dc [ 37.027975][ T2016] x8 : 0000000000000004 x7 : ffff800008e00000 x6 : ffff8000 08e00000 [ 37.035816][ T2016] x5 : ffff08be41a93c80 x4 : 0000000000000908 x3 : 00000000 00000000 [ 37.043656][ T2016] x2 : 0000000000000000 x1 : ffff08be4a0de000 x0 : ffff8000 08202400 [ 37.051494][ T2016] Call trace: [ 37.054631][ T2016] pci_generic_config_read+0x44/0xcc [ 37.059774][ T2016] dw_pcie_rd_other_conf+0x24/0x7c [ 37.064741][ T2016] pci_user_read_config_dword+0x84/0x124 [ 37.070229][ T2016] pci_read_config+0xf0/0x2a0 [ 37.074760][ T2016] sysfs_kf_bin_read+0x78/0xa0 [ 37.079378][ T2016] kernfs_fop_read_iter+0xac/0x1d4 [ 37.084344][ T2016] new_sync_read+0xd8/0x160 [ 37.088700][ T2016] vfs_read+0x19c/0x1e4 [ 37.092710][ T2016] ksys_read+0x78/0x10c [ 37.096718][ T2016] __arm64_sys_read+0x28/0x34 [ 37.101248][ T2016] invoke_syscall+0x78/0x100 [ 37.105693][ T2016] el0_svc_common.constprop.0+0x58/0x190 [ 37.111181][ T2016] do_el0_svc+0x30/0x90 [ 37.115191][ T2016] el0_svc+0x34/0x130 [ 37.119029][ T2016] el0t_64_sync_handler+0x10c/0x140 [ 37.124080][ T2016] el0t_64_sync+0x1a0/0x1a4 [ 37.128439][ T2016] Code: 7100067f 540001c0 71000a7f 54000280 (b9400001) [ 37.135228][ T2016] ---[ end trace 0000000000000000 ]--- [ 37.140539][ T2016] note: cp[2016] exited with preempt_count 1 And from Leap 15.4: Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) aarch64 aarch64 aarch64 GNU/Linux [ 445.922445][ T2950] Call trace: [ 445.925582][ T2950] pci_generic_config_read+0x40/0x100 [ 445.930810][ T2950] dw_pcie_rd_other_conf+0x20/0x80 [ 445.935777][ T2950] pci_user_read_config_dword+0x88/0x140 -- You are receiving this mail because: You are the assignee for the bug.