http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c2 Mathew McBride changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |afaerber@suse.com --- Comment #2 from Mathew McBride --- CC Andreas: Are you able to reproduce on your Ten64? I assume SLES 15 SP3 is also affected. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c4 --- Comment #4 from Mathew McBride --- Created attachment 850592 --> http://bugzilla.opensuse.org/attachment.cgi?id=850592&action=edit error on loading ahci - possibly related PCIe issue I suspect this might be a general PCIe issue instead of just nvme. I setup a system with a SATA SSD via a PCIe controller card (JMicron JMB545 based) and it failed to boot as well. Booting from USB I was able to blacklist nvme and ahci, and then obtain a crashdump when nvme was modprobe'd after boot. The NVMe crashdump has been uploaded to: ftp://support-ftp.us.suse.com/incoming/bug-1187701-ten64-nvme-crash.tar modprobe of ahci did not cause a kernel crash but the call trace has been attached. I also have an ath10k_pci wireless card (QCA6174) in this system and 'modprobe ath10k_pci' simply stalls with no error messages. So a general PCIe issue is quite likely here. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c6 --- Comment #6 from Daniel Wagner ---

The NVMe crashdump has been uploaded to: ftp://support-ftp.us.suse.com/incoming/bug-1187701-ten64-nvme-crash.tar

I get no such file. Could you upload it to ziu? -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c8 --- Comment #8 from Daniel Wagner --- I am sorry Mathew. I have to ask our L3 folks (they have access to the ftp servers) to help out with uploading it to ziu. Anyway, your download link works for me so we don't have to bother L3. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c10 --- Comment #10 from Daniel Wagner --- Before the nvme core crashes this happens right before:

[ 78.869666] nvme 0002:01:00.0: Adding to iommu group 3 [ 78.875156] nvme nvme0: pci function 0002:01:00.0 [ 78.879934] Internal error: synchronous external abort: 96000210 [#2] SMP

-- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c14 --- Comment #14 from Daniel Wagner --- The ahci crash happens doing the readl in:

static void ahci_enable_ahci(void __iomem *mmio) { int i; u32 tmp;

/* turn on AHCI_EN */ tmp = readl(mmio + HOST_CTL); if (tmp & HOST_AHCI_EN) return;

/* Some controllers need AHCI_EN to be written multiple times. * Try a few times before giving up. */ for (i = 0; i < 5; i++) { tmp |= HOST_AHCI_EN; writel(tmp, mmio + HOST_CTL); tmp = readl(mmio + HOST_CTL); /* flush && sanity check */ if (tmp & HOST_AHCI_EN) return; msleep(10); }

WARN_ON(1); }

The ath10k crash happens doing a ioread32 in:

static bool ath10k_pci_is_awake(struct ath10k *ar) { struct ath10k_pci *ar_pci = ath10k_pci_priv(ar); u32 val = ioread32(ar_pci->mem + PCIE_LOCAL_BASE_ADDRESS + RTC_STATE_ADDRESS);

return RTC_STATE_V_GET(val) == RTC_STATE_V_ON; }

-- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c16 --- Comment #16 from Daniel Wagner --- Created attachment 850710 --> http://bugzilla.opensuse.org/attachment.cgi?id=850710&action=edit dmesg-ahci.txt -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c18 J�rgen Gro� changed: What |Removed |Added ---------------------------------------------------------------------------- CC|jgross@suse.com |jroedel@suse.com Flags|needinfo?(jgross@suse.com) |needinfo?(jroedel@suse.com) --- Comment #18 from J�rgen Gro� --- I'm not Joerg. Adding him. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c20 --- Comment #20 from Daniel Wagner --- No ARM expert either. Internal error: synchronous external abort: 96000210 [#1] SMP Stackoverflow[1] says: """ The ARMv7 ARM section "VMSA Memory aborts" covers this as thoroughly as one would expect (given that it's the authoritative definition of the architecture), but to summarise in slightly less than 14 pages; An abort means the CPU tried to make a memory access, which for whatever reason, couldn't be completed so raises an exception. An external abort is one from, well, externally to the processor, i.e. something on the bus. In other words, the access didn't fault in the MMU, went out onto the bus, and either some device or the interconnect itself came back and said "hey, I can't deal with this". A synchronous external abort means you're rather fortunate, in that it's not going to be utterly hideous to debug - in the case of a prefetch abort, it means the IFAR is going to contain a valid VA for the faulting instruction, so you know exactly what caused it. The unpleasant alternative is an asynchronous external abort, which is little more than an interrupt to say "hey, something you did a while ago didn't actually work. No I don't know what is was either." """ and the number is the ESR register content. Seems to be a very common register value. Found a lot of bug reports with this value. There is a bit of documentation on it [2] and [3]. Not sure if this helps at all. [1] https://stackoverflow.com/questions/27507013/synchronous-external-abort-on-a... [2] https://developer.arm.com/documentation/ddi0595/2020-12/AArch64-Registers/ES... [2] https://developer.arm.com/documentation/den0024/a/AArch64-Exception-Handling -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c22 --- Comment #22 from Daniel Wagner --- #define ESR_ELx_FSC (0x3F) static inline const struct fault_info *esr_to_fault_info(unsigned int esr) { return fault_info + (esr & ESR_ELx_FSC); } Ok, it's just the index and the fault_info table contains

{ do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" },

TL;DR: the value says 'synchronous external abort' :) -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1187701 http://bugzilla.opensuse.org/show_bug.cgi?id=1187701#c31 Mathew McBride changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|NORESPONSE |--- --- Comment #31 from Mathew McBride --- Apologies, I got distracted on other projects. This is still a problem, thankfully it does not exist with the newer kernel on Leap 15.4. I have even attempted to bisect this but can't find any obvious cause. I think it came with this merge: commit 691336b369202a4fd8d75a0a255a88ef592f00d9 Merge: 30270eafb6d5 2c269e3edad3 Author: Denis Kirjanov Date: Fri May 21 15:57:15 2021 +0300 Merge branch 'users/dkirjanov/SLE15-SP3-UPDATE/for-next' into SLE15-SP3-UPDATE Merge SLE15-SP2 into SLE15-SP3-UPDATE with a kabi fix from Michal Suchanek suse-commit: d970db8f5beecbc55b223922ed9f3d6dbb885814 A kernel compiled on the commit immediately prior (according to Github) - d4cb7424a2afeb740fd05840619e26de413aed45 "lpfc: Decouple port_template and vport_template (bsc#185032)" works.

Might this be an issue with the DTB? Can you please try to boot the broken kernel with a DTB from the good kernel and see whether this makes a difference?

Note that I've grabbed the dmesg's/kernel logs from several different systems, depending on what was in front of me at the time. On the Ten64 the DTB is built into the flash and passed to the kernel via EFI, so the DTB is from a 'good kernel' in that respect. I can try producing a DTB from the Leap/SLE kernel source if needed. -- You are receiving this mail because: You are the assignee for the bug.