http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c1 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bpetkov@suse.com, | |tiwai@suse.com --- Comment #1 from Takashi Iwai --- IIRC, it's a new AMD machine, right? Is this specific to these machines? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c3 --- Comment #3 from Adrian Schröter --- yes, I see these kinds of errors only on them: processor : 23 vendor_id : AuthenticAMD cpu family : 21 model : 2 model name : AMD Opteron(tm) Processor 6348 stepping : 0 microcode : 0x6000836 However, I have to say that I only see it on two systems (lamb11 and 15) so far. A redeploy is running, let's see if they go back into this state. this is the first trace I see when searching from the beginning: [154559.080520] BUG: unable to handle kernel paging request at ffffffffb0ffd8d0 [154559.087820] IP: [<000056302566db83>] 0x56302566db83 [154559.093025] PGD 1e11067 PUD 1e12063 PMD 0 [154559.097453] Oops: 0002 [#1] PREEMPT SMP [154559.101735] Modules linked in: ipmi_devintf iscsi_ibft iscsi_boot_sysfs ipmi_si ipmi_msghandler kvm_amd kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper amd64_edac_mod ablk_helper cryptd joydev sp5100_tco edac_core pcspkr edac_mce_amd serio_raw fam15h_power k10temp i2c_piix4 shpchp acpi_cpufreq tpm_tis 8250_fintek tpm processor sg fuse dm_mod squashfs loop brd af_packet hid_generic usbhid syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper ohci_pci ehci_pci ohci_hcd igb ehci_hcd drm dca ptp usbcore pps_core crc32c_intel pata_atiixp usb_common i2c_algo_bit button [last unloaded: dm_mod] [154559.160976] CPU: 12 PID: 7739 Comm: qemu-system-x86 Not tainted 4.1.20-11-default #1 [154559.169226] Hardware name: Supermicro H8DGT /H8DGT , BIOS 3.5b 10/06/2015 [154559.177369] task: ffff8817fbd06290 ti: ffff88153d2fc000 task.ti: ffff88153d2fc000 [154559.185295] RIP: 0010:[<000056302566db83>] [<000056302566db83>] 0x56302566db83 [154559.193078] RSP: 0018:ffff88153d2ffdc0 EFLAGS: 00010283 [154559.198662] RAX: 00000000000000e8 RBX: 000056302566db80 RCX: 000060dfe1016010 [154559.206243] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880fc84e8010 [154559.213837] RBP: ffff880fc84e8010 R08: 0100028100000001 R09: 0000000000000000 [154559.225432] R10: ffffc90016b97000 R11: 0000000000000000 R12: ffff8817fbd06290 [154559.233051] R13: ffff88153d300000 R14: ffff8817fb81c040 R15: 0000000000000001 [154559.240647] FS: 00007f7f9364e700(0000) GS:ffff8817ffc00000(0000) knlGS:0000000000000000 [154559.249200] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [154559.255231] CR2: ffffffffb0ffd8d0 CR3: 000000153d042000 CR4: 00000000000406e0 [154559.262833] Stack: [154559.265107] ffffffffa04e784d 000000000000000c fffffffe7ffbfeff ffff880fc84e8010 [154559.273073] ffff88125034a840 ffff880fe0f7f200 ffff8817fbd06290 0000000000000000 [154559.281020] ffffffffa04d0429 00000000ffffffff 0000000000000000 0000000000000000 [154559.288989] Call Trace: [154559.291722] Inexact backtrace: [154559.296812] [<ffffffffa04e784d>] ? kvm_arch_vcpu_ioctl_run+0x2bd/0x430 [kvm] [154559.304405] [<ffffffffa04d0429>] ? kvm_vcpu_ioctl+0x329/0x620 [kvm] [154559.311040] [<ffffffff810e756d>] ? do_futex+0x10d/0x4d0 [154559.316631] [<ffffffff811f218f>] ? do_vfs_ioctl+0x2ff/0x510 [154559.322567] [<ffffffff810e799e>] ? SyS_futex+0x6e/0x150 [154559.328157] [<ffffffff811fbb88>] ? __fget+0x68/0xa0 [154559.333399] [<ffffffff811f2421>] ? SyS_ioctl+0x81/0xa0 [154559.338898] [<ffffffff81664f32>] ? system_call_fastpath+0x16/0x75 [154559.345357] Code: Bad RIP value. [154559.348982] RIP [<000056302566db83>] 0x56302566db83 [154559.354248] RSP <ffff88153d2ffdc0> [154559.358006] CR2: ffffffffb0ffd8d0 [154559.362028] ---[ end trace 46561098e05d9468 ]--- -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c5 Borislav Petkov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adrian@suse.com Flags| |needinfo?(adrian@suse.com) --- Comment #5 from Borislav Petkov --- Yikes, rIP corruption this time. Even with the fixed microcode... I don't have another idea right now besides maybe trying latest upstream 4.6 on one of them (would that be even possible?) to rule out some bug the fix for which we haven't backported yet... Also, how reproducible is this? Are you seeing the same type of splat or everytime different ones? Can you copy all dmesg logs somewhere for me to stare at? I'm assuming those machines have ECC DIMMs and you have mcelog running on them? Can you point me to a full dmesg of one from the beginning? Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c7 Ruediger Oertel changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ro@suse.com) | --- Comment #7 from Ruediger Oertel --- ah new OBS worker images ... and I'm pretty sure they do not have the updated AMD microcode included or at least not the hack to update the microcode upon boot ... ping me for the pointers ... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c9 Adrian Schröter changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #9 from Adrian Schröter --- k, building the kernel-firmware from Factory for 42.1 and will re-deploy. sorry for the noise, I thought we had merged everything. Let's close this, will reopen if it is not working. thanks a lot for your assistance! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c11 --- Comment #11 from Borislav Petkov --- Yep, it does. I just looked at http://download.opensuse.org/repositories/openSUSE:/Leap:/42.1/standard/src/... and amd-ucode/microcode_amd_fam15h.bin in there has the b0rked patch 0x06000832. The updated version is already here git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=979853 http://bugzilla.suse.com/show_bug.cgi?id=979853#c13 --- Comment #13 from Takashi Iwai --- I also submitted MR 396950 for openSUSE:Leap:42.1:Update/kernel-firmware, just syncing with the current FACTORY package. -- You are receiving this mail because: You are on the CC list for the bug.