https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c1 --- Comment #1 from Graham Anderson 2011-02-25 18:53:01 UTC --- Created an attachment (id=416330) --> (http://bugzilla.novell.com/attachment.cgi?id=416330) zypper DUP'd boot failure, similar to installation kernel crash This photo is from the system which was upgraded via zypper dup. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c3 --- Comment #3 from Graham Anderson 2011-02-25 18:57:20 UTC --- Created an attachment (id=416332) --> (http://bugzilla.novell.com/attachment.cgi?id=416332) boot crash on RC1 upgraded system Another boot crash on the upgraded system -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c5 --- Comment #5 from Graham Anderson 2011-02-25 18:59:56 UTC --- Created an attachment (id=416334) --> (http://bugzilla.novell.com/attachment.cgi?id=416334) /var/log/messages /var/log/messages from the only time I was able to boot to runlevel 3, the system hung almost immediately after I tried to login -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c7 --- Comment #7 from Graham Anderson 2011-02-27 12:00:35 UTC --- Created an attachment (id=416429) --> (http://bugzilla.novell.com/attachment.cgi?id=416429) Oops booting 2.6.37.1 (default kernel) Oops booting 2.6.37.1 (default kernel) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c8 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO CC| |jslaby@novell.com InfoProvider| |graham@andtech.eu --- Comment #8 from Jiri Slaby 2011-02-27 22:03:59 UTC --- It looks like a perfect memory corruption bug. Does it happen if you blacklist/delete the microcode module? Maybe it's in initrd too (check by lsinitrd). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c10 --- Comment #10 from Jiri Slaby 2011-02-28 14:43:20 UTC --- (In reply to comment #9)

Using kernel 2.6.37.2-1-default, I checked the initrd and the microcode module is not there. I blacklisted microcode in modprobe.d and also manually moved the microcode.ko file out of the modules tree.

I see the same re-occuring boot crashes/hangs as per the previous attachments.

Well, so it's some other module or kernel part causing this. It might be worth trying to feed the output through mcelog.

Additionally, when I previously saw the MCE panics, I made sure to boot to memtest on both machines, I quit out of memtest after 4 pass cycles.

I don't think it's a HW problem as I saw some of those already. But who knows. Did it happen for example with 11.3 or other distros (if you tried it)? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c11 Graham Anderson changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|graham@andtech.eu | --- Comment #11 from Graham Anderson 2011-03-08 16:35:47 UTC --- (In reply to comment #10)

Well, so it's some other module or kernel part causing this. It might be worth trying to feed the output through mcelog.

I will attempt to do this, I'm currently waiting on an rs232 cable arriving so I can boot with a serial console. Curiously, I recall seeing "failed" messages beside mcelog when trying to boot with the 2.6.37-desktop and 2.6.37.1-destkop kernels in RC1/2

Did it happen for example with 11.3 or other distros (if you tried it)?

I didn't have any issues with the 2.6.34 kernels from 11.3, I've not had time or cause to install another distro but will maybe try to find time. I'm not sure what other distros have been recently released that would be using 2.6.37 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c13 --- Comment #13 from Graham Anderson 2011-03-11 10:16:15 UTC --- Further narrowing down the boot params... 2.6.37.1 boots OK with nohz=off, and after doing a bit of poking on b.k.o I found this. https://bugzilla.kernel.org/show_bug.cgi?id=29992 So for me, 2.6.37.1 boot just fine if I pass intel_idle.max_cstate=0 I will try the following options as well and report back, as per the bug thread "nolapic_timer" or "hpet=disable" -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c15 --- Comment #15 from Jiri Slaby 2011-03-11 14:39:24 UTC --- (In reply to comment #14)

I have the following results for 2.6.37.1

pass only: intel_idle.max_cstate=0 result: System boots normally and behaves as expected with no further issues

Ok, so revert of: commit 0f212b87548cc4598fb7c77d92bfef23d5ee4d1a Author: Shaohua Li Date: Mon Jan 24 08:00:01 2011 +0000 fix a shutdown regression in intel_idle commit ec30f343d61391ab23705e50a525da1d55395780 upstream. commit 0f076e96eae1e03f5fd988911c7062dee22e14a6 Author: Shaohua Li Date: Mon Jan 10 09:38:12 2011 +0800 intel_idle: open broadcast clock event commit 2a2d31c8dc6f1ebcf5eab1d93a0cb0fb4ed57c7c upstream. should help here. The sideeffect of the revert is that somebody won't be able to boot because their machine will stall. But regressions take precedence. I'll build a kernel with those reverted to test. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c17 Graham Anderson changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|graham@andtech.eu | --- Comment #17 from Graham Anderson 2011-03-13 10:25:40 UTC --- Same results as per comment 14, i.e. panic/oops on normal boot unless intel_idle.max_cstate=0 is passed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c18 Thomas Renninger changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |len.brown@intel.com, | |rjw@novell.com AssignedTo|rjw@novell.com |trenn@novell.com --- Comment #18 from Thomas Renninger 2011-04-16 16:55:55 UTC --- Sounds as if lapic timer stops in deeper idle states? I added a patch to our master branch: Can you fetch it in some hours (should get build and pushed automatically): ftp://ftp.suse.com/pub/projects/kernel/kotd/master/x86_64 kernel-default.rpm and kernel-default-base.rpm (or desktop). Best double check whether the patch is really included by: rpm -qp --changelog kernel-default.rpm |less commit 85b156b3a18467a940cb72308116b51acb689a1e Author: Thomas Renninger Date: Sat Apr 16 18:42:01 2011 +0200 - intel_idle: Make lapic_timer_reliable_states a boot param (bnc#675161). To get the initial value of lapic_timer_reliable_states on your machine you can boot with these params: nolapic ignore_log_level ddebug_query="file intel_idle.c +p" If it's bigger than 1, then set it to 1 with the new kernel with boot param: intel_idle.lapic_timer_reliable_states=1 Does this help? Hm, I wonder why there is kind of random memory corruption when the timer stops, maybe it's something else (leave_mm()/TLB flushing?), but as it's related to lapic, it might help and it's easy to test... I have to look at this driver anyway, I can take over the bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c Thomas Renninger changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |graham@andtech.eu -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c21 --- Comment #21 from Len Brown 2011-04-17 11:55:01 UTC ---

Point of failure of installation kernel (x86_64 RC2 DVD)

That one sure looks like a hardware error... Hmm, two systems failed with the same Fatal Machine Check? There are three, apparently independent, crashes in this bug report...

intel_idle.max_cstate=0

How about intel_idle.max_cstate=1 or intel_idle.max_cstate=2 or intel_idle.max_cstate=3 please attache the complete dmesg for a boot with intel_idle driver loaded.

pass only: hpet=disable result: system sometimes boots, but completely locks up soon after

pleaes attach the complete dmesg from one of the boots that makes it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c23 --- Comment #23 from Graham Anderson 2011-04-20 07:26:36 UTC ---

Do you remember which kernel was the last one working?

From 11.3 so I guess 2.6.34.7

I've been out of town, I'll make some time before the weekend to follow your suggestions/kernel build from comment #18 and comment #20 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c25 --- Comment #25 from Tim Manchester 2011-08-01 20:05:47 UTC --- Hi Sorry if I should have put this message in another place first time I have done this. Ok So I have read the bug report above and it looks exactly the same as an issue I have. Key difference I am using x86 (32 bit version) running on a jetway J7F4 motherboard which is a via c7 single cpu. Installation was built from SuSe studio based on 11.4 plus additions since installation. Linux holly 2.6.37.6-0.5-default #1 SMP i686 i686 i386 GNU/Linux openSUSE 11.4 (i586) VERSION = 11.4 CODENAME = Celadon With acpi enabled - Lock ups (dead with two flashing keyboard lights number lock cap lock) generally occur at boot time but can happen during any time frame up to maximum of 1 hours approximately. Tried the intel_idle.max_cstate=1 this appeared to extended the uptime before eventual failure. Plus various other combinations no luck. With acpi disabled - No problems except of course I can't power the machine down, from shutdown -hP now. What additional information could be useful to help diagnose the issue ? Regards Tim -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c27 --- Comment #27 from Graham Anderson 2011-09-25 11:06:43 UTC --- Created an attachment (id=452879) --> (http://bugzilla.novell.com/attachment.cgi?id=452879) dmesg booting with nolapic ignore_log_level ddebug_query="file intel_idle.c +p" RE comment #18 , comment #19 & comment #20 12.1 M5 Install with zypper dup to latest factory @ 25/09/2011 -- cenuij@mercury:~> cat /etc/SuSE-release openSUSE 12.1 Milestone 5 (x86_64) VERSION = 12.1 CODENAME = Asparagus cenuij@mercury:~> uname -a Linux mercury 3.1.0-rc6-2-desktop #1 SMP PREEMPT Thu Sep 15 20:18:41 UTC 2011 (532c24f) x86_64 x86_64 x86_64 GNU/Linux cenuij@mercury:~> dmesg | grep lapic_timer [ 0.298234] intel_idle: lapic_timer_reliable_states 0x2 -- boot with only: intel_idle.lapic_timer_reliable_states=1 result: panic boot with only: intel_idle.tlb_flushing=1 result: panic -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c29 --- Comment #29 from Graham Anderson 2011-09-25 20:16:08 UTC --- (In reply to comment #26)

Did someone ever try a desktop instead a default kernel? This could be related to: Bug 672008 - [i915, mtrr] Complete system freeze at start While there it's a real deadlock, the wrong page states could be related to wrong mtrr settings. And the fact that -default kernel is used very much sounds like above bug.

kernel 2.6.37.6-0.5-desktop and 3.1.0-rc6-2-desktop exhibits the same problems

If desktop kernel works, default should also work again with a recent kernel from here: ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD/x86_64

I am unable to browse/cd anonymously to that location to try a KOTD -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c31 --- Comment #31 from Graham Anderson 2011-09-25 20:51:41 UTC --- Created an attachment (id=452896) --> (http://bugzilla.novell.com/attachment.cgi?id=452896) oops after booting to runlevel3 with hpet=disable booting with hpet=disable I'm not sure if this log is chaff/noise because boot was using systemd unlike previously. Anyway things fell over straight after I logged this dmesg to file after logging in at runlevel 3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c33 Thomas Renninger changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |graham@andtech.eu --- Comment #33 from Thomas Renninger 2011-09-26 09:51:14 UTC ---

boot with: intel_idle.max_cstate=0 OR boot with: intel_idle.max_cstate=1 result: success always, see attached dmesg

boot only: intel_idle.max_cstate=2 or boot with: intel_idle.max_cstate=3 result: panic

With intel_idle.max_cstate=0 the acpi idle driver should get used. If this one works, the reason might be that we should ignore a specific idle state on purpose? Can you attach acpidump and dmidecode and run: cpupower idle-info (with intel_idle.max_cstate=0 boot param set) May need zypper install cpupower, package did not exist on 11.4 yet. Have you also already tried Nvidia's binary graphics driver? If not, it would be great if you could give it a try. Memory accesses which cause MCEs could likely be graphics driver related. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c35 --- Comment #35 from Graham Anderson 2011-09-26 16:52:12 UTC --- (In reply to comment #33)

Can you attach acpidump and dmidecode and run: cpupower idle-info (with intel_idle.max_cstate=0 boot param set) May need zypper install cpupower, package did not exist on 11.4 yet.

mercury:/ # grep . /sys/devices/system/cpu/cpuidle/* /sys/devices/system/cpu/cpuidle/current_driver:acpi_idle /sys/devices/system/cpu/cpuidle/current_governor_ro:menu mercury:/ # cpupower idle-info CPUidle driver: acpi_idle CPUidle governor: menu Analyzing CPU 0: CPU 0: No idle states

Have you also already tried Nvidia's binary graphics driver? If not, it would be great if you could give it a try. Memory accesses which cause MCEs could likely be graphics driver related.

With nouveau blacklisted (and removed from initrd) and using the Nvidia blobs MCE panics still appear in all previously problematic boot configurations, both with kernels 2.6.37.6 for 11.4 and 3.x kernels for 11.4 and 12.1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c37 Graham Anderson changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|graham@andtech.eu | --- Comment #37 from Graham Anderson 2011-09-26 16:58:43 UTC --- Created an attachment (id=453058) --> (http://bugzilla.novell.com/attachment.cgi?id=453058) acpidump acpidump warns "Wrong checksum for generic table!" on the console between EUDS & TAMG e.g. 0550: d3 44 10 00 00 00 6c 00 00 00 53 43 85 7c 85 7c .D....l...SC.|.| Wrong checksum for generic table! TAMG @ 0xdfbe6ba0 0000: 54 41 4d 47 32 0a 00 00 01 10 47 42 54 20 20 20 TAMG2.....GBT -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c39 --- Comment #39 from Len Brown 2011-09-26 17:47:05 UTC --- so with intel_idle.max_cstate=0, there is nothing under /sys/devices/system/cpu/cpu0/cpuidle ? Is the default BIOS SETUP configuration being used? Are there any BIOS SETUP options for processor power management related to C-states? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=675161 https://bugzilla.novell.com/show_bug.cgi?id=675161#c41 Graham Anderson changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |CLOSED Resolution| |FIXED --- Comment #41 from Graham Anderson 2011-09-27 00:51:07 UTC --- The issue persists after setting BIOS to defaults however there appears to be an issue with toggling and _saving_ the turbo boost option. I twice attempted to boot with parameters known to trigger mce panics e.g. intel_idle.max_cstates=2, using the following BIOS settings. Both boots failed. Intel(R) Turbo Boost Tech.: Disable CPU Cores Enabled: All CPU Multi-Threading: Disable CPU Enhanced Halt(C1E): Enable C3/C6/C7 State Support: Enable CPU EIST Function: Enable Bi-Directional PROCHOT: Enable The turbo boost option state did not save after toggling it to "Disable" between both boot attempts. It flipped back to "Auto". I see there's several newer BIOS releases for the board (Gigabyte p55-ud3) and I see one of them has a release note: "Improve Intel Turbo Boost compatibility". I'm going to flash a newer BIOS and give it one last whirl for this evening. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.