http://bugzilla.novell.com/show_bug.cgi?id=623680 http://bugzilla.novell.com/show_bug.cgi?id=623680#c1 --- Comment #1 from Martin Wilck 2010-07-19 21:33:12 UTC --- I just retried with "sysrq=9" parameter and saw some more messages. The last ones were ehci_hcd 0000:00.1d.7: cache line size of 32 not supported ehci_hcd 0000:00.1d.7: irq 23, io mem 0xc0000000 in the default kernel, the following happens next: [ 2.486964] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xc0000000 [ 2.500081] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00 [ 2.500128] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 2.500131] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.500134] usb usb1: Product: EHCI Host Controller -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=623680 http://bugzilla.novell.com/show_bug.cgi?id=623680#c Daniel Rahn changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium CC| |daniel.rahn@novell.com -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=623680 http://bugzilla.novell.com/show_bug.cgi?id=623680#c3 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Found By|--- |Third Party | |Developer/Partner InfoProvider| |martin.wilck@ts.fujitsu.com --- Comment #3 from Jan Beulich 2010-08-06 11:04:11 UTC --- In the hope that the machine has a serial port or you can reproduce this on one that has, we'll need a full log (Xen and kernel) along with the native kernel's /var/log/boot.msg for comparison. That would then also allow for getting a state dump from Xen (sending either '0' or 'd' depending on what component it is that hangs). With your last observation it may as well be some interrupt handling issue, so it would be really useful to know whether any other (earlier) Xen release works on that system. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c5 --- Comment #5 from Martin Wilck 2010-08-16 21:38:34 UTC --- I got the system up now in ACPI mode with "nomodeset" and by blacklisting the "thermal", "fan", and "processor" (and removing them from the initrd). Once the system is booted, all three modules can be loaded without error - I am typing in the Dom0 now. Apparently there's a problem only at boot time. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c7 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |martin.wilck@ts.fujitsu.com --- Comment #7 from Jan Beulich 2010-08-17 07:14:32 UTC --- Since this is machine specific, we will still need a log including the requested debug output (or alternatively access to a machine we can debug this on). As an alternative to using a serial cable you could try whether the NMI watchdog works on that box ("watchdog" on the Xen command line) - if it does and if it triggers, you could put the screen in a large enough mode so that you could take a snapshot of the full printed information (requires that you also make Xen continue printing to the VGA console post-boot via attaching ",keep" to the end of your "vga=" Xen option). Another thing to try would be to use interactive startup mode (resulting in less parallelism of the boot process) to see whether that also hangs (if it doesn't it would point to a concurrency/serialization issue). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c9 --- Comment #9 from Martin Wilck 2010-08-17 19:54:56 UTC --- With "watchdog", the system automatically reboots. The system comes up with "processor.max_cstate=1". -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c11 --- Comment #11 from Martin Wilck 2010-08-18 19:25:13 UTC --- (In reply to comment #10)

...

With "watchdog", the system automatically reboots.

Even with "noreboot"?

No, but I haven't succeeded to get any XEN messages on the console when the system freezes. "vga=mode-0x314,keep" should be correct for me, shouldn't it? Btw the SUSE menu.lst default XEN option "vgamode=0x314" is wrong, it should be "vga=mode-0x314" instead.

Does the same also hold when passing "max_cstate=1" to Xen?

Yes. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c13 --- Comment #13 from Jan Beulich 2010-08-19 08:50:48 UTC --- (In reply to comment #11)

"vga=mode-0x314,keep" should be correct for me, shouldn't it?

Yes, that's how it is supposed to be.

Btw the SUSE menu.lst default XEN option "vgamode=0x314" is wrong, it should be "vga=mode-0x314" instead.

That would need to be a separate (YaST) bug report. (In reply to comment #12)

This screenshot was made with "vga=mode-0x31A,keep watchdog noreboot" for XEN, and "debug sysrq=9" for the kernel. I also activated ACPI debugging for the processor module.

I'll look at this in more detail later. Could you attach the corresponding native kernel boot's boot.msg for reference?

Is there a way to force udev to load drivers serially at boot time? I tried udev.log_priority=debug udev.children_max=16 on the kernel command line, but it had no effect.

What does udev have to do with this problem? Supposedly you have processor.ko in the list of modules to be loaded explicitly from initrd (i.e. its loading isn't controlled by udev). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c15 --- Comment #15 from Martin Wilck 2010-08-19 09:06:31 UTC --- Opened bug #632730 for the "vgamode" problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c17 --- Comment #17 from Jan Beulich 2010-08-19 12:09:47 UTC --- (In reply to comment #14)

Now that I know I can boot with max_cstate=1, I can put processor.ko back into INITRD_MODULES if you prefer.

Depends on whether keeping it the way you have it now makes debugging easier. Not sure whether adding the module to MODULES_LOADED_ON_BOOT would make the loading of the module (indirectly) controllable - boot.loadmodules runs after boot.udev, but a comment there says its purpose is to control load order. If that works, simply turning on interactive startup mode (PROMPT_FOR_CONFIRM="yes" in /etc/sysconfig/boot) would allow you to separate the individual steps. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c Ihno Krumreich changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ihno@novell.com OS/Version|Other |openSUSE 11.3 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c20 --- Comment #20 from Martin Wilck 2010-09-29 09:54:17 UTC --- Sorry this is my home machine and I'm currently loaded. I will follow up on this one but I need to ask for some more patience. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c22 --- Comment #22 from Martin Wilck 2010-10-01 20:01:05 UTC --- Created an attachment (id=392653) --> (http://bugzilla.novell.com/attachment.cgi?id=392653) boot.msg of default kernel -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c24 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #24 from Jan Beulich 2010-10-04 08:34:39 UTC --- I see no way to deal with this without having a way to debug it. We can only declare "max_cstate=1" (or "cpuidle=0") as workarounds until we'll find a way to actually debug the problem. Btw., did you ever try running the xen-unstable hypervisor underneath the OpenSuSE (or SLE11 SP1) kernel? If the issue reproduces there (which I assume it will), we might have better chances of getting it addressed by publishing the problem on xen-devel. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c26 --- Comment #26 from Jan Beulich 2010-10-04 10:44:34 UTC --- (In reply to comment #25)

I was hoping to get some debugging hints here.

I would really want to give you some, but for cpuidle specific ones I can't really (not knowing the code well enough), and anything generic would involve having serial console access. This is why reproducing against xen-unstable and, if reproducible, reporting on xen-devel is likely the only realistic chance. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c28 --- Comment #28 from Martin Wilck 2010-10-06 15:20:21 UTC --- The same happens with xen-unstable, both with the OS11.3 kernel and with Jeremy Fitzhardinge's 2.6.32-next kernel. The difference is that I can't use xenpm and xm to communicate with the 4.1 hypervisor, both the OS11.3 tools and the self-compiled tools from xen-unstable always fail with "permission denied". Another difference is that if I load processor.ko after the system is up (which is causing no hang, just as originally reported), I see no /proc/acpi/processor/CPU0/power with the 4.1 hypervisor (with the SUSE one, the file is present). With Jeremy's kernel, I also tried to put ACPI processor support into the kernel proper (rather than a module). This causes the functionality to be initialized in an initcall. The system hangs in this case, too. Do you want me to mail to xen-devel with this information? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c30 --- Comment #30 from Martin Wilck 2010-10-19 21:34:12 UTC --- I sent my mail to xen-devel on Oct. 8 (Jan, you should have received the cc), but it hasn't been received apparently. I also got no error mail. Do I need to be subscribed to xen-devel to post to it? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c32 --- Comment #32 from Jan Beulich 2011-03-31 08:54:57 UTC --- Created an attachment (id=422391) --> (http://bugzilla.novell.com/attachment.cgi?id=422391) serial log on 11.4 with debug key output -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c34 --- Comment #34 from Jan Beulich 2011-03-31 09:42:48 UTC --- Perhaps this <6>[ 2.220466] Marking TSC unstable due to TSC halts in idle from the native kernel log is the key. Xen has TSC recovery code for this case, but maybe it doesn't work or doesn't even get used on your system. I'm in the process of preparing a debugging patch... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c36 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |martin.wilck@ts.fujitsu.com --- Comment #36 from Jan Beulich 2011-03-31 09:57:25 UTC --- Will you be able to do a test hypervisor build for yourself, or do you depend on us providing one? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c38 --- Comment #38 from Jan Beulich 2011-03-31 14:52:31 UTC --- Charles, could you do a 32-bit 11.4 one-off debugging build with the above patch included for Martin? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |martin.wilck@ts.fujitsu.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c41 --- Comment #41 from Martin Wilck 2011-04-01 19:29:19 UTC --- Created an attachment (id=422842) --> (http://bugzilla.novell.com/attachment.cgi?id=422842) debug capture with clocksource=acpi -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #422843|application/cap |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c44 --- Comment #44 from Jan Beulich 2011-04-06 11:19:48 UTC --- Created an attachment (id=423413) --> (http://bugzilla.novell.com/attachment.cgi?id=423413) tentative fix -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c46 Charles Arnold changed: What |Removed |Added ---------------------------------------------------------------------------- InfoProvider|carnold@novell.com |martin.wilck@ts.fujitsu.com --- Comment #46 from Charles Arnold 2011-04-06 16:10:18 UTC --- (In reply to comment #45)

Charles, could you do another 32-bit 11.4 one-off test build with the above patch included (but for the moment without the earlier debugging patch, as I'm not certain the two would both apply when used together) for Martin?

The second version of the 32bit RPMs are available for download at, ftp://ftp.novell.com/forge/XenTechnicalPreview/openSUSE/11.4/debug/623680/ Please download and install all three RPMs (xen, xen-libs, and xen-tools). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c48 --- Comment #48 from Martin Wilck 2011-04-11 19:01:04 UTC --- When this hang occurs, not even the watchdog seems to work any more. I also can't get to the XEN console by pressing ctrl-a 3 times. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c50 --- Comment #50 from Jan Beulich 2011-04-12 07:11:20 UTC --- (In reply to comment #47)

With this fix, the XEN hypervisor stops after printing (XEN) TSC: 0:0:0 (XEN) TSC: 000b48cf9339:000048cf9394:000048cf937c (XEN) TSC only partially writable

Very bad, and completely unexpected. On my dual PentiumIII system, this works just fine (just that it doesn't have any C states, and hence the actual intended effect can't be verified), so for the moment it escapes me why it would hang for you. Will need to add some more debugging code to see whether we can spot where it actually dies (I expect this is not immediately after the printed message). (In reply to comment #49)

Jan, if I'm reading your patch correctly, it would simply disable C2 and higher on my system (if it worked). But I could do the same easier with the max_cstate=1 parameter, as we discovered early on.

Yes, but the goal is to have Xen notice this by itself instead of needing a command line option, the more that we're dealing with fully specified behavior here.

OTOH, the Linux kernel runs just fine on this system with C2 and C3. What is the deeper reason behind Xen's need to write the TSC when C2/C3 are enabled?

The reason for this lies in how Xen manages time, which is completely different from Linux. Making C2/C3 usable under Xen on this systems of yours is certainly theoretically doable, but I see no point in spending time here - you'll have to acknowledge your system is not the youngest anymore ;-) . -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c52 --- Comment #52 from Jan Beulich 2011-04-12 08:05:49 UTC --- Other than on -unstable, hpet_disable_legacy_broadcast() doesn't tolerate being called before hpet_broadcast_init(). Will attach a fixed patch in a few minutes. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c54 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |carnold@novell.com --- Comment #54 from Jan Beulich 2011-04-12 08:13:46 UTC --- Charles, could you do yet another 32-bit 11.4 one-off test build with the above patch included? I'm sorry for the hickup with the previous one. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c56 Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|martin.wilck@ts.fujitsu.com | --- Comment #56 from Martin Wilck 2011-04-12 18:35:38 UTC --- Created an attachment (id=424509) --> (http://bugzilla.novell.com/attachment.cgi?id=424509) Serial log from hypervisor with "tentative fix v2" With this patch, the XEN kernel comes up cleanly. Only C0 and C1 are used. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c58 --- Comment #58 from Martin Wilck 2011-04-12 20:19:57 UTC --- Just a very crude idea: On CPUs such as mine, where writing the TSC zeroes out the upper 32bits, wouldn't it be possible to treat the TSC as a 32bit timer only? Similar code is already in place for the 32bit and 24bit platform timers. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c60 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #60 from Jan Beulich 2011-04-28 07:25:25 UTC --- The fix is now queued for future maintenance updates on SLE11 SP1, 11.3, and 11.4. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=623680 https://bugzilla.novell.com/show_bug.cgi?id=623680#c62 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard|maint:released:11.3:40798 |maint:released:11.3:40798 | |maint:released:sle11-sp1:40 | |493 --- Comment #62 from Swamp Workflow Management 2011-05-31 20:01:52 UTC --- Update released for: libvirt, libvirt-debuginfo, libvirt-debugsource, libvirt-devel, libvirt-doc, libvirt-python, vm-install, xen, xen-debuginfo, xen-debugsource, xen-devel, xen-doc-html, xen-doc-pdf, xen-kmp-debug, xen-kmp-default, xen-kmp-pae, xen-kmp-trace, xen-kmp-vmi, xen-libs, xen-tools, xen-tools-domU Products: SLE-DEBUGINFO 11-SP1 (i386, x86_64) SLE-DESKTOP 11-SP1 (i386, x86_64) SLE-SDK 11-SP1 (i386, x86_64) SLE-SERVER 11-SP1 (i386, x86_64) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.