https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c1 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |frank.arnold@amd.com --- Comment #1 from Jan Beulich 2008-05-21 09:00:18 MST --- I don't think the 'time went backward messages', given the not too high delta, indicate a significant problem. Nevertheless, once the box locks up we'd need you to obtain some state information (sending 'd' over serial to the hypervisor as a first step). Also, more complete messages (to understand how bad the time issue is) should be attached. Likewise we'd want to see the hypervisor messages over the whole lifetime of the system (please use 'loglvl=all guest_loglvl=all' on the Xen command line). Finally, narrowing down the conditions for the problem to occur would be rather helpful. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User frank.arnold@amd.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c3 --- Comment #3 from Frank Arnold 2008-05-26 07:35:17 MDT --- Created an attachment (id=218117) --> (https://bugzilla.novell.com/attachment.cgi?id=218117) serial console output -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c5 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- CC|jbeulich@novell.com | QAContact|qa@suse.de |jdouglas@novell.com --- Comment #5 from Jan Beulich 2008-05-28 01:46:04 MDT ---

There's plain nothing on the output that would indicate a problem, it just hangs. We don't have the resources to investigate this further, so here is a summary of the last run (froze about 8 hours after start). Log files will be attached.

Unless we would be able to reproduce this in our lab, there's pretty little we can do if you're unavailable to assist with the analysis. According to the logs you didn't even get Xen's response to pressing 'd' once the box hung (of course I can't tell whether you tried and it didn't work). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c6 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Info Provider|jdouglas@novell.com |frank.arnold@amd.com --- Comment #6 from Jan Beulich 2008-05-28 02:06:00 MDT --- The kernel log provided could hint at a TSC instability, although I didn't think Phenoms would do hardware initiated clock modulation affecting the TSC tick rate behind the back of the OS. The box doesn't appear to have a HPET, so you should try clocksource=pit to see whether this alternatively has something to do with the PM timer latency issues currently being fixed upstream. Also, I take it for granted that the problem happens regardless of the sync_console and watchdog command line options. As the watchdog doesn't appear to trigger, it's pretty certain you'd be able to get output from Xen once hung - please get the output from pressing 'd' as indicated above (and be sure to switch input to Xen before attempting this). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User frank.arnold@amd.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c9 Frank Arnold changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|frank.arnold@amd.com | --- Comment #9 from Frank Arnold 2008-06-05 07:20:17 MDT --- OK, here we go. (In reply to comment #6 from Jan Beulich)

The kernel log provided could hint at a TSC instability, although I didn't think Phenoms would do hardware initiated clock modulation affecting the TSC tick rate behind the back of the OS.

That's true, Phenoms don't have this issue.

The box doesn't appear to have a HPET, so you should try clocksource=pit to see whether this alternatively has something to do with the PM timer latency issues currently being fixed upstream.

Tried clocksource=pit and got the same result.

Also, I take it for granted that the problem happens regardless of the sync_console and watchdog command line options.

When I first saw this issue no debug options were used. Just a clean install.

As the watchdog doesn't appear to trigger, it's pretty certain you'd be able to get output from Xen once hung - please get the output from pressing 'd' as indicated above (and be sure to switch input to Xen before attempting this).

Done. Serial logs from runs with clocksources PIT and ACPI PM are attached. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c11 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |frank.arnold@amd.com --- Comment #11 from Jan Beulich 2008-06-05 08:27:35 MDT --- In order to make sense of the back traces - what exact kernel version did you run in Dom0? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c13 --- Comment #13 from Jan Beulich 2008-06-05 09:50:39 MDT --- All vCPU-s visible in the backtraces which apparently belong to dom0 are waiting for the xtime_lock. This may be a time handling issue in Xen, but it also may be a scheduler issue (in that the vCPU holding the lock doesn't get scheduled within a reasonable amount of time). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User thomas.friebel@amd.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c16 Thomas Friebel changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |thomas.friebel@amd.com --- Comment #16 from Thomas Friebel 2008-06-18 04:30:46 MDT --- Lock-holder preemption is a real problem for systems w/ ticket spinlocks (2.6.25+). Under CPU overcommitment the ticket locks will prevent the system from making progress. Exchanging the spinlock implementation with the 2.6.24 version will probably solve this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 Thomas Friebel changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Normal |Major OS/Version|Other |openSUSE 11.0 Priority|P5 - None |P2 - High Platform|Other |x86 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c24 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |frank.arnold@amd.com --- Comment #24 from Jan Beulich 2008-10-27 02:46:57 MDT --- Could this be tried with 11.1/SLE11 beta 4 (as soon as available), where the ticket lock implementation and a false positive issue in the reporting of 'time went backwards' have been fixed? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=392585 User frank.arnold@amd.com added comment https://bugzilla.novell.com/show_bug.cgi?id=392585#c26 Frank Arnold changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|frank.arnold@amd.com | --- Comment #26 from Frank Arnold 2008-11-07 10:50:45 MST --- (In reply to comment #24 from Jan Beulich)

Could this be tried with 11.1/SLE11 beta 4 (as soon as available)

Done, and it's looking nice. Used a similar setup to the one mentioned in comment #2. openSUSE 11.1 Beta3 x86_64 ran for 104 hours without issues. No 'time went backwards' messages. SLE11 Beta4 x86_64 ran for 88 hours without issues. Also no 'time went backwards' messages. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.