[Bug 584554] New: random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

newer
[Bug 647562] New: [yast-timezone]...

bugzilla_noreply＠novell.com

2 Mar 2010 2 Mar '10

09:34

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c0 Summary: random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25 Classification: openSUSE Product: openSUSE 11.2 Version: Final Platform: x86-64 OS/Version: openSUSE 11.2 Status: NEW Severity: Critical Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: samuel.kvasnica@ims.co.at QAContact: qa@suse.de Found By: --- Blocker: --- Created an attachment (id=345885) --> (http://bugzilla.novell.com/attachment.cgi?id=345885) syslog snippet User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100110 Mandriva Linux/1.9.1.7-0.1mdv2010.0 (2010.0) Firefox/3.5.7 We experience random "61s" CPU soft lockups on kernel 2.6.31.12-xen. We even verified the same behavior using the last head kernel 2.6.33-25-xen. We use drbd + xen configuration, 2 drbd partitions + 1 xen pvm guest. The system is running almost idle. We can observe (using snmp logging) rapid increase of load during the lockup. The system does not crash but the corresponding CPU will be loaded/blocked during this time. We are not able to determine what really triggers the lockup. High load does not seem to correlate. Typically, system would run up to 12hours cleanly before lockup happens. If lockup happens, more lockups will typically follow within few minutes. Afterwards, system runs cleanly again. Typically, there is the xen_safe_halt() on stack trace. The same system gives absolutely no lockups on older hardware (2-core Pentium-D, 8G RAM), 1 month uptime. Our hardware: -Supermicro X8SIL-F -Chipset Ibex Peak, XEON X3450 -8GB ECC RAM -Last BIOS updates applied -conservative BIOS settings -setting hyperthreading/turbo/c-states in BIOS make no difference Log of 2 last events is attached below (for 2.6.33 kernel but 2.6.31 looks very same). Reproducible: Sometimes Steps to Reproduce: 1. let system run 2. wait 3. wait, see above description... -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Show replies by date

bugzilla_noreply＠novell.com

4 Mar 4 Mar

10:42

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c1 --- Comment #1 from Samuel Kvasnica 2010-03-04 10:42:06 UTC --- Workaround: restricting the dom0-cpus count in /etc/xen/xend_config.sxp to 1 seems to help. What's going on here ? Is there really no stable, usable xenified kernel at the moment ? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

8 Mar 8 Mar

19:12

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c Jeff Mahoney changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffm@novell.com AssignedTo|kernel-maintainers@forge.pr |jbeulich@novell.com |ovo.novell.com | -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

9 Mar 9 Mar

09:56

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c2 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |samuel.kvasnica@ims.co.at --- Comment #2 from Jan Beulich 2010-03-09 09:56:55 UTC --- Am I understanding correctly that those events happen in Dom0? How many CPUs does the system have (a full Xen/kernel log would certainly help)? Does the presence of the DomU matter? Does "xm top" indicate which (one/some/all) vCPU(s) are getting under high load? At the point you see the load grow, do other statistics indicate e.g. an interrupt storm? Also, assuming you're able to observe the behavior that will eventually lead to the lockup warnings, could you send 'e', '0', 'd', and 'q' from the serial console to Xen while the system is in this state? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10 Mar 10 Mar

11:46

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c3 Samuel Kvasnica changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|samuel.kvasnica@ims.co.at | --- Comment #3 from Samuel Kvasnica 2010-03-10 11:46:56 UTC --- The system has 4 physical cores (xeon X3450)+ HT enabled => 8 cores. But it makes no difference if we enable/disable HT in Bios. I'm attaching latest kernel boot log for version 2.6.33 below. We observe these lockup messages only in Dom0 syslog. DomU itself is pretty clean. Additionaly, we observe peaks of cpu load / load average when looking at snmp log data - and these correlate with the timestamp of locks in syslog. If DomU is running or not makes difference has to be tested - takes some more time... Unfortunalely, we could never observe such lockup "live". It takes sometimes up to 12 hours until this happens and then typically bunch of such lockups will follow within few minutes. I'm attaching a cacti rrdgraph of load average for illustration. Sometimes, there was drbd sync active, sometimes absolutely no cpu-load or I/O. The system is in principle fully idle - we are not able to find any trigger event for the lockups. But I've got following important observation: since we reduced the dom0-cpus to 1 we have uptime of 1 week without any single lockup message in log. BUT(!): the load average baseline was around 0.0 all the time until 3 days ago when it abruptly changed to 1.0 - i.e. whatever I do (and kill), it won't ever go below 1.0, see top output: top - 14:00:22 up 8 days, 2:12, 2 users, load average: 1.00, 1.00, 1.00 Tasks: 137 total, 4 running, 133 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1960960k total, 548648k used, 1412312k free, 16k buffers Swap: 8385920k total, 65512k used, 8320408k free, 92232k cached I would suspect this is the single-cpu variant of the same bug, just w/o the lockup message in syslog. Could it be there is some cpu-timer-related problem in kernel and all these lockups are just fake messages due to incorrectly collected data ? In the last HEAD-kernel Changelog, I've found namely following message, could it be someway related ? * Tue Mar 2 13:00:00 2010 jbeulich@novell.com - Update Xen patches to 2.6.33 and c/s 1003. - patches.xen/xen-clockevents: replace Xen's custom time handling with such using GENERIC_CLOCKEVENTS infrastructure. - Update Xen config files. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

11:49

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c4 --- Comment #4 from Samuel Kvasnica 2010-03-10 11:49:13 UTC --- Created an attachment (id=347564) --> (http://bugzilla.novell.com/attachment.cgi?id=347564) rrdgraph load average shows lockup events -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

12:02

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c5 --- Comment #5 from Samuel Kvasnica 2010-03-10 12:02:45 UTC --- Created an attachment (id=347571) --> (http://bugzilla.novell.com/attachment.cgi?id=347571) boot log 2.6.33-25-xen -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

12 Mar 12 Mar

22:17

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c6 --- Comment #6 from Samuel Kvasnica 2010-03-12 22:17:23 UTC --- WOW, just gave the latest 2.6.33-29 a try: the load average is immediately after boot to >=4.0 !!! while CPU is totally idle (and hat 35 degC temperature...) This is the top output (note the ridiculous kthreadd & ha_logd time while having 21min uptime): top - 22:41:57 up 21 min, 1 user, load average: 4.01, 3.94, 2.98 Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.5%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.2%st Mem: 4062208k total, 544840k used, 3517368k free, 3824k buffers Swap: 8385920k total, 0k used, 8385920k free, 178148k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4185 root 20 0 35848 908 608 S 0.0 0.0 5124095h ha_logd 2 root 20 0 0 0 0 S 0.0 0.0 19215:21 kthreadd 1625 root 20 0 28840 1820 1320 S 0.7 0.0 0:07.78 lsi_mrdsnmpagen 1577 root 39 19 0 0 0 S 0.3 0.0 0:02.31 kipmi0 27 root 20 0 0 0 0 S 0.0 0.0 0:02.13 events/0 All kernels seem to have some timer problem. Going back to 2.6.31.12-12 and hoping dom0-cpus=1 will work... -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

22:21

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c7 --- Comment #7 from Samuel Kvasnica 2010-03-12 22:21:08 UTC --- Created an attachment (id=348290) --> (http://bugzilla.novell.com/attachment.cgi?id=348290) 2.6.31.12-12-xen boot.msg -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

18 Mar 18 Mar

21:42

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c8 --- Comment #8 from Samuel Kvasnica 2010-03-18 21:42:08 UTC --- Conclusion: 2.6.33.x unusable at the moment, timer goes totally crazy. 2.6.31.12 seems to be kind of "working" for special case of dom0-cpus=1. However, recently while having 4 vcpus within domU(!) suddenly got the following strange picture: top - 21:26:40 up 4 days, 1:28, 4 users, load average: 0.00, 0.00, 0.00 Tasks: 164 total, 1 running, 163 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.3%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4209248k total, 2194780k used, 2014468k free, 744k buffers Swap: 0k total, 0k used, 0k free, 1390992k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2020 mysql 20 0 228m 33m 6900 S 0 0.8 5428430h mysqld 20476 open-xch 20 0 1023m 367m 11m S 0 8.9 5128098h java 986 root 20 0 171m 3300 2112 S 0 0.1 5126016h console-kit-dae 2402 root 20 0 69824 7124 3292 S 0 0.2 4:40.14 snmpd 20402 root 20 0 489m 86m 9980 S 0 2.1 0:03.00 java 25192 wwwrun 20 0 313m 27m 3672 S 0 0.7 0:01.43 httpd2-prefork 24226 wwwrun 20 0 308m 23m 3700 S 0 0.6 0:01.29 httpd2-prefork 20907 wwwrun 20 0 307m 22m 3668 S 0 0.5 0:00.89 httpd2-prefork -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

19 Mar 19 Mar

08:31

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c9 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Status|NEW |NEEDINFO CC| |pmillett@novell.com Info Provider| |samuel.kvasnica@ims.co.at --- Comment #9 from Jan Beulich 2010-03-19 08:31:55 UTC --- This last post of yours indicates nothing more than possibly broken accounting. I'll see whether I can observe something similar, but certainly I haven't seen 2.6.33 misbehaviors like the one you describe on any of my test systems. Hence, without you obtaining detail on what the CPUs actually do when you observe these lockups (and btw. I'm not even clear whether on .33 you are observing lockups at all) there's pretty little we can do I'm afraid (unless this can be reproduced in our lab - Preston?). Also, just to not confuse things - let's please separate the 11.2 and Factory issues: Time handling (as noted by you in #3) is now sufficiently different in those two kernels, and hence unless proven otherwise I have to assume that the similar symptoms (if so in the first place) are caused by different problems. Consequently, please open a new bug for the Factory kernel problems (transferring all relevant information from here), and we'll continue looking at the 2.6.31 issue here. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

28 May 28 May

07:00

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c10 --- Comment #10 from Jan Beulich 2010-05-28 07:00:37 UTC --- A tentative fix for this was just committed, will be available in a future kernel maintenance update - please re-test against that (the fix has been in the HEAD kernel for a couple of days, so re-testing against a new enough Factory kernel should also be possible). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

31 May 31 May

10:46

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c11 --- Comment #11 from Samuel Kvasnica 2010-05-31 10:46:38 UTC --- Jan, sorry for not taking care about the separate factory kernel bug - I'm terribly busy over last few weeks. However, your last comment sounds positive ! What 2.6.31 kernel version should contain that fix ? Or where can I find the relevant patch to try it out ? One more interesting info: I did some testing on another new supermicro system, this time it was a X8DTi-F with XEON E5530 CPU (intel 5520 chipset). In contrast to X8SIL-F with X3450 above, I cannot reproduce the 61s-bug on this system using the same system images. So it seem to be chipset- or mainboard-related. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

12:19

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c12 --- Comment #12 from Jan Beulich 2010-05-31 12:19:05 UTC --- (In reply to comment #11)

...

sorry for not taking care about the separate factory kernel bug - I'm terribly busy over last few weeks. However, your last comment sounds positive ! What 2.6.31 kernel version should contain that fix ? Or where can I find the relevant patch to try it out ?

There is no separate patch, the fix is part of one of the upstream merge patches. The relevant change is that under Xen sched_clock_stable must never be set to a non-zero value.

...

One more interesting info: I did some testing on another new supermicro system, this time it was a X8DTi-F with XEON E5530 CPU (intel 5520 chipset). In contrast to X8SIL-F with X3450 above, I cannot reproduce the 61s-bug on this system using the same system images. So it seem to be chipset- or mainboard-related.

If you look at the condition under which sched_clock_stable was set to 1 previously, you'll realize that only Nehalems (and newer) would be affected. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

12:39

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c13 --- Comment #13 from Samuel Kvasnica 2010-05-31 12:39:26 UTC --- Ok, thanks for the links. But regarding the last part: both X3450 & E5530 are very recent "Nehalems (and newer)". Just the X3450 is the smaller single-cpu LGA-1156 variant. So there must be other condition related to the chipset ? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

13:07

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c14 --- Comment #14 from Jan Beulich 2010-05-31 13:07:50 UTC --- Of course you always may just be lucky and not notice any bad effects despite the problem actually being there. It all depends on bit 8 of CPUID's EDX output for index 0x80000007 (plus being an Intel CPU). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10 Jun 10 Jun

13:00

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c15 --- Comment #15 from Samuel Kvasnica 2010-06-10 13:00:54 UTC --- Ok, it seems like this was added into the "xen3-patch-2.6.30" patch, file arch/x86/kernel/cpu/intel.c and the latest 2.6.31.13-18.1 kernel seems to contain this patch. I'm upgrading now and going to observe what happens. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

21 Jun 21 Jun

14:43

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

http://bugzilla.novell.com/show_bug.cgi?id=584554 http://bugzilla.novell.com/show_bug.cgi?id=584554#c16 Samuel Kvasnica changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |RESOLVED InfoProvider|samuel.kvasnica@ims.co.at | Resolution| |FIXED --- Comment #16 from Samuel Kvasnica 2010-06-21 14:43:30 UTC --- Seems to solve the issue, system is running couple of weeks now, no soft-lockups anymore with dom0-cpus = 2. I've tested only the 2.6.31.13-18-xen kernel. So _not_ setting sched_clock_stable to 1 really helps. What I still do not understand is the relationship to Nehalem chips. Observed so far too many different systems with bit8 of c->x86_power set 1 not having this issue, just the Ibex Peak Xeon34xx makes problems. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

23 Sep 23 Sep

13:10

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c17 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard| |maint:released:11.2:35929 --- Comment #17 from Swamp Workflow Management 2010-09-23 13:10:01 UTC --- Update released for: kernel-debug, kernel-debug-base, kernel-debug-base-debuginfo, kernel-debug-debuginfo, kernel-debug-debugsource, kernel-debug-devel, kernel-debug-devel-debuginfo, kernel-default, kernel-default-base, kernel-default-base-debuginfo, kernel-default-debuginfo, kernel-default-debugsource, kernel-default-devel, kernel-default-devel-debuginfo, kernel-desktop, kernel-desktop-base, kernel-desktop-base-debuginfo, kernel-desktop-debuginfo, kernel-desktop-debugsource, kernel-desktop-devel, kernel-desktop-devel-debuginfo, kernel-pae, kernel-pae-base, kernel-pae-base-debuginfo, kernel-pae-debuginfo, kernel-pae-debugsource, kernel-pae-devel, kernel-pae-devel-debuginfo, kernel-source, kernel-source-vanilla, kernel-syms, kernel-trace, kernel-trace-base, kernel-trace-base-debuginfo, kernel-trace-debuginfo, kernel-trace-debugsource, kernel-trace-devel, kernel-trace-devel-debuginfo, kernel-vanilla, kernel-vanilla-base, kernel-vanilla-base-debuginfo, kernel-vanilla-debuginfo, kernel-vanilla-debugsource, kernel-vanilla-devel, kernel-vanilla-devel-debuginfo, kernel-xen, kernel-xen-base, kernel-xen-base-debuginfo, kernel-xen-debuginfo, kernel-xen-debugsource, kernel-xen-devel, kernel-xen-devel-debuginfo, preload-kmp-default, preload-kmp-desktop Products: openSUSE 11.2 (debug, i586, x86_64) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

18 Oct 18 Oct

21:03

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c18 Samuel Kvasnica changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED | --- Comment #18 from Samuel Kvasnica 2010-10-18 21:03:01 UTC --- I must reopen this report. On both X3450 & E5530 systems I've got again soft lockups. I noticed it due to the ntp daemon in Xen Dom0 going crazy. This happened after an upgrade to 2.6.31.14-11-xen and xen-4.0.1_21326_01-75.2. I'm getting now 2815s lockups on both machines: Oct 15 19:23:13 imap-host kernel: [1056492.395636] BUG: soft lockup - CPU#0 stuck for 2815s! [swapper:0] Oct 15 19:23:13 imap-host kernel: [1056492.395643] Modules linked in: drbd ipmi_devintf usbbk ebt_log gntdev ebt_ip netbk ipmi_si blkb k blkback_pagemap ebtable_filter blktap xenbus_be ebtables ipmi_msghandler x_tables bridge stp llc edd microcode fuse loop dm_mod ide_ pci_generic ide_core ata_generic usbhid i2c_i801 ata_piix hid e1000e pcspkr i2c_core 8250_pnp 8250 sg button serial_core xfs exportfs ehci_hcd xenblk cdrom xennet fan processor pata_acpi thermal thermal_sys hwmon megaraid_sas [last unloaded: drbd] Oct 15 19:23:13 imap-host kernel: [1056492.395706] CPU 0: Oct 15 19:23:13 imap-host kernel: [1056492.395710] Modules linked in: drbd ipmi_devintf usbbk ebt_log gntdev ebt_ip netbk ipmi_si blkb k blkback_pagemap ebtable_filter blktap xenbus_be ebtables ipmi_msghandler x_tables bridge stp llc edd microcode fuse loop dm_mod ide_ pci_generic ide_core ata_generic usbhid i2c_i801 ata_piix hid e1000e pcspkr i2c_core 8250_pnp 8250 sg button serial_core xfs exportfs ehci_hcd xenblk cdrom xennet fan processor pata_acpi thermal thermal_sys hwmon megaraid_sas [last unloaded: drbd] Oct 15 19:23:13 imap-host kernel: [1056492.395772] Pid: 0, comm: swapper Not tainted 2.6.31.14-11-xen #1 X8SIL Oct 15 19:23:13 imap-host kernel: [1056492.395777] RIP: e030:[<ffffffff800083aa>] [<ffffffff800083aa>] 0xffffffff800083aa Oct 15 19:23:13 imap-host kernel: [1056492.395786] RSP: e02b:ffffffff80737e90 EFLAGS: 00000246 Oct 15 19:23:13 imap-host kernel: [1056492.395791] RAX: 0000000000000000 RBX: ffffffff80783388 RCX: ffffffff800083aa Oct 15 19:23:13 imap-host kernel: [1056492.395797] RDX: ffffffff80737e68 RSI: 0000000000000000 RDI: 0000000000000001 Oct 15 19:23:13 imap-host kernel: [1056492.395803] RBP: ffffffff80737eb8 R08: 0000000000000000 R09: 0000000000000000 Oct 15 19:23:13 imap-host kernel: [1056492.395809] R10: ffff8801eeba2000 R11: 0000000000000246 R12: 0000000000000000 Oct 15 19:23:13 imap-host kernel: [1056492.395816] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Oct 15 19:23:13 imap-host kernel: [1056492.395825] FS: 00007f0e73c2e6f0(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000 Oct 15 19:23:13 imap-host kernel: [1056492.395832] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 15 19:23:13 imap-host kernel: [1056492.395837] CR2: 0000000001794e90 CR3: 00000001ea7ff000 CR4: 0000000000002660 Oct 15 19:23:13 imap-host kernel: [1056492.395843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 15 19:23:13 imap-host kernel: [1056492.395850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 15 19:23:13 imap-host kernel: [1056492.395856] Call Trace: Oct 15 19:23:13 imap-host kernel: [1056492.395869] [<ffffffff8000fa08>] xen_safe_halt+0x28/0x50 Oct 15 19:23:13 imap-host kernel: [1056492.395877] [<ffffffff800149a9>] xen_idle+0x59/0x120 Oct 15 19:23:13 imap-host kernel: [1056492.395884] [<ffffffff8000b0fb>] cpu_idle+0xbb/0x100 Oct 15 19:23:13 imap-host kernel: [1056492.395892] [<ffffffff8045b525>] rest_init+0x75/0x90 Oct 15 19:23:13 imap-host kernel: [1056492.395902] [<ffffffff80793c6b>] start_kernel+0x36d/0x38a Oct 15 19:23:13 imap-host kernel: [1056492.395910] [<ffffffff80792d94>] x86_64_start_reservations+0xf4/0x10f Oct 15 19:23:13 imap-host kernel: [1056492.395918] [<ffffffff80792dfa>] x86_64_start_kernel+0x4b/0x66 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

19 Oct 19 Oct

07:36

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c19 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Found By|--- |Community User Resolution| |FIXED --- Comment #19 from Jan Beulich 2010-10-19 07:36:32 UTC --- I am rather convinced that this is now a different problem, and would therefore suggest opening a new report: For one it is very odd that you get a lockup report only after over 45 minutes (which may certainly be a result of the ntp daemon problems you noted but didn't go into detail on). And you're not using "normal" 11.2 anymore. So please do a full new report, including full logs, using the "normal" up-to-date kernel and hypervisor, and ideally on 11.3 rather than 11.2 (as, if the problem can't be reproduced there, we would ask you to update anyway). In any case, downgrading the kernel by one revision to understand whether this is a regression would then be necessary too. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

09:54

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c20 --- Comment #20 from Samuel Kvasnica 2010-10-19 09:54:02 UTC --- Hi Jan, I agree this will be probably a different issue, but before I put more effort into further investigation: the "normal" 11.2 config is unfortunately not usable due to several issues, a full upgrade to 11.3 is too risky on running/tuned system only due to a kernel bug. But - the previous 2.6.31.13 was running w/o lockup problems for several months. The 2.6.31.14 got lockup on 2 systems about 2 weeks after the upgrade. Do you have an idea what was changed with respect to clock/timers between .13 and .14 ? Is there any repository where I can get the last .13 kernel (I already removed it...) ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

10:39

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c21 --- Comment #21 from Jan Beulich 2010-10-19 10:38:59 UTC --- (In reply to comment #20)

...

the "normal" 11.2 config is unfortunately not usable due to several issues, a full upgrade to 11.3 is too risky on running/tuned system only due to a kernel bug.

In order to give support, we generally need a reasonably consistent setup. Any exception to this may (and likely will) result in lowered priority and slower response times.

...

But - the previous 2.6.31.13 was running w/o lockup problems for several months. The 2.6.31.14 got lockup on 2 systems about 2 weeks after the upgrade.

Which still doesn't in any way mean that the old problem is back.

...

Do you have an idea what was changed with respect to clock/timers between .13 and .14 ?

Not offhand, no. And particularly not because you're apparently talking about the differences between kernels that never got released.

...

Is there any repository where I can get the last .13 kernel (I already removed it...) ?

I don't think they're being kept. You may want to try older released update kernels (http://download.opensuse.org/pub/opensuse/update/11.2/rpm/x86_64/; there's a .12 and an earlier .14 there). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

11:00

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c22 --- Comment #22 from Samuel Kvasnica 2010-10-19 11:00:07 UTC --- I fully acknowledge your comments, and you are right. Normally I try to solve issues myself and submit directly the solution if possible - at the moment I'm just trying to get some more initial info/help before I start further investigations, so please do not understand my input as obvious unnecessary questions. The real target behind that is to get a kernel with satisfactory working xen, xfs, nfs, cifs,... and none of the "released" kernels is satisfactory. As experience shown previously, even not when using SLES releases. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

bugzilla_noreply＠novell.com

11:04

New subject: [Bug 584554] random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

https://bugzilla.novell.com/show_bug.cgi?id=584554 https://bugzilla.novell.com/show_bug.cgi?id=584554#c23 --- Comment #23 from Jan Beulich 2010-10-19 11:04:20 UTC --- At least as far as Xen is concerned, I don't think we've seen that many (namely SLE) bug reports from you... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

4945

Age (days ago)

5176

Last active (days ago)

List overview

Download

24 comments

1 participants

participants (1)

bugzilla_noreply＠novell.com

[Bug 584554] New: random 61s soft lockups using xenified kernel 2.6.31.12 or even 2.6.33-25

tags

participants (1)