http://bugzilla.novell.com/show_bug.cgi?id=553690 User jbeulich@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c1 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Status|NEW |NEEDINFO Found By|--- |Community User Info Provider| |bderzhavets@yahoo.com AssignedTo|jdouglas@novell.com |jbeulich@novell.com --- Comment #1 from Jan Beulich 2009-11-09 01:09:51 MST --- We'll need the full kernel/hypervisor log (or at the very least the full kernel register/stack dump, including the last few lines that precede it). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User bderzhavets@yahoo.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c3 --- Comment #3 from Boris Derzhavets 2009-11-09 11:36:07 MST --- Created an attachment (id=326328) --> (http://bugzilla.novell.com/attachment.cgi?id=326328) Runtime snapshot F12 PV DomU -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User jbeulich@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c5 --- Comment #5 from Jan Beulich 2009-11-10 04:03:23 MST --- (In reply to comment #2)

End of output of kernel console in case of crash :-

do_page_fault page_fault memcpy_c swiotlb_balance

swiotlb_bounce ???

unmap_single swiotlb_umap_sg_attrs _ata_sg_clean __ata_qc_comlete _ata_qc_comlete _ata_qc_comlete_multiple ahci_interrupt handle_IRQ_event handle_level_irq evtchn_do_upcall do_hypercall_callback 0xfffff...f802063ac xen_safe_halt xen_idle cpu_idle rest_init start_kernel x86_64_start_reservations x86_64_start_kernel

Just the function names don't tell much, unfortunately. However, it seems inconsistent that you have unmap_single() and memcpy_c() on the stack: swiotlb_bounce() calls memcpy() only for DMA_TO_DEVICE, but do_unmap_single() passes DMA_FROM_DEVICE. This may indicate there's earlier corruption, and hence we'll get nowhere without seeing the full hypervisor and kernel logs, i.e. we need to wait for you to set up serial. Btw., does this also occur for file:/ backed guest disks? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User bderzhavets@yahoo.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c7 --- Comment #7 from Boris Derzhavets 2009-11-10 08:06:41 MST --- Created an attachment (id=326606) --> (http://bugzilla.novell.com/attachment.cgi?id=326606) Another serial on different drive for kernel crash One more serial log for kernel crash on /dev/sda9 ( first one was for /dev/sdb9). It looks different. Output has no disk errors similar to kernel console output submitted yesterday. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User jbeulich@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c9 --- Comment #9 from Jan Beulich 2009-11-10 08:21:37 MST --- I think these

(XEN) mm.c:4206:d0 Global bit is set to kernel page f6454555a9 (XEN) mm.c:4206:d0 Global bit is set to kernel page 4736f480a0

are the indicators of the beginning problems (in the log this is followed by severe problems in the SATA driver, likely because of interrupts no longer arriving). The frame numbers, however, are completely bogus, and I'm unaware of any code path in our kernel that could lead to the global bit to be set on a kernel page. Unless you can assist with debugging this, I don't think we can do much here without reproducing this internally.

Linux version 2.6.31.5-0.1-desktop (geeko@buildhost) (gcc version 4.4.1 [gcc-4_4-branch revision 150839] (SUSE Linux) ) #3 SMP Sat Nov 7 13:41:03 EST 2009

But - what kernel was this created with? Our Xen kernels should call themselves -xen, not -desktop. Did you build this yourself? We need you to use the provided kernel in order to be useful for analysis. And if rebuilding the kernel is indeed unavoidable, it'd be nice for the tag to identify it clearly is such (we do have a -desktop kernel flavor). Finally, for eventual future logs, I'd like to ask that to avoid (if possible) making the logs as redundant as this one (most messages are there several times, which likely is a result of your use of the various command line options). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User bderzhavets@yahoo.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c11 --- Comment #11 from Boris Derzhavets 2009-11-10 08:44:33 MST --- (In reply to comment #10)

Okay, the second log indeed comes closer to your previous description, and we see

...

[ 1380.923876] Thread overran stack, or stack corrupted

Without knowing what kernel this is there's nothing we can do here.

View bug:- https://bugzilla.novell.com/show_bug.cgi?id=552492 ------- Comment #20 From Jan Beulich 2009-11-06 02:23:13 MST (-) ------- (In reply to comment #18)

Sorry, my experience with Suse is limited. I would be glad to test patch with step by step instruction. I need xen-kernel source installed on the machine , but don't know where to get kernel-source-???.x86_64.rpm ( i suspect kernel-xen-source ...)

Patch for X-server suggested by you ************************************* If you don't need the exact RC2 kernel, you could try ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.2/noarch/ ***************** Go to this link ***************** Index of ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.2/noarch/ Up to higher level directory Name Size Last Modified File:kernel-source-2.6.31.5-0.1.99.23.2e8a968.noarch.rpm 68957 KB 11/09/2009 04:02:00 PM File:kernel-source-vanilla-2.6.31.5-0.1.99.23.2e8a968.noarch.rpm 67280 KB 11/09/2009 04:02:00 PM kernel-source-vanilla.rpm 11/09/2009 11:04:00 PM kernel-source.rpm 11/09/2009 11:04:00 PM I downloaded and installed kernel-source.rpm ( suggested by you ) total 12 lrwxrwxrwx 1 root root 32 Nov 7 04:59 linux -> linux-2.6.31.5-0.1.99.21.446052c drwxr-xr-x 25 root root 4096 Nov 8 08:10 linux-2.6.31.5-0.1.99.21.446052c drwxr-xr-x 8 root root 4096 Oct 27 20:11 packages -rw-r--r-- 1 root root 2140 Nov 7 05:01 v32.patch1 dhcppc2:~ # cd /usr/src/linux Applied your patch and built xenified kernel. dhcppc2:/usr/src/linux # make menuconfig ( tuned as xenified as usual for rebased ones) # make -j4 # make modules_install install It appears to be named 2.6.31.5-01-desktop works under Xen and brings up X-Server wit no memory limit. I don't think it's important what name it has. Xen patches and v32,patch1 are coming obviously from you. From my side only "make menuconfig" to activate Xen Dom0 kernel feature. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User bderzhavets@yahoo.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c13 --- Comment #13 from Boris Derzhavets 2009-11-10 09:15:00 MST --- Kernel was just rebuilt latter reporting first issue with drm disabled. I believe it makes sense to wait for final release with 2.6.31.5-xen. If problem will persist i will report with serial log from the very beginning -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User jbeulich@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c15 --- Comment #15 from Jan Beulich 2009-11-11 00:59:55 MST --- (In reply to comment #14)

How could i use your's .config which didn't exist under /usr/src/linux ?

By either installing the full set of kernel-* packages (it moved a number of time, so I'm not sure it's kernel-devel, but I'd guess it is), or more trivially by reading /proc/config.gz while our Xen kernel is running. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User jbeulich@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c17 --- Comment #17 from Jan Beulich 2009-11-11 03:51:16 MST --- Yes. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 User bderzhavets@yahoo.com added comment http://bugzilla.novell.com/show_bug.cgi?id=553690#c19 --- Comment #19 from Boris Derzhavets 2009-11-14 01:26:04 MST --- I mean mem=4G was applied to newly installed Suse 11.2 final to able brought it up with X-Server and picked up your's .config for xenified kernel. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 --- Comment #21 from Boris Derzhavets 2009-11-16 11:22:45 UTC --- 1. Please confirm that you want me to submit serial log of crash kernel linux-2.6.31.5-0.1.99.21.446052c been built via your's .config for xenified kernel. 2. If you point me to any other kernel-source-xen.rpm i can build another kernel (probably more recent) with your's .config for xenified kernel previously applied patch for X-Server and obtain serial log the kernel.

together with a pointer where the kernel binaries (in particular, vmlinux and > any modules involved in the backtrace) used live,

3.Does it mean, that you want to run ? # gdb vmlinux # dissamble particular_module_from_trace Search for certain [] mentioned is stack trace of serial log -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690#c23 --- Comment #23 from Boris Derzhavets 2009-11-18 14:40:34 UTC --- (In reply to comment #22)

(In reply to comment #21)

...

1. Please confirm that you want me to submit serial log of crash kernel linux-2.6.31.5-0.1.99.21.446052c been built via your's .config for xenified kernel.

Of course I'd prefer you to use a pre-built kernel (in which case I could just retrieve the binaries I need for analysis myself), but short of that I'm indeed asking for some other consistent pair of (log,kernel). Btw., to be maximally useful here (given that we're at least suspecting stack overrun/corruption), it would be a good idea for you to include "kstack=1024" on the Dom0 (kernel) command line. Serial log of F12 domU crash submitted as requested , kstack=1024 included in xen kernel command line. View attachment.

-- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690#c25 --- Comment #25 from Jan Beulich 2009-11-18 16:10:51 UTC --- So where can I pick up the corresponding binary? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690#c27 --- Comment #27 from Jan Beulich 2009-11-18 16:54:15 UTC --- The try it via mail attachment (just vmlinux, perhaps compressed). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690#c29 --- Comment #29 from Boris Derzhavets 2009-11-18 19:03:58 UTC --- (In reply to comment #27)

The try it via mail attachment (just vmlinux, perhaps compressed). Done. http://free.mailbigfile.com/0982c32dc12fc361ae43d945fc43bdab/listFiles.php

-- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c31 --- Comment #31 from Jan Beulich 2009-11-19 15:33:19 UTC --- This log together with one of the earlier provided ones makes it clear that swiotlb code is being instructed to write over a page table, due to running off the end of a valid buffer. It is not clear however whether the buffer was originally specified improperly, or whether stored data got corrupted e.g. during I/O. Since I can't reproduce the issue myself, I'm hoping that you would be able to rebuild your kernel with the debugging patch just attached, and then try and see whether it captures the problem any earlier (and of course doesn't have any adverse side effects). Do you, btw, also run into this issue when using mem=4G on the Xen command line? Also I assume you're not having the machine do any other things while starting the guest? And from the last log I'm having the impression that only the third guest that got started actually crashed the machine - were the first two of different type, or does the problem not always occur? It would also be nice if you attached "lspci -nn" output for the machine, unless you know the problem is present on two sufficiently different ones. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c33 --- Comment #33 from Boris Derzhavets 2009-11-19 18:06:21 UTC --- Created an attachment (id=328502) --> (http://bugzilla.novell.com/attachment.cgi?id=328502) Serial log after applying the most recent patch DomU's install crashes in the very beginning attempting either detect or partition image device -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c35 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #328458|0 |1 is obsolete| | --- Comment #35 from Jan Beulich 2009-11-20 09:46:40 UTC --- Created an attachment (id=328617) --> (http://bugzilla.novell.com/attachment.cgi?id=328617) debugging patch (kernel, v2) Sorry, oversight on my part. Should be better now. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c37 --- Comment #37 from Jan Beulich 2009-11-20 12:40:41 UTC --- Yes. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c39 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #328617|0 |1 is obsolete| | --- Comment #39 from Jan Beulich 2009-11-23 07:52:07 UTC --- Created an attachment (id=328910) --> (http://bugzilla.novell.com/attachment.cgi?id=328910) debugging patch (kernel, v3) As I understand it, v2 still brought the machine down too early. I hope that v3 finally gets us forward. I'm sorry for not having spotted this earlier. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c41 --- Comment #41 from Boris Derzhavets 2009-11-23 19:19:02 UTC ---

As I understand it, v2 still brought the machine down too early. I hope that v3 finally gets us forward. I'm sorry for not having spotted this earlier. Kernel patched with V3 crashes at same point as with V2. Attempt to format partitions on image device. To get serial log i need to move the box again.

-- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c43 --- Comment #43 from Jan Beulich 2009-12-07 09:06:34 UTC --- Following the observation in https://bugzilla.novell.com/show_bug.cgi?id=551695#c10, did you ever try running a VM on file:/ rather than phy:/ (see also my similar question in #5)? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c45 --- Comment #45 from Jan Beulich 2009-12-14 15:44:24 UTC --- Please see bug 559047 for an updated version of the debugging patch (and a potential fix). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c47 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gleixner@bib-bvb.de --- Comment #47 from Jan Beulich 2009-12-15 14:55:15 UTC --- *** Bug 551695 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=551695 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c49 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rob@sailors.ch --- Comment #49 from Jan Beulich 2010-01-04 15:53:57 UTC --- *** Bug 567306 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=567306 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=553690 http://bugzilla.novell.com/show_bug.cgi?id=553690#c51 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |RESOLVED Info Provider|bderzhavets@yahoo.com | Resolution| |FIXED Target Milestone|--- |unspecified --- Comment #51 from Jan Beulich 2010-01-05 12:11:34 UTC --- Thanks. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.