http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c1 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Status|NEW |NEEDINFO Found By|--- |Community User InfoProvider| |0.bugs.only.0@gmail.com --- Comment #1 from Jan Beulich 2010-07-12 15:41:06 UTC --- Could we please get a full log obtained with a released kernel (so we can connect the stack dump to actual binary code). Are there any Xen messages during shutdown? I take it that this happens only after having run one or more guests with passed through devices using PCI MSI. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c3 --- Comment #3 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-12 17:28:45 PDT --- with, uname -a Linux server 2.6.34.1-5-xen #1 SMP 2010-07-10 08:18:53 +0200 x86_64 x86_64 x86_64 GNU/Linux installed from, grep base /etc/zypp/repos.d/Kernel113.repo baseurl=http://download.opensuse.org/repositories/Kernel:/openSUSE-11.3/openSUSE_11.... 'xm create' of a DomU with PCI passthrough didn't seem to trigger the problem. but, creating, then destroying TWO DomUs, both with PCI passthrough, does -- at 'shutdown -r now'. at least, it did three times in a row. xm list Name ID Mem VCPUs State Time(s) Domain-0 0 1010 1 r----- 34.4 cd /etc/xen/vm ( starting to monitor/capture serial port here ... output follows ) xm create nas.cfg <-- PCI passthrough of sata_sil24 RAID card xm create fw.cfg <-- PCI passthrough of 2x r8169 Eth NICs xm list Name ID Mem VCPUs State Time(s) Domain-0 0 1010 1 r----- 38.9 nas 1 1024 4 -b---- 7.6 fw 2 512 4 -b---- 4.3 xm destroy 2 xm destroy 1 shutdown -r now then, serial console output: Broadcast message from root@server (pts/0) (Mon Jul 12 17:14:54 2010): The system is going down for reboot NOW! Welcome to minicom 2.3 OPTIONS: I18n Compiled on Oct 24 2009, 02:06:27. Port /dev/ttyUSB0 Press CTRL-A Z for help on special keys (XEN) [2010-07-13 00:12:02] cpupool_add_domain(dom=1,pool=0) n_dom 2 (XEN) [2010-07-13 00:12:33] mm.c:799:d1 Non-privileged (1) attempt to map I/O space 000000f0 (XEN) [2010-07-13 00:12:52] cpupool_add_domain(dom=2,pool=0) n_dom 3 (XEN) [2010-07-13 00:13:13] mm.c:799:d2 Non-privileged (2) attempt to map I/O space 000000f0 (XEN) [2010-07-13 00:14:38] cpupool_rm_domain(dom=2,pool=0) n_dom 2 (XEN) [2010-07-13 00:14:53] cpupool_rm_domain(dom=1,pool=0) n_dom 1 INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal Boot logging started on /dev/char/../ttyS0(/dev/console) at Mon Jul 12 17:14:55 2010 Master Resource Control: previous runlevel: 3, switching to runlevel: 6 Shutting down mdadmd done Shutting down CRON daemon done Shutting down smartd done Shutting down libvirtd done Shutting down Name Service Cache Daemon done Shutting down mail service (Postfix) done Shutting down cupsd done Shutting down automount done Shutting down Xen domains done Shutting down irqbalance done /etc/init.d/kbd stop done Shutting down sensors: done Stopping rquota daemon done Stopping xend (pid 5035 5034) done Shutting down network time protocol daemon (NTPD) done Shutting down auditd done Shutting down acpid done Shutting down kernel based NFS server: nfsd statd mountd done Shutting down D-Bus daemon done ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... Shutting down HAL daemon done Saving random seed done Usage: /etc/init.d/xen.setup {start} Shutting down xinetd: (waiting for all children to terminate) done Shutting down (remotefs) network interfaces: Umount CIFS File Systems done Shutting down NFS client services: idmapd done Shutting down rpcbind done Shutting down syslog services done ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... Shutting down (localfs) network interfaces: brINT name: Realtek RTL8111/8168B PCI Express GigE Controller (rev 02) brINT done eth0 device: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 20) No configuration found for eth0 Nevertheless the interface will be shut down. eth0 done Shutting down service (localfs) network . . . . . . . . . done Running /etc/init.d/halt.local done Unmounting fuse control filesystem done The System Time is in sync with Hardware Clock good Turning off quota done done ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... Turning off swap files Unmounting file systems tmpfs umounted /dev/mapper/VG0-LV01 umounted /dev/mapper/VG0-LV00_HOME umounted /dev/md0 umounted done Shutting down MD Raid done Stopping udevd: done File descriptor 5 (/dev/blog) leaked on vgchange invocation. Parent PID 10734: /bin/sh Can't deactivate volume group "VG0" with 2 open logical volume(s) skipped Sending all processes the TERM signal... done Please stand by while rebooting the system... [ 309.397334] ------------[ cut here ]------------ [ 309.397342] kernel BUG at /usr/src/packages/BUILD/kernel-xen-2.6.34.1/linux-2.6.34/drivers/xen/core/evtchn.c:1764! [ 309.397345] invalid opcode: 0000 [#1] SMP [ 309.397348] last sysfs file: /sys/devices/pci0000:00/0000:00:11.0/host2/target2:0:0/2:0:0:0/scsi_disk/2:0:0:0/manage_start_stop [ 309.397351] CPU 0 [ 309.397352] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables usbbk gntdev netbk blkbk blkback_pagemap blktap domctl evtchn autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device b0 [ 309.397428] [ 309.397430] Pid: 10474, comm: reboot Not tainted 2.6.34.1-5-xen #1 M3A78-CM/System Product Name [ 309.397433] RIP: e030:[<ffffffff802d2dd1>] [<ffffffff802d2dd1>] evtchn_get_xen_pirq+0x61/0x70 [ 309.397442] RSP: e02b:ffff88003acbfd88 EFLAGS: 00010246 [ 309.397444] RAX: 0000000000000000 RBX: 0000000000000036 RCX: 0000000000000000 [ 309.397446] RDX: 0000000000000000 RSI: 0000000000000036 RDI: 0000000000000001 [ 309.397448] RBP: 0000000000000036 R08: 0000000000000000 R09: ffff88003d5f5cc0 [ 309.397450] R10: 00000000ffffffff R11: 0000000000000001 R12: ffff880001c4d000 [ 309.397452] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 309.397458] FS: 00007fc6e64e4700(0000) GS:ffff8800036d1000(0000) knlGS:0000000000000000 [ 309.397460] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 309.397462] CR2: 00007fc6e606a1c0 CR3: 000000003fec5000 CR4: 0000000000000660 [ 309.397465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 309.397467] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 309.397470] Process reboot (pid: 10474, threadinfo ffff88003acbe000, task ffff88003e0ca5c0) [ 309.397472] Stack: [ 309.397473] ffff880001c4d000 ffffffff802474bf 0000000000000000 0000000000000000 [ 309.397476] <0> ffffffff80887ff0 ffff880001c4d000 ffff880001c4d000 ffff88003b3baa40 [ 309.397480] <0> 0000000001234567 ffffffff802480b7 ffff880001c4cce0 ffff880001c4d000 [ 309.397484] Call Trace: [ 309.397495] [<ffffffff802474bf>] msi_unmap_pirq+0x7f/0xc0 [ 309.397500] [<ffffffff802480b7>] pci_msi_shutdown+0x57/0xd0 [ 309.397504] [<ffffffff8023a4f7>] pci_device_shutdown+0x27/0x30 [ 309.397509] [<ffffffff802c1def>] device_shutdown+0x3f/0x120 [ 309.397519] [<ffffffff8005c167>] kernel_restart_prepare+0x27/0x30 [ 309.397524] [<ffffffff8005c179>] kernel_restart+0x9/0x50 [ 309.397552] [<ffffffff8005c36d>] sys_reboot+0x19d/0x1e0 [ 309.397557] [<ffffffff80007438>] system_call_fastpath+0x16/0x1b [ 309.397564] [<00007fc6e606a1d6>] 0x7fc6e606a1d6 [ 309.397566] Code: 83 f8 01 75 26 89 df 31 db e8 7c c8 dc ff 48 85 c0 74 14 48 8b 40 38 48 85 c0 74 0b 8b 18 c1 eb 0c 81 e3 ff ff 00 00 89 d8 5b c3 <0f> 0b 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 89 f5 53 89 fb [ 309.397592] RIP [<ffffffff802d2dd1>] evtchn_get_xen_pirq+0x61/0x70 [ 309.397595] RSP <ffff88003acbfd88> [ 309.397598] ---[ end trace 5d1bc80d0db78440 ]--- Segmentation fault Master Resource Control: runlevel 6 has been reached Failed services in runlevel 6: xen.setup INIT: no more processes left in this runlevel i'll see what other logs tell me, as well. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c5 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |0.bugs.only.0@gmail.com --- Comment #5 from Jan Beulich 2010-07-13 08:22:39 UTC --- (In reply to comment #2)

sure. can you specify what you're looking for? which log? is 'full' output from serial console -- when this happens -- sufficient?

The serial console output is fine as long as you run the kernel with ignore_loglevel. Otherwise /var/log/boot.msg and the relevant /var/log/messages fragment would be needed.

sure. please define which specific kernel you'd like -- as this is being reported for 11.3, which itself is not yet released.

Atm I would refer to the GMC kernel, but the kernel you chose last happens to be fine too. You just have to be aware that when you update your kernel from transient sources, I may not be able to grab the same binary once I get to analyze the data you provided. Additionally to the stuff requested above, please obtain contents of /proc/interrupts right after boot, right after bringing up the guest(s), and between destroying the guests and shutting down. And please, for all larger chunks of data, attach them rather than including them inline, to keep the information manageable. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c7 mail ignored <0.bugs.only.0@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|0.bugs.only.0@gmail.com | --- Comment #7 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-13 07:40:14 PDT --- (In reply to comment #6)

It seems like this is because pciback.ko is not loaded at the time the system goes down. Assuming the driver is present while you run the domains,

it is, xm list Name ID Mem VCPUs State Time(s) Domain-0 0 1010 1 r----- 175.1 fw 1 1024 4 -b---- 13.2 nas 2 1024 4 -b---- 6.9 lsmod | grep -i pciback pciback 55565 0 xenbus_be 3706 5 usbbk,netbk,blkbk,blktap,pciback

are you later removing it manually for some reason?

no, neither manually, nor intentionally. checking after 'destroy', prior to shutdown, xm destroy 1 xm destroy 2 grep -i pciback pciback 55565 0 xenbus_be 3706 5 usbbk,netbk,blkbk,blktap,pciback it's still there.

If not, we'll have to find what it is that triggers its removal. Are you keeping the devices you passed through bound to pciback after the guests terminate?

I'm doing nothing manually to either keep, or dispose of, the bindings.

A xenstore listing of the tree under /local/domain/0/backend/pci/ would also likely be helpful to see (obtained right before shutting down Dom0).

xenstore-list /local/domain/0 vm device control error memory guest hvmpv data cpu description console domid name xm create fw.cfg xm create nas.cfg xenstore-list /local/domain/0 (... as above ...) backend device-model xenstore-list /local/domain/0/backend/pci 1 2 xm destroy 1 xm destroy 2 xenstore-list /local/domain/0/backend/pci xenstore-list: could not list path /local/domain/0/backend/pci xenstore-list /local/domain/0 (... as above ...) device-model I'll get to work on providing the other data requested above. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c9 mail ignored <0.bugs.only.0@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|0.bugs.only.0@gmail.com | --- Comment #9 from mail ignored <0.bugs.only.0@gmail.com> 2010-07-13 09:03:52 PDT --- Created an attachment (id=375631) --> (http://bugzilla.novell.com/attachment.cgi?id=375631) cat /proc/interrupts etc. for 11.3 GMC & KOTD kernels attachment added ignore_level use kernel Linux server 2.6.34.1-5-xen #1 SMP 2010-07-10 08:18:53 +0200 x86_64 x86_64 x86_64 GNU/Linux baseurl=http://download.opensuse.org/repositories/Kernel:/openSUSE-11.3/openSUSE_11.... cat /proc/interrupts after boot after bringing up guests between destroying the guests and shutting down --> OOPS switch to GMC kernel test xm create xm destroy xm shutdown -r --> no OOPS! cat /proc/interrupts after boot after bringing up guests between destroying the guests and shutting down --> OOPS serial console output, GMC kernel, no OOPS! NOTE -- the difference in cat /proc/interrupts @ 'between destroying the guests and shutting down' for the two different kernels. b4 dumping more output -- please advise as to whether worthwhile to gather more info, what info is req'd, and for which kernel(s). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=620224 http://bugzilla.novell.com/show_bug.cgi?id=620224#c10 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- InfoProvider|jbeulich@novell.com |0.bugs.only.0@gmail.com --- Comment #10 from Jan Beulich 2010-08-06 11:44:11 UTC --- What we need is a set of /proc/interrupts snapshots (as described above) accompanied by the matching kernel log when the crash occurred, so we can match up the various bits of data. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.