http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c Jason Douglas changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cyliu@novell.com, | |jbeulich@novell.com, | |jdouglas@novell.com, | |jsong@novell.com AssignedTo|jdouglas@novell.com |jfehlig@novell.com -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c2 Harald Koenig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|koenig@linux.de | --- Comment #2 from Harald Koenig 2010-07-01 16:27:50 UTC --- (In reply to comment #1)

Getting xenstore's view (utility xenstore-ls) on both the corresponding frontends' and backends' states would possibly shed some light on this - the

I have one full xenstore dump which I took after all domUs have been shut down. only dom0 was still running, but xenstore showed all the machines which have been running before. one note about xenstore: I'm not using xenstore myself (no "xm new foo"), all domUs are started via config file either in /etc/init.d/xendomains or with "xm create foo", so I'm a bit surprised to find all the domU info in xenstore *after* all those domUs are shut down ?! I'll attach that dump..

driver terminates those threads when the devices get disconnected (hinting at some failure path [in tools or kernel] doing insufficient cleanup).

The kernel side code didn't change in quite a while - did you observe similar problems with earlier versions?

I'm afraid it'll be difficult for us to do anything if the issue doesn't re-occur for you.

it does reoccur 100% for me, right now xen 4.0 with 11.3 is more or less unusable: whenever the first domU gets shut down or crases it's not possible anymore to start any new domU -- so far I only can reboot the dom0 :-( now I did a new test: no iscsi but one single PVM domU being started with one disk image as local file. start and shutdown workds fine, and the "blkback" kernel thread vanished. but still it's not possible to start a new domU! with identical config file I get # time xm cre -c os-centos5 Using config file "./os-centos5". # Error: Device 0 (vif) could not be connected. Hotplug scripts not working. so I again commented out the "vif=..." def only to get this (both times after 100 secs timeout) # xm cre -c os-centos5 Using config file "./os-centos5". # Error: Device 768 (vbd) could not be connected. Hotplug scripts not working. so I have to reboot again... -- next/last "single domU" test will use a 64bit suse 11.1 domU instead of cenots, just in case you want to argue... -- so stay tuned;-) now I'm no longer sure if the remaining blkback threads are the real problem, or just one more symtom of a even bigger problem:-( -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c4 --- Comment #4 from Harald Koenig 2010-07-01 18:46:50 UTC --- (In reply to comment #2)

so I have to reboot again... -- next/last "single domU" test will use a 64bit suse 11.1 domU instead of cenots, just in case you want to argue... -- so stay tuned;-)

uh, oh! I've now tried to start/stop/start/stop other VMs and surprise: the problem does not get triggered by suse 11.1, sles 11 and centos 5 (both 32 and 64 bit PVMs). some of them have been changed to use disk images as local files, others still use iscsi disks (as I was lacy to change all configs;), so at least it's not an iscsi issue;) right now it looks like the problem is related to using vmlinuz-2.6.9-55.0.12.ELxenU or vmlinuz-2.6.9-89.0.23.ELxenU kernels form centos 4u5 and 4u8, again both 32 and 64 bit domU seem to trigger the problem! next update -- even more obfuscation:-( I replaced that vmlinuz-2.6.9* kernel in a centos4u4 domU by a 2.6.18-8.el5xen kernel (32bit for 1st test) from centos 5u0 (which I use successfully for a centos 5u0 domU), the initrd was directly copied from that working 5u0 domU. using that 5u0 kernel/initrd the 4u4 did not come up correctly, it crashed before it could access the rootfs (because the real 5u0 uses a plain rootfs partition while the 4u4 has rootfs in LVM -- console log of the crash below). BUT even if it only used the same kernel/initrd, this domU start again triggers that problem which blocks any further domU start:-( there is no additional output from "xm dmesg" when starting/stopping such a problematic domU, sono additional clue from that channel:-( right no I do not understand - why centos 5u0 works fine while booting 5u0 kernel with 4u4 kernel triggers further problems after crashing at end of initrd - how to get the centos4 machines working in 11.3/xen4 :-( just to state again: these PVMs run just fine with 11.1 (xen 3.3) and 11.2 (xen 3.4). are there known issues in xen 4.0 with older xen kernel versions in domU ? or any specific issues with centos 4u5/4u8 in xen4 ? this is the diff between the config files of centos 4u4 and 5u0: ------------------------------------------------------------------------------- < name="os-centos4u4" < bootargs="--entry=hda1:/boot/vmlinuz-2.6.18-8.el5xen,/boot/initrd-2.6.18-8.el5xen.img.c5" < disk=[ diskroot + 'os-centos4u4-flat.vmdk,hda,w' , diskroot + 'os-centos4u4-builddisk-flat.vmdk,hdb,w' ] < extra="root=/dev/mapper/VolGroup00-LogVol01" < vif=[ 'mac=00:0c:29:18:4b:5a,bridge=br0,model=rtl8139', ] -------------------------------------------------------------------------------

name="os-centos5" bootargs="--entry=hda1:/boot/vmlinuz-2.6.18-8.el5xen,/boot/initrd-2.6.18-8.el5xen.img" disk=[ diskroot + 'os-centos5-flat.vmdk,hda,w' ] extra="root=/dev/hda1" vif=[ 'mac=00:0c:29:4a:bf:fc,bridge=br0,model=rtl8139', ]

so there is no significant difference... "expected" (now that I see it;) panic when booting 4u4 disk with 5u0 kernel/initrd: Started domain os-centos4u4 (id=1) Linux version 2.6.18-8.el5xen (mockbuild@builder4.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Thu Mar 15 21:02:53 EDT 2007 BIOS-provided physical RAM map: Xen: 0000000000000000 - 000000007d800000 (usable) 1280MB HIGHMEM available. 727MB LOWMEM available. NX (Execute Disable) protection: active ACPI in unprivileged domain disabled Built 1 zonelists. Total pages: 514048 Kernel command line: root=/dev/mapper/VolGroup00-LogVol01 [...] Creating root device. Mounting root filesystem. mount: could not find filesystem '/dev/root' Setting up other filesystems. Setting up new root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Kernel panic - not syncing: Attempted to kill init! Error: Domain 'os-centos4u4' does not exist. Error: Domain 'os-centos4u4' does not exist. and here is the full xen config file for the centos4u4 vm -- still hoping that there will be some bells ringing when you see some "unimportant" bits of information ;-) --- 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< --- name="os-centos4u4" memory=1000 maxmem=2000 vcpus=4 on_poweroff="destroy" on_reboot="restart" on_crash="restart" localtime=0 extra="root=/dev/mapper/VolGroup00-LogVol01" bootloader="/usr/lib/xen/boot/domUloader.py" builder="linux" #bootargs="--entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img" #bootargs="--entry=hda1:/boot/vmlinuz-2.6.9-89.0.23.ELxenU,/boot/initrd-2.6.9-89.0.23.ELxenU.img" #bootargs="--entry=hda1:/boot/vmlinuz-2.6.9-89.0.15.plus.c4xenU,/boot/initrd-2.6.9-89.0.15.plus.c4xenU.img" bootargs="--entry=hda1:/boot/vmlinuz-2.6.18-8.el5xen,/boot/initrd-2.6.18-8.el5xen.img.c5" diskroot='file:/etc/xen/images/' #iskroot='iscsi:iqn.2010-04.de.science-computing:' disk=[ diskroot + 'os-centos4u4-flat.vmdk,hda,w' , diskroot + 'os-centos4u4-builddisk-flat.vmdk,hdb,w' ] vif=[ 'mac=00:0c:29:18:4b:5a,bridge=br0,model=rtl8139', ] nographic=1 apic=1 acpi=1 pae=1 # serial="pty" --- 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< --- so, enough for today, I'm running out of ideas for now:-( -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c6 --- Comment #6 from James Fehlig 2010-07-01 19:47:58 UTC --- (In reply to comment #4)

name="os-centos4u4"

memory=1000 maxmem=2000

Any luck if you set memory = maxmem? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c8 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |coolo@novell.com, | |ksrinivasan@novell.com --- Comment #8 from James Fehlig 2010-07-02 00:52:27 UTC --- I've been doing some testing with RC2 and haven't had any problems with doing start/stop sequences. However, when I do a save operation the vif device is not removed. /sys/devices/xen-backend/vif-X-Y still exists and there was no udev event was triggered for the tools. Ky, this seems like the tiresome problem we've encountered in the past. netbk does not cleanup the device properly on domU shutdown and won't respond to xenstore events for new devices. xen75 is currently in this state. This bug is quite serious IMO and it would be nice to have a resolution for 11.3 GM. I might find time later tonight to build netbk with debug messages enabled and see if it reports anything interesting. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

From the xenstore dump I conclude that when you took it all backends were in 'closing' state. Unfortunately you didn't provide matched information about left over blkback threads: The driver switches a device to 'closing' only after having initiated termination of the respective thread, so (minus eventual bugs)

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c9 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |koenig@linux.de --- Comment #9 from Jan Beulich 2010-07-02 08:34:59 UTC --- the threads should be gone for all backend devices that are in this state. If that isn't the case, try turning on some debug messages (module parameter "debug_lvl") and/or printing of statistics (module parameter "log_stats"). Both parameters can be turned on via sysfs after the module was already loaded. If that doesn't get us further, we may need to sprinkle further debug messages around the blkback sources - would you be able to rebuild an eventual test kernel for yourself? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c11 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |koenig@linux.de --- Comment #11 from James Fehlig 2010-07-02 17:20:58 UTC --- (In reply to comment #10)

can you reproduce that either (1) or (2) will lock you dom0 too ?

In each of these cases do you have a vif device that has not been released by netbk? I.e. do you have a /sys/devices/xen-backend/vif-X-Y? I think you are just hitting failure cases (different bugs) that cause netbk to not cleanup properly. Once netbk is in this state, further domUs cannot be started until a dom0 reboot. As mentioned in comment #8, I'm able to get netbk in this state by doing 'xm save' or 'xm destroy' on a domU. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c13 --- Comment #13 from James Fehlig 2010-07-02 19:24:05 UTC --- Same "Unknown" change event when doing xm save Jul 2 13:19:36 xen48 kernel: [ 657.280667] JWF (frontend_changed:220) Unknown. Jul 2 13:19:36 xen48 kernel: [ 657.280806] JWF (netback_uevent:162) .. Jul 2 13:19:38 xen48 kernel: [ 659.424781] JWF (netbk_check_gop:529) Bad status -1 from copy to DOM1. Jul 2 13:19:40 xen48 kernel: [ 661.383212] JWF (netbk_check_gop:529) Bad status -1 from copy to DOM1. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c15 --- Comment #15 from Jan Beulich 2010-07-05 09:56:32 UTC --- The semaphore is required to avoid races between user mode access through sysfs (calling netback_uevent()) and teardown of a backend (see the email thread surrounding http://lists.xensource.com/archives/html/xen-devel/2010-05/msg01570.html). It does look like the locking I added was indeed too aggressive - I just wonder how that got through all sorts of testing during the minimally 5 weeks that this has been in place. Apparently the write lock ought to be acquired only after calling kobject_uevent(). With that adjustment I hope we can get back to the problem originally reported here... -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c23 --- Comment #23 from Harald Koenig 2010-07-06 13:48:12 UTC --- (In reply to comment #18)

Harold, it should fix your latest issues so we can return to the originally reported problem :-).

where/when can I et a rpm for testing ? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c25 --- Comment #25 from Stephan Kulow 2010-07-07 06:28:15 CEST --- I don't want a new kernel at this point, but from what I understand we can easily include it in a kernel maintenance update. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c30 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |meissner@novell.com --- Comment #30 from James Fehlig 2010-07-19 14:19:04 UTC --- (In reply to comment #29)

any ETA for a 11.3 update kernel with this issue fixed, pleeeease.... ? ;-)

Marcus, the xen kernel problem affecting SLE11 SP1 also affects 11.3 (actually the problem was found in 11.3 first). When can users expect an 11.3 kernel update? Thanks! -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c33 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |koenig@linux.de --- Comment #33 from James Fehlig 2010-07-19 21:29:45 UTC --- Harald, Just to clarify, do you have any remaining issues after installing the fixed kernel? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c35 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #35 from James Fehlig 2010-07-20 14:03:04 UTC --- Ok, I'll close this one now. Thanks for the report and testing the fix. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=618678 https://bugzilla.novell.com/show_bug.cgi?id=618678#c37 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard| |maint:running:35398:moderat | |e --- Comment #37 from Swamp Workflow Management 2010-08-24 15:23:17 UTC --- The SWAMPID for this issue is 35398. This issue was rated as moderate. Please submit fixed packages until 2010-09-07. When done, please reassign the bug to security-team@suse.de. Patchinfo will be handled by security team. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=618678 https://bugzilla.novell.com/show_bug.cgi?id=618678#c Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Status Whiteboard|maint:running:35398:moderat |maint:released:11.3:35403 |e maint:released:11.3:35403 | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.