[Bug 606921] New: last factory update completely broke XEN dom0 of domUs
http://bugzilla.novell.com/show_bug.cgi?id=606921 http://bugzilla.novell.com/show_bug.cgi?id=606921#c0 Summary: last factory update completely broke XEN dom0 of domUs Classification: openSUSE Product: openSUSE 11.3 Version: Factory Platform: x86-64 OS/Version: Other Status: NEW Severity: Critical Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: koenig@linux.de QAContact: qa@suse.de Found By: --- Blocker: --- I just updated my 11.3 dom0 to factory again -- and now no domU starts up anymore :-( versions installed: # rpm -qa \*xen\* | sort iscsitarget-kmp-xen-1.4.19_2.6.34.0_8-2.18.x86_64 kernel-xen-2.6.34-8.1.x86_64 kernel-xen-devel-2.6.34-8.1.x86_64 xen-4.0.0_21091_05-1.3.x86_64 xen-doc-html-4.0.0_21091_05-1.3.x86_64 xen-doc-pdf-4.0.0_21091_05-1.3.x86_64 xen-kmp-default-4.0.0_21091_05_2.6.34.0_8-1.3.x86_64 xen-libs-32bit-3.3.1_18546_20-0.1.1.x86_64 xen-libs-4.0.0_21091_05-1.3.x86_64 xen-tools-4.0.0_21091_05-1.3.x86_64 the startup script hangs and the domU stays in state "p" with no cpu time activity: root 5441 0.3 0.0 100040 13640 ? S 21:24 0:00 /usr/bin/python /usr/sbin/xm create --quiet --defconfig /etc/xen/auto/os2-sles11 # xm lis Name ID Mem VCPUs State Time(s) Domain-0 0 15173 4 r----- 65.3 os2-sles11 10 1024 1 --p--- 0.0 I'll attach xend.log xend-debug.log domain-builder-ng.log ... -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c1
--- Comment #1 from Harald Koenig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c2
--- Comment #2 from Harald Koenig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c3
--- Comment #3 from Harald Koenig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c
James Fehlig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c
James Fehlig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c4
James Fehlig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c5
Harald Koenig
These same domUs started fine before updating your dom0 to Factory? In
yes. all 12 domUs started up fine before that last update.
xend.log I see the vbd uname for os2-sles11 is file:/etc/xen/images/os2-sles11-flat.vmdk. That's not an actual vmdk image correct?
it was converted from a vmdk to a flat file, now it's just a plain disk image, nothing special except for the file name extension;)
I assume it is just a raw file with '.vmdk' extension. For vmdk images you would need to use 'tap:vmdk:/path/to/vmdk-image' - but I'd not recommend use of the vmdk subdriver in production environments yet. IIRC, there are known issues/limitations in the vmdk subdriver upstream.
ah, interesting. I didn't know that tap:vmdk:... feature -- nce to know;) but no, I won't use it in the future anyway... -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c6
Charles Arnold
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c
Ihno Krumreich
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c7
Thomas E. Schreiner
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c8
Thomas E. Schreiner
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c9
--- Comment #9 from Thomas E. Schreiner
Preston, Could you do some testing with Milestone 7 and check for any problems running pre-existing images?
Dear Mr. Arnold, I've tested with Milestone 7 and RC1 and the problem with the DomU's still exist. When I delete the line with the vif= declaration in the configuration file, the DomU starts without errors. Greetings Thomas E. Schreiner -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c10
--- Comment #10 from Harald Koenig
I've tested with Milestone 7 and RC1 and the problem with the DomU's still exist. When I delete the line with the vif= declaration in the configuration file, the DomU starts without errors.
just a short "me too" from my cuurent test of 11.3: with vif=... line being commented out, a test domU starts (with no net, so not useful), but with vif=... I'm still in trouble: # rpm -q kernel-xen xen kernel-xen-2.6.34-9.5.x86_64 xen-4.0.0_21091_05-6.2.x86_64 # xm cre -c os-centos3u6 Using config file "./os-centos3u6". Error: Device 0 (vif) could not be connected. Hotplug scripts not working. xend.log doesn't give any real information (not even an error msg): [2010-06-29 17:05:05 4222] DEBUG (DevController:139) Waiting for devices tap2. [2010-06-29 17:05:05 4222] DEBUG (XendDomainInfo:1922) XendDomainInfo.handleShutdownWatch [2010-06-29 17:05:05 4222] DEBUG (DevController:139) Waiting for devices vif. [2010-06-29 17:05:05 4222] DEBUG (DevController:144) Waiting for 0. [2010-06-29 17:05:05 4222] DEBUG (DevController:653) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status. [ ... waiting for 100 secs ... ] [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:3111) XendDomainInfo.destroy: domid=3 [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:2446) Destroying device model [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:2453) Releasing devices [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:2459) Removing vif/0 [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:1289) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2010-06-29 17:06:45 4222] DEBUG (XendDomainInfo:2459) Removing console/0 .. BUT there is a kernel oops for that time stamp in udevd / netback_uevent -- from /var/log/messages, hopefully this can help to fiure out what's going wrong ?!? Jun 29 17:05:03 os4 kernel: [ 1204.644580] kjournald starting. Commit interval 15 seconds Jun 29 17:05:03 os4 kernel: [ 1204.644594] EXT3-fs (dm-0): mounted filesystem with ordered data mode Jun 29 17:05:05 os4 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/768 Jun 29 17:05:05 os4 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/832 Jun 29 17:05:05 os4 kernel: [ 1205.844787] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 Jun 29 17:05:05 os4 kernel: [ 1205.845157] IP: [<ffffffffa079a9ce>] netback_uevent+0x2e/0x100 [netbk] Jun 29 17:05:05 os4 kernel: [ 1205.845385] PGD 3e896d067 PUD 3ea605067 PMD 0 Jun 29 17:05:05 os4 kernel: [ 1205.845741] Oops: 0000 [#1] SMP Jun 29 17:05:05 os4 kernel: [ 1205.846023] last sysfs file: /sys/devices/xen-backend/vif-3-0/uevent Jun 29 17:05:05 os4 kernel: [ 1205.846180] CPU 1 Jun 29 17:05:05 os4 kernel: [ 1205.846262] Modules linked in: ib_iser rdma_cm ib_cm iw_cm ib_sa ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iscsi_trgt crc32c nfs fscache ip6table_filter ip6_tables iptable_filter ip_tables x_tables usbbk gntdev netbk blkbk blkback_pagemap blktap domctl xenbus_be evtchn autofs4 nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs binfmt_misc bridge stp llc joydev usbhid fuse hid loop dm_mod iTCO_wdt 8250_pnp ib_mthca iTCO_vendor_support i5k_amb 8250 floppy serial_core ib_mad ioatdma i5000_edac sg sr_mod shpchp ib_core i2c_i801 intel_rng container button dca edac_core pci_hotplug pcspkr e1000e uhci_hcd radeon ttm drm_kms_helper ehci_hcd drm agpgart i2c_algo_bit i2c_core sd_mod usbcore xenblk cdrom xennet edd fan ata_generic ata_piix ahci libata scsi_mod thermal processor thermal_sys hwmon [last unloaded: scsi_transport_iscsi] Jun 29 17:05:05 os4 kernel: [ 1205.848591] Jun 29 17:05:05 os4 kernel: [ 1205.848591] Pid: 499, comm: udevd Not tainted 2.6.34-9-xen #1 X7DBR-3/X7DBR-3 Jun 29 17:05:05 os4 kernel: [ 1205.854497] RIP: e030:[<ffffffffa079a9ce>] [<ffffffffa079a9ce>] netback_uevent+0x2e/0x100 [netbk] Jun 29 17:05:05 os4 kernel: [ 1205.854497] RSP: e02b:ffff8803e92dbde8 EFLAGS: 00010246 Jun 29 17:05:05 os4 kernel: [ 1205.854914] RAX: 0000000000000000 RBX: ffff8803e1d01400 RCX: ffff8803d929513b Jun 29 17:05:05 os4 kernel: [ 1205.854914] RDX: ffff8803e8451840 RSI: ffff8803d9295000 RDI: ffff8803e1d01448 Jun 29 17:05:05 os4 kernel: [ 1205.854914] RBP: ffff8803d9295000 R08: 0000000000000000 R09: ffff8803d9295000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] R10: dead000000200200 R11: dead000000100100 R12: ffff8803d9295000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] R13: ffff8803d9295000 R14: ffff8803d9295000 R15: 0000000000000000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] FS: 00007fe99a03b7a0(0000) GS:ffff880004026000(0000) knlGS:0000000000000000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 29 17:05:05 os4 kernel: [ 1205.854914] CR2: 0000000000000008 CR3: 00000003ea612000 CR4: 0000000000002660 Jun 29 17:05:05 os4 kernel: [ 1205.854914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 29 17:05:05 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/768/node /dev/loop0 to xenstore. Jun 29 17:05:05 os4 kernel: [ 1205.854914] Process udevd (pid: 499, threadinfo ffff8803e92da000, task ffff8803e8954180) Jun 29 17:05:05 os4 kernel: [ 1205.854914] Stack: Jun 29 17:05:05 os4 kernel: [ 1205.854914] ffff8803d9295000 ffff8803e1d01448 0000000000000000 ffff8803d9295000 Jun 29 17:05:05 os4 kernel: [ 1205.854914] <0> ffff8803d9295000 ffffffff802c3301 0000000000100073 ffff8803ec199f40 Jun 29 17:05:05 os4 kernel: [ 1205.854914] <0> ffff8803ec199f40 ffff8803e1d01458 ffff8803a12d7000 ffffffff802c28e3 Jun 29 17:05:05 os4 kernel: [ 1205.854914] Call Trace: Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff802c3301>] dev_uevent+0x81/0x250 Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff802c28e3>] show_uevent+0xf3/0x190 Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff802c1ece>] dev_attr_show+0x1e/0x50 Jun 29 17:05:05 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/768/physical-device 7:0 to xenstore. Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff8017f9a1>] sysfs_read_file+0x111/0x1f0 Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff801134c3>] vfs_read+0xb3/0x190 Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff801135ee>] sys_read+0x4e/0x90 Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<ffffffff80007438>] system_call_fastpath+0x16/0x1b Jun 29 17:05:05 os4 kernel: [ 1205.854914] [<00007fe9999a83e0>] 0x7fe9999a83e0 Jun 29 17:05:05 os4 kernel: [ 1205.854914] Code: 28 48 89 5c 24 08 48 89 fb 48 8d 7f 48 48 89 6c 24 10 4c 89 64 24 18 48 89 f5 4c 89 6c 24 20 e8 e9 b0 b2 df f6 05 b5 93 0d e0 02 <4c> 8b 60 08 75 75 48 8b 73 08 31 c9 31 ff 48 c7 c2 6f df 79 a0 Jun 29 17:05:05 os4 kernel: [ 1205.854914] RIP [<ffffffffa079a9ce>] netback_uevent+0x2e/0x100 [netbk] Jun 29 17:05:05 os4 kernel: [ 1205.854914] RSP <ffff8803e92dbde8> Jun 29 17:05:05 os4 kernel: [ 1205.854914] CR2: 0000000000000008 Jun 29 17:05:05 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/768/hotplug-status connected to xenstore. Jun 29 17:05:05 os4 kernel: [ 1205.865505] ---[ end trace a50cffe5bd685bfd ]--- Jun 29 17:05:05 os4 kernel: [ 1206.384407] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/3/768 Jun 29 17:05:05 os4 kernel: [ 1206.384719] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=95) type:0 Jun 29 17:05:06 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/832/node /dev/loop1 to xenstore. Jun 29 17:05:06 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/832/physical-device 7:1 to xenstore. Jun 29 17:05:06 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/3/832/hotplug-status connected to xenstore. Jun 29 17:05:06 os4 kernel: [ 1206.990894] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/3/832 Jun 29 17:05:06 os4 kernel: [ 1206.991217] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=95) type:0 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c11
James Fehlig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c
Jan Beulich
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c12
Jan Beulich
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c13
--- Comment #13 from Harald Koenig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c14
--- Comment #14 from Jan Beulich
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c15
--- Comment #15 from Harald Koenig
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c16
Jan Engelhardt
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c17
--- Comment #17 from Harald Koenig
Still experiencing problems with 2.6.34-12-xen. In particular, xend-debug.log still shows the crash/backtrace as before.
see bug #618678: http://bugzilla.novell.com/show_bug.cgi?id=618678#c29 that newer "kernel of the day" includes a patch which fixes all my problems. from that kernel-xen.rpm changelog: * Tue Jul 06 2010 jbeulich@novell.com - patches.xen/xen3-patch-2.6.22: Fix a uevent deadlock in netback (secondary issue from bnc#618678). * Tue Jul 06 2010 gregkh@suse.de - patches.xen/xen3-fixup-xen: Refresh for 2.6.34.1 update (parts of it were already in the .1 patch.) unfortunately there is still not kernel update for 11.3 so 11.3 xen support remains broken for now:-(( -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c18
--- Comment #18 from Jan Engelhardt
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c19
--- Comment #19 from Jan Engelhardt
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c20
--- Comment #20 from Charles Arnold
Newly created domus seem to work fine though. One thing I noticed is that on 11.3, /etc/xen/vm/mine contains, for example:
vif=[ 'mac=00:16:3e:59:89:12,bridge=br0', ]
whereas on 11.2, if I remember correctly, it was just:
vif=[ 'bridge=br0', ]
The missing MAC address was a bug in the 11.2 vm-install. See bug 565037. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=606921
http://bugzilla.novell.com/show_bug.cgi?id=606921#c21
Jan Beulich
participants (1)
-
bugzilla_noreply@novell.com