http://bugzilla.novell.com/show_bug.cgi?id=618678 http://bugzilla.novell.com/show_bug.cgi?id=618678#c0 Summary: blkback thread hangs after unsuccessful xen domU start Classification: openSUSE Product: openSUSE 11.3 Version: Factory Platform: x86-64 OS/Version: Other Status: NEW Severity: Major Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: koenig@linux.de QAContact: qa@suse.de Found By: --- Blocker: --- one of my xen domUs did not start up as expected, which uses 2 iscsi disks. now, trying to start again I get # xm cre -c os-centos4u4 Using config file "./os-centos4u4". Error: Device /dev/xvdp (51952, vbd) is already connected. those two disks still show up with "lsscsi" # lsscsi -t [0:0:0:0] disk sata: /dev/sda [1:0:0:0] disk sata: /dev/sdb [6:0:1:0] cd/dvd ata: /dev/sr0 [35:0:0:0] disk iqn.2010-04.de.science-computing:os-centos4u4-builddisk-flat.vmdk,t,0x1 /dev/sdt [36:0:0:0] disk iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk,t,0x1 /dev/sdu and there are two kernel threads for "domU id #13" which does not exist (highest domU id running is 10): root 12345 0.0 0.0 0 0 ? S 13:42 0:00 [blkback.13.hda] root 12346 0.0 0.0 0 0 ? S 13:42 0:00 [blkback.13.hdb] I don't see any mappings with dmsetup or losetup # dmsetup ls No devices found # losetup -a iscsi logout does not do anything, and login throws an "not found" error, but it's shown in the list of available disks: # /sbin/iscsiadm -m node -T iscsi:iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk --logout # /sbin/iscsiadm -m node -T iscsi:iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk --login iscsiadm: no records found! # /sbin/iscsiadm -m node | grep os-centos4u4 192.168.178.4:3260,1 iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk 192.168.178.4:3260,1 iqn.2010-04.de.science-computing:os-centos4u4-builddisk-flat.vmdk # after shuttig down *all* domUs things changed a bit, but still very bad: now "lsscsi" does not show any virtual/iscsi disks anymore, but still *all* blkback threads exist # ps uax | grep blk root 4289 0.0 0.1 117332 7268 ? Ssl 12:00 0:00 blktapctrl root 4818 0.0 0.0 0 0 ? S 12:00 0:00 [blkback.1.hda] root 4819 0.0 0.0 0 0 ? S 12:00 0:00 [blkback.1.hdb] root 5296 0.0 0.0 0 0 ? S 12:00 0:00 [blkback.3.hda] root 5736 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.4.hda] root 5737 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.4.hdb] root 6188 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.5.hda] root 6189 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.5.hdb] root 6666 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.6.hda] root 6667 0.0 0.0 0 0 ? S 12:01 0:00 [blkback.6.hdb] root 7637 0.0 0.0 0 0 ? S 12:02 0:00 [blkback.8.hda] root 7638 0.0 0.0 0 0 ? S 12:02 0:00 [blkback.8.hdb] root 8151 0.0 0.0 0 0 ? S 12:03 0:00 [blkback.9.hda] root 8152 0.0 0.0 0 0 ? S 12:03 0:00 [blkback.9.hdb] root 8690 0.0 0.0 0 0 ? S 12:03 0:00 [blkback.10.hda] root 8691 0.0 0.0 0 0 ? S 12:03 0:00 [blkback.10.hdb] root 12345 0.0 0.0 0 0 ? S 13:42 0:00 [blkback.13.hda] root 12346 0.0 0.0 0 0 ? S 13:42 0:00 [blkback.13.hdb] now I'll reboot, but any suggestion how to correctly clean up such a mess next time, or which other information are important for further debugging ? how can I get rid of those blkback.* threads ? even after stopping iscsi, there are 17 blkback threads, and the iscsi_tcp kernel module has a usage cound of 17, so it's not possible to completely reload/restart without reboot: # ps uax | grep iscsi iscsi_tcp 11666 17 libiscsi_tcp 18437 1 iscsi_tcp libiscsi 50884 2 iscsi_tcp,libiscsi_tcp scsi_transport_iscsi 41815 2 iscsi_tcp,libiscsi scsi_mod 191208 7 iscsi_tcp,libiscsi,scsi_transport_iscsi,sr_mod,sg,sd_mod,libata some rpm versions: # rpm -qa xen kernel-xen \*iscsi\*| sort iscsitarget-1.4.19-2.31.x86_64 iscsitarget-kmp-default-1.4.19_k2.6.34.0_12-2.31.x86_64 iscsitarget-kmp-xen-1.4.19_k2.6.34.0_12-2.31.x86_64 kernel-xen-2.6.34-12.1.x86_64 open-iscsi-2.0.870-31.8.x86_64 xen-4.0.0_21091_05-6.3.x86_64 yast2-iscsi-client-2.19.5-1.4.noarch yast2-iscsi-server-2.19.0-1.5.noarch here are the kernel msgs from that last startup Jun 30 13:41:57 os4 kernel: [ 6131.884391] blkback: ring-ref 8, event-channel 80, protocol 1 (x86_64-abi) Jun 30 13:41:57 os4 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/0/51952 Jun 30 13:41:57 os4 logger: /etc/xen/scripts/block-iscsi: add XENBUS_PATH=backend/vbd/0/51952 Jun 30 13:41:57 os4 kernel: [ 6131.988562] blkback: ring-ref 8, event-channel 80, protocol 1 (x86_64-abi) Jun 30 13:41:57 os4 kernel: [ 6132.377246] scsi34 : iSCSI Initiator over TCP/IP Jun 30 13:41:57 os4 kernel: [ 6132.630238] scsi 34:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4 Jun 30 13:41:57 os4 kernel: [ 6132.630451] sd 34:0:0:0: Attached scsi generic sg3 type 0 Jun 30 13:41:57 os4 kernel: [ 6132.630867] sd 34:0:0:0: [sdt] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB) Jun 30 13:41:57 os4 kernel: [ 6132.631006] sd 34:0:0:0: [sdt] Write Protect is off Jun 30 13:41:57 os4 kernel: [ 6132.631010] sd 34:0:0:0: [sdt] Mode Sense: 77 00 00 08 Jun 30 13:41:57 os4 kernel: [ 6132.631596] sd 34:0:0:0: [sdt] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Jun 30 13:41:57 os4 kernel: [ 6132.632709] sdt: sdt1 sdt2 Jun 30 13:41:57 os4 kernel: [ 6132.642926] sd 34:0:0:0: [sdt] Attached SCSI disk Jun 30 13:41:58 os4 iscsid: connection27:0 is operational now Jun 30 13:42:01 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/0/51952/physical-device 41:30 to xenstore. Jun 30 13:42:01 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/0/51952/hotplug-status connected to xenstore. Jun 30 13:42:01 os4 kernel: [ 6136.654486] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=108) nodename :backend/vbd/0/51952 Jun 30 13:42:01 os4 kernel: [ 6136.654491] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=95) type:0 Jun 30 13:42:01 os4 kernel: [ 6136.669605] blkfront: xvdp: barriers enabled Jun 30 13:42:01 os4 kernel: [ 6136.669934] xvdp: xvdp1 xvdp2 Jun 30 13:42:02 os4 kernel: [ 6137.665224] kjournald starting. Commit interval 15 seconds Jun 30 13:42:02 os4 kernel: [ 6137.665243] EXT3-fs (dm-0): mounted filesystem with ordered data mode Jun 30 13:42:03 os4 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/0/51952 Jun 30 13:42:03 os4 logger: /etc/xen/scripts/block-iscsi: remove XENBUS_PATH=backend/vbd/0/51952 Jun 30 13:42:03 os4 kernel: [ 6138.703708] connection27:0: detected conn error (1020) Jun 30 13:42:04 os4 logger: /etc/xen/scripts/block: Writing backend/vbd/0/51952/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/0/51952/hotplug-status error to xenstore. Jun 30 13:42:04 os4 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected. Jun 30 13:42:04 os4 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/0/51952 Jun 30 13:42:08 os4 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/768 Jun 30 13:42:08 os4 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/832 Jun 30 13:42:08 os4 logger: /etc/xen/scripts/block-iscsi: add XENBUS_PATH=backend/vbd/13/768 Jun 30 13:42:08 os4 logger: /etc/xen/scripts/block-iscsi: add XENBUS_PATH=backend/vbd/13/832 Jun 30 13:42:08 os4 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/13/0 Jun 30 13:42:08 os4 kernel: [ 6143.055984] device vif13.0 entered promiscuous mode Jun 30 13:42:08 os4 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif13.0, bridge br0. Jun 30 13:42:08 os4 kernel: [ 6143.060799] br0: port 11(vif13.0) entering forwarding state Jun 30 13:42:08 os4 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/13/0/hotplug-status connected to xenstore. Jun 30 13:42:08 os4 kernel: [ 6143.576594] scsi35 : iSCSI Initiator over TCP/IP Jun 30 13:42:08 os4 kernel: [ 6143.578691] scsi36 : iSCSI Initiator over TCP/IP Jun 30 13:42:09 os4 kernel: [ 6143.832288] scsi 35:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4 Jun 30 13:42:09 os4 kernel: [ 6143.832500] sd 35:0:0:0: Attached scsi generic sg3 type 0 Jun 30 13:42:09 os4 kernel: [ 6143.833215] scsi 36:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4 Jun 30 13:42:09 os4 kernel: [ 6143.834392] sd 36:0:0:0: Attached scsi generic sg4 type 0 Jun 30 13:42:09 os4 kernel: [ 6143.837179] sd 35:0:0:0: [sdt] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB) Jun 30 13:42:09 os4 kernel: [ 6143.837238] sd 35:0:0:0: [sdt] Write Protect is off Jun 30 13:42:09 os4 kernel: [ 6143.837240] sd 35:0:0:0: [sdt] Mode Sense: 77 00 00 08 Jun 30 13:42:09 os4 kernel: [ 6143.837344] sd 35:0:0:0: [sdt] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Jun 30 13:42:09 os4 kernel: [ 6143.841716] sd 36:0:0:0: [sdu] 23068672 512-byte logical blocks: (11.8 GB/11.0 GiB) Jun 30 13:42:09 os4 kernel: [ 6143.841800] sd 36:0:0:0: [sdu] Write Protect is off Jun 30 13:42:09 os4 kernel: [ 6143.841803] sd 36:0:0:0: [sdu] Mode Sense: 77 00 00 08 Jun 30 13:42:09 os4 kernel: [ 6143.841944] sd 36:0:0:0: [sdu] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Jun 30 13:42:09 os4 kernel: [ 6143.842426] sdu: sdu1 sdu2 Jun 30 13:42:09 os4 kernel: [ 6143.843052] sdt: Jun 30 13:42:09 os4 kernel: [ 6143.843582] sd 36:0:0:0: [sdu] Attached SCSI disk Jun 30 13:42:09 os4 kernel: [ 6143.850831] sdt1 Jun 30 13:42:09 os4 kernel: [ 6143.851797] sd 35:0:0:0: [sdt] Attached SCSI disk Jun 30 13:42:09 os4 iscsid: connection28:0 is operational now Jun 30 13:42:09 os4 iscsid: connection29:0 is operational now Jun 30 13:42:10 os4 avahi-daemon[3835]: Registering new address record for fe80::fcff:ffff:feff:ffff on vif13.0.*. Jun 30 13:42:13 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/13/832/physical-device 41:30 to xenstore. Jun 30 13:42:13 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/13/832/hotplug-status connected to xenstore. Jun 30 13:42:13 os4 kernel: [ 6147.857183] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/13/832 Jun 30 13:42:13 os4 kernel: [ 6147.857188] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=95) type:0 Jun 30 13:42:13 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/13/768/physical-device 41:40 to xenstore. Jun 30 13:42:13 os4 logger: /etc/xen/scripts/block-iscsi: Writing backend/vbd/13/768/hotplug-status connected to xenstore. Jun 30 13:42:13 os4 kernel: [ 6147.868988] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/13/768 Jun 30 13:42:13 os4 kernel: [ 6147.868994] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.34/linux-2.6.34/drivers/xen/blkback/cdrom.c, line=95) type:0 Jun 30 13:42:16 os4 kernel: [ 6151.780136] blkback: ring-ref 8, event-channel 15, protocol 2 (x86_32-abi) Jun 30 13:42:17 os4 kernel: [ 6151.880532] alloc irq_desc for 902 on node 0 Jun 30 13:42:17 os4 kernel: [ 6151.880535] alloc kstat_irqs on node 0 Jun 30 13:42:17 os4 kernel: [ 6151.884791] blkback: ring-ref 9, event-channel 16, protocol 2 (x86_32-abi) Jun 30 13:42:17 os4 kernel: [ 6151.992027] alloc irq_desc for 903 on node 0 Jun 30 13:42:17 os4 kernel: [ 6151.992031] alloc kstat_irqs on node 0 Jun 30 13:42:17 os4 kernel: [ 6152.128027] alloc irq_desc for 904 on node 0 Jun 30 13:42:17 os4 kernel: [ 6152.128030] alloc kstat_irqs on node 0 Jun 30 13:42:19 os4 kernel: [ 6153.980512] vif13.0: no IPv6 routers present thanks for any idea! -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.