http://bugzilla.novell.com/show_bug.cgi?id=600150 http://bugzilla.novell.com/show_bug.cgi?id=600150#c0 Summary: domUloader.py leaves device-mapper remainders Classification: openSUSE Product: openSUSE 11.1 Version: Final Platform: Other OS/Version: openSUSE 11.1 Status: NEW Severity: Major Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: koenig@linux.de QAContact: qa@suse.de Found By: --- Blocker: --- starting xen pvm domUs which use LVM internally in their disks sometimes cause probelems: sometimes after starting such a domU with LVM, there are "left over" DMs which can be shown with "dmsetup ls" (these are from a centos 4u4 machine): loop4-part2 (253, 1) VolGroup00-LogVol01 (253, 0) VolGroup00-LogVol00 (253, 2) or xvdp-part2 (253, 3) VolGroup00-LogVol01 (253, 0) VolGroup00-LogVol00 (253, 1) loop4-part2 is mapped if the boot disk image is stored on a local dom0 disk, xvdp-part2 shows up for the domU boot disk being read via iscsi. here is the partition layout for this centos4u4 disk in this example: root@os4 xen # fdisk -l /etc/xen/images/os-centos4u4-flat.vmdk Device Boot Start End Blocks Id System /etc/xen/images/os-centos4u4-flat.vmdk1 * 1 13 104391 83 Linux /etc/xen/images/os-centos4u4-flat.vmdk2 14 1435 11422215 8e Linux LVM even after shutting down this domU, the /dev/xvdp* still exist and are valid or "busy": root@os4 xen # ll /dev/xvdp* brw-rw---- 1 root root 202, 240 Apr 27 16:54 /dev/xvdp brw-rw---- 1 root root 202, 241 Apr 27 16:54 /dev/xvdp1 brw-rw---- 1 root root 202, 242 Apr 27 16:54 /dev/xvdp2 root@os4 xen # dmsetup ls xvdp-part2 (253, 3) VolGroup00-LogVol01 (253, 0) VolGroup00-LogVol00 (253, 1) root@os4 xen # fdisk -l /dev/xvdp Device Boot Start End Blocks Id System /dev/xvdp1 * 1 13 104391 83 Linux /dev/xvdp2 14 1435 11422215 8e Linux LVM and without manual "dmsetup remove xvdp-part2" the next domU start which needs iscsi will fail -- from xend.log in this case: [2010-04-27 16:59:30 4873] DEBUG (XendDomainInfo:91) XendDomainInfo.create(['vm', ['name', 'os-centos4u4'], ['memory', 1024], ['maxmem', 1024], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['bootloader', '/usr/lib/xen/boot/domUloader.py'], ['bootloader_args', '--entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img'], ['image', ['linux', ['args', 'root=/dev/mapper/VolGroup00-LogVol01']]], ['device', ['vbd', ['uname', 'iscsi:iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'iscsi:iqn.2010-04.de.science-computing:os-centos4u4-builddisk-flat.vmdk'], ['dev', 'hdb'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'br0'], ['mac', '00:0c:29:18:4b:5a'], ['model', 'rtl8139']]]]) [2010-04-27 16:59:30 4873] DEBUG (XendDomainInfo:2068) XendDomainInfo.constructDomain [2010-04-27 16:59:30 4873] DEBUG (balloon:151) Balloon: 1061956 KiB free; need 2048; done. [2010-04-27 16:59:30 4873] DEBUG (XendDomain:450) Adding Domain: 19 [2010-04-27 16:59:30 4873] DEBUG (XendDomainInfo:2232) XendDomainInfo.initDomain: 19 256 [2010-04-27 16:59:30 4873] INFO (XendDomainInfo:2637) Mounting iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk on /dev/xvdp. [2010-04-27 16:59:30 4873] DEBUG (DevController:122) DevController: writing {'backend-id': '0', 'virtual-device': '51952', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/0/51952'} to /local/domain/0/device/vbd/51952. [2010-04-27 16:59:30 4873] DEBUG (DevController:124) DevController: writing {'domain': 'Domain-0', 'frontend': '/local/domain/0/device/vbd/51952', 'uuid': '87138f00-1b37-e6f6-98f1-cf144e9a3d9d', 'bootable': '0', 'dev': '/dev/xvdp', 'state': '1', 'params': 'iqn.2010-04.de.science-computing:os-centos4u4-flat.vmdk', 'mode': 'w', 'online': '1', 'frontend-id': '0', 'type': 'iscsi'} to /local/domain/0/backend/vbd/0/51952. [2010-04-27 16:59:31 4873] DEBUG (DevController:171) Waiting for 51952. [2010-04-27 16:59:31 4873] DEBUG (DevController:681) hotplugStatusCallback /local/domain/0/backend/vbd/0/51952/hotplug-status. [2010-04-27 17:01:11 4873] ERROR (XendDomainInfo:3348) Device 51952 (vbd) could not be connected. Hotplug scripts not working. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 3344, in create_vbd dev_control.waitForDevice(devid) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 182, in waitForDevice (devid, self.deviceClass)) VmError: Device 51952 (vbd) could not be connected. Hotplug scripts not working. [2010-04-27 17:01:11 4873] ERROR (XendDomainInfo:470) VM start failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 450, in start XendTask.log_progress(31, 60, self._initDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2234, in _initDomain self._configureBootloader() File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2646, in _configureBootloader vbd_uuid = dom0.create_vbd(vbd, disk) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 3344, in create_vbd dev_control.waitForDevice(devid) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 182, in waitForDevice (devid, self.deviceClass)) VmError: Device 51952 (vbd) could not be connected. Hotplug scripts not working. [2010-04-27 17:01:11 4873] DEBUG (XendDomainInfo:2446) XendDomainInfo.destroy: domid=19 I've tried the following command several times (from local disk image, but it's likely the same from the iscsi device): touch /tmp/K /usr/lib/xen/boot/domUloader.py --verbose --args=root=/dev/mapper/VolGroup00-LogVol01 --output=/tmp/K --entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img /etc/xen/images/os-centos4u4-flat.vmdk and if all goes well the output looks like this root@os4 xen # /usr/lib/xen/boot/domUloader.py --verbose --args=root=/dev/mapper/VolGroup00-LogVol01 --output=/tmp/K --entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img /etc/xen/images/os-centos4u4-flat.vmdk domUloader: ['/usr/lib/xen/boot/domUloader.py', '--verbose', '--args=root=/dev/mapper/VolGroup00-LogVol01', '--output=/tmp/K', '--entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img', '/etc/xen/images/os-centos4u4-flat.vmdk'] domUloader: vdisk not specified; guessing 'hda' based on 'hda1' domUloader: losetup /dev/loop6 '/etc/xen/images/os-centos4u4-flat.vmdk' domUloader: kpartx -l: loop6-part1 : 0 208782 /dev/loop6 63 domUloader: Found partition: vdev hda1, pdev /dev/mapper/loop6-part1 domUloader: kpartx -l: loop6-part2 : 0 22844430 /dev/loop6 208845 domUloader: Found partition: vdev hda2, pdev /dev/mapper/loop6-part2 domUloader: copyKernelAndRamdisk([hda,/etc/xen/images/os-centos4u4-flat.vmdk,/dev/loop6,2,mapped 0x], hda1, /boot/vmlinuz-2.6.9-55.0.12.ELxenU, /boot/initrd-2.6.9-55.0.12.ELxenU.img) domUloader: kpartx -p -part -a '/dev/loop6' domUloader: mount -o ro '/dev/mapper/loop6-part1' /var/lib/xen/tmp/hda1.AQ4Pxe domUloader: cp /var/lib/xen/tmp/hda1.AQ4Pxe/boot/vmlinuz-2.6.9-55.0.12.ELxenU /var/lib/xen/tmp/kernel.ERe1A5 Copy kernel /boot/vmlinuz-2.6.9-55.0.12.ELxenU from hda1 to /var/lib/xen/tmp/kernel.ERe1A5 for booting domUloader: cp /var/lib/xen/tmp/hda1.AQ4Pxe/boot/initrd-2.6.9-55.0.12.ELxenU.img /var/lib/xen/tmp/ramdisk.9mYR_x domUloader: umount /var/lib/xen/tmp/hda1.AQ4Pxe domUloader: kpartx -p -part -d '/dev/loop6' domUloader: losetup -d /dev/loop6 root@os4 xen # dmsetup ls No devices found but sometimes it "fails" like this: root@os4 xen # /usr/lib/xen/boot/domUloader.py --verbose --args=root=/dev/mapper/VolGroup00-LogVol01 --output=/tmp/K --entry=hda1:/boot/vmlinuz-2.6.9-55.0.12.ELxenU ,/boot/initrd-2.6.9-55.0.12.ELxenU.img /etc/xen/images/os-centos4u4-flat.vmdk domUloader: ['/usr/lib/xen/boot/domUloader.py', '--verbose', '--args=root=/dev/mapper/VolGroup00-LogVol01', '--output=/tmp/K', '--entry=hda1:/boot/vmlinuz-2.6.9-55. 0.12.ELxenU,/boot/initrd-2.6.9-55.0.12.ELxenU.img', '/etc/xen/images/os-centos4u4-flat.vmdk'] domUloader: vdisk not specified; guessing 'hda' based on 'hda1' domUloader: losetup /dev/loop6 '/etc/xen/images/os-centos4u4-flat.vmdk' domUloader: kpartx -l: loop6-part1 : 0 208782 /dev/loop6 63 domUloader: Found partition: vdev hda1, pdev /dev/mapper/loop6-part1 domUloader: kpartx -l: loop6-part2 : 0 22844430 /dev/loop6 208845 domUloader: Found partition: vdev hda2, pdev /dev/mapper/loop6-part2 domUloader: copyKernelAndRamdisk([hda,/etc/xen/images/os-centos4u4-flat.vmdk,/dev/loop6,2,mapped 0x], hda1, /boot/vmlinuz-2.6.9-55.0.12.ELxenU, /boot/initrd-2.6.9-5 5.0.12.ELxenU.img) domUloader: kpartx -p -part -a '/dev/loop6' domUloader: mount -o ro '/dev/mapper/loop6-part1' /var/lib/xen/tmp/hda1.Jj8zUr domUloader: cp /var/lib/xen/tmp/hda1.Jj8zUr/boot/vmlinuz-2.6.9-55.0.12.ELxenU /var/lib/xen/tmp/kernel.BbI1bz Copy kernel /boot/vmlinuz-2.6.9-55.0.12.ELxenU from hda1 to /var/lib/xen/tmp/kernel.BbI1bz for booting domUloader: cp /var/lib/xen/tmp/hda1.Jj8zUr/boot/initrd-2.6.9-55.0.12.ELxenU.img /var/lib/xen/tmp/ramdisk.oXBlOE domUloader: umount /var/lib/xen/tmp/hda1.Jj8zUr domUloader: kpartx -p -part -d '/dev/loop6' device-mapper: remove ioctl failed: Device or resource busy domUloader: losetup -d /dev/loop6 ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy domUloader: losetup -d /dev/loop6 ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy ioctl: LOOP_CLR_FD: Device or resource busy device-mapper: remove ioctl failed: Device or resource busy root@os4 xen # dmsetup ls loop6-part2 (253, 1) VolGroup00-LogVol01 (253, 0) VolGroup00-LogVol00 (253, 2) with iscsi disks I have to run "dmsetup remove xvdp-part2" maually (the VolGroup* do not matter) before it's possible to start the next domU which does use iscsi disks... with local disk images and loop6-part2, VolGroup00-LogVol00 and VolGroup00-LogVol01 block loop6-part2 beeing dmsetup-removed. here I need this sequence to get "clean" again : dmsetup remove VolGroup00-LogVol00 dmsetup remove VolGroup00-LogVol01 dmsetup remove loop6-part2 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.