[Bug 865682] New: Cant start xen domU using drbd block device
https://bugzilla.novell.com/show_bug.cgi?id=865682 https://bugzilla.novell.com/show_bug.cgi?id=865682#c0 Summary: Cant start xen domU using drbd block device Classification: openSUSE Product: openSUSE 13.1 Version: Final Platform: x86-64 OS/Version: openSUSE 13.1 Status: NEW Severity: Major Priority: P5 - None Component: Xen AssignedTo: jdouglas@suse.com ReportedBy: suse@aastral.net QAContact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.90 Safari/537.1 domU config file: ------------------ bootloader="pygrub" on_reboot = 'restart' on_crash = 'restart' name = "test" memory = "384" vcpus=1 cpus="1-15" audio=1 soundhw='es1370' vif = [ 'mac=00:16:3E:32:E3:EE, vifname=test.0, bridge=br400', 'mac=00:16:3E:34:42:DF, vifname=test.1, bridge=br600', 'mac=00:16:3E:5A:CD:3F, vifname=test.2, bridge=br300' ] disk = [ 'drbd:test.root,xvda1,w', 'drbd:test.swap,xvda2,w', ] ----------------- Reproducible: Always Steps to Reproduce: 1. Create DRBD devices 2. Create Xen guest config file specifying drbd device 3. xl -vvvvvvv create <domU> Actual Results: xl -vvvvvv create test Parsing config from test libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x22666a0: create: how=(nil) callback=(nil) poller=0x2262ef0 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvda1, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvda2, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=(null), uses script=... assuming phy backend libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk test.root libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x22666a0: inprogress: poller=0x2262ef0, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x22666a0: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.39.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.39.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: test.root libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: event epath=/local/domain/39 libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.39.log libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [11949] exited with error status 1 libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x2263fa8 wpath=/local/domain/39 token=3/0: deregister slotnum=3 libxl: error: libxl_create.c:900:domcreate_rebuild_done: cannot (re-)build domain: -3 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x22666a0: complete, rc=-3 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x22666a0: destroy xc: debug: hypercall buffer: total allocations:27 total releases:27 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:23 misses:2 toobig:2 Expected Results: Expected that DomU guest starts up. This same setup works on SLES 11.2 OpenSUSE 13.1 with all patches applied as of 2014-02-25 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c1
James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c2
--- Comment #2 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c3
Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c4
Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c5
--- Comment #5 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c6
--- Comment #6 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c7
--- Comment #7 from Mike Latimer
Correction:
This is the package I installed. Not the one in the previous post.
http://download.opensuse.org/repositories/home:/smarty12/openSUSE_13.1/x86_6...
Thank you. That package gets me into a working drbd state, but I am still encountering segfaults that might be the root of your problem as well. With the drbd resource active, can you get me the output of the following command: # /sbin/drbdadm sh-dev test.root On my test machine, that command segfaults, which in turn causes the entire block attach to fail. If the above command is successful, add 'set -x' to the beginning of /etc/xen/scripts/block-drbd and try to bring up the domain again. After it fails, grab the /var/log/xen/xen-hotplug.log file and attach it to this bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c8
Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c9
--- Comment #9 from Mike Latimer
I can save you the trouble.
With the definition disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ] The block-drbd script is never called
That's not what is happening in my environment. The block-drdb script is executing just fine. However, you do need the fix Jim mentioned in comment #1. With that in place, the block-attach is still failing - but due to drbdadm segfaulting in two places. If I replace those commands with the hard-coded correct response (in my system), the disk attaches properly and the VM is started. (At least in my environment.)
If the disk definition is this: (completely bypassing drbd) the guest starts up. But only if I make the drbd device primary before starting the guest: disk = [ 'phy:/dev/drbd/by-res/test.root/0,sda1,w', 'phy:/dev/drbd/by-res/test.swap/0,sda2,w' ]
This makes sense, as Xen would not even know about drbd with this setup.
This is the only way I can get block-drbd script to run, but the guest doesn't start up. So it looks like xl is having trouble parsing the stuff after the drbd: directive. Maybe it is expecting /some/sort/of/path instead of just the test.root drbd device.
One difference between your environment and mine is that I am not using drbd for root. This changes the startup a bit, but still leaves me in a situation where xvdb1 cannot be attached. It might be interesting to change your disk configuration to something like: disk = [ 'phy:/dev/drbd/by-res/test.root/0,sda1,w', 'drbd:test.swap,xvda2,w', ] Just to see if the system logs end up closer to mine... Either way, the key to tracking this down is a debug xen-hotpug log. Please add 'set -x' at the beginning of /etc/xen/scripts/block-drbd, try the test again (with 'drbd:test.root' syntax), and attach /var/log/xen/xen-hotplug.log after it fails. If there is nothing in this log when using the correct syntax, try it again using your /dev/drbd/... syntax and send me the debug log from that run. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c10
--- Comment #10 from Bill Weidman
#!/bin/bash 251c251 < drbd|phy)
drbd)
279c279 < drbd|phy) ---
drbd)
=========== xl -vvvvvv create test Parsing config from test libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x1bef510: create: how=(nil) callback=(nil) poller=0x1befa80 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda1 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda1, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda2 spec.backend=unknown libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=sda2, uses script=... assuming phy backend libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda2, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=phy libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=(null), uses script=... assuming phy backend libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: locally attaching PHY disk test.root libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x1bef510: inprogress: poller=0x1befa80, flags=i libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: register slotnum=3 libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x1bef510: progress report: ignored libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.7.out libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0 libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.7.d libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: test.root libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: event epath=/local/domain/7 libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.7.log libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [12772] exited with error status 1 libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x1befde8 wpath=/local/domain/7 token=3/0: deregister slotnum=3 libxl: error: libxl_create.c:900:domcreate_rebuild_done: cannot (re-)build domain: -3 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x1bef510: complete, rc=-3 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x1bef510: destroy xc: debug: hypercall buffer: total allocations:27 total releases:27 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:23 misses:2 toobig:2 ============ cat /var/log/xen/xen-hotplug.log +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set ++++ grep '^LC_' ++++ set ++++ cut -d= -f1 ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/4/2049 +++ local level=debug +++ unset LC_CTYPE +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/4/2049 +++ trap sigerr ERR +++ log debug add XENBUS_PATH=backend/vbd/4/2050 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: add XENBUS_PATH=backend/vbd/4/2050 ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ findCommand add ++ for arg in '"$@"' ++ expr index add = ++ command=add ++ return ++ command=add ++ return ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/4/2049 ++ '[' add '!=' add ']' ++ XENBUS_PATH=backend/vbd/4/2050 ++ xenstore_read_default backend/vbd/4/2050/type MISSING ++ xenstore_read_default backend/vbd/4/2049/type MISSING ++ xenstore-read backend/vbd/4/2050/type ++ xenstore-read backend/vbd/4/2049/type + t=phy + t=phy + case "$command" in + case "$command" in ++ xenstore_read_default backend/vbd/4/2049/physical-device MISSING ++ xenstore-read backend/vbd/4/2049/physical-device ++ xenstore_read_default backend/vbd/4/2050/physical-device MISSING ++ xenstore-read backend/vbd/4/2050/physical-device ++ echo MISSING ++ echo MISSING + phys=MISSING + phys=MISSING + '[' MISSING '!=' MISSING ']' + '[' MISSING '!=' MISSING ']' + '[' -n phy ']' + '[' -n phy ']' ++ xenstore_read backend/vbd/4/2049/params ++ xenstore_read backend/vbd/4/2050/params +++ xenstore-read backend/vbd/4/2049/params +++ xenstore-read backend/vbd/4/2050/params ++ local v=/dev/drbd/by-res/test.root/0 ++ local v=/dev/drbd/by-res/test.swap/0 ++ '[' /dev/drbd/by-res/test.root/0 '!=' '' ']' ++ '[' /dev/drbd/by-res/test.swap/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.root/0 ++ echo /dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.root/0 ++ xenstore_read backend/vbd/4/2050/mode ++ xenstore_read backend/vbd/4/2049/mode +++ xenstore-read backend/vbd/4/2050/mode +++ xenstore-read backend/vbd/4/2049/mode ++ local v=w ++ '[' w '!=' '' ']' ++ echo w ++ local v=w ++ '[' w '!=' '' ']' ++ echo w + mode=w + case $t in + drbd_resource=/dev/drbd/by-res/test.swap/0 + mode=w + case $t in + drbd_resource=/dev/drbd/by-res/test.root/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 dev/drbd/by-res/test.swap/0 is not a valid number dev/drbd/by-res/test.root/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error + drbd_role= ++ log debug 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ local level=debug ++ sigerr ++ shift ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ _xenstore_write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log debug 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ xenstore-write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ xenstore-write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ exit 1 ++ exit 1 +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ set +++ export PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games +++ PATH=/usr/bin:/usr/sbin:/usr/lib/xen/bin:/usr/lib64/xen/bin:/sbin:/bin:/usr/bin:/usr/sbin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games ++++ grep '^LC_' +++ export LD_LIBRARY_PATH=/usr/lib64 +++ LD_LIBRARY_PATH=/usr/lib64 +++ export LANG=POSIX +++ LANG=POSIX ++++ cut -d= -f1 ++++ set ++++ grep '^LC_' ++++ cut -d= -f1 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug remove XENBUS_PATH=backend/vbd/4/2049 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: remove XENBUS_PATH=backend/vbd/4/2049 +++ unset LC_CTYPE +++ trap sigerr ERR +++ log debug remove XENBUS_PATH=backend/vbd/4/2050 +++ local level=debug +++ shift +++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: remove XENBUS_PATH=backend/vbd/4/2050 ++ findCommand remove ++ for arg in '"$@"' ++ expr index remove = ++ command=remove ++ return ++ findCommand remove ++ '[' remove '!=' add ']' ++ for arg in '"$@"' ++ '[' remove '!=' remove ']' ++ expr index remove = ++ XENBUS_PATH=backend/vbd/4/2049 ++ xenstore_read_default backend/vbd/4/2049/type MISSING ++ command=remove ++ return ++ xenstore-read backend/vbd/4/2049/type ++ '[' remove '!=' add ']' ++ '[' remove '!=' remove ']' ++ XENBUS_PATH=backend/vbd/4/2050 ++ xenstore_read_default backend/vbd/4/2050/type MISSING ++ xenstore-read backend/vbd/4/2050/type + t=phy + case "$command" in + case $t in ++ xenstore_read backend/vbd/4/2049/params +++ xenstore-read backend/vbd/4/2049/params + t=phy + case "$command" in + case $t in ++ xenstore_read backend/vbd/4/2050/params +++ xenstore-read backend/vbd/4/2050/params ++ local v=/dev/drbd/by-res/test.root/0 ++ '[' /dev/drbd/by-res/test.root/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.root/0 + p=/dev/drbd/by-res/test.root/0 + drbd_resource=/dev/drbd/by-res/test.root/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 ++ local v=/dev/drbd/by-res/test.swap/0 ++ '[' /dev/drbd/by-res/test.swap/0 '!=' '' ']' ++ echo /dev/drbd/by-res/test.swap/0 + p=/dev/drbd/by-res/test.swap/0 + drbd_resource=/dev/drbd/by-res/test.swap/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 dev/drbd/by-res/test.root/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ log debug 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2049/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status 'error to xenstore.' dev/drbd/by-res/test.swap/0 is not a valid number + drbd_role= ++ sigerr ++ fatal '/etc/xen/scripts/block-drbd failed; error detected.' ++ _xenstore_write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ log debug 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ local level=debug ++ shift ++ logger -p daemon.debug -- /etc/xen/scripts/block-drbd: 'Writing backend/vbd/4/2050/hotplug-error' '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status 'error to xenstore.' ++ xenstore-write backend/vbd/4/2049/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2049/hotplug-status error ++ xenstore-write backend/vbd/4/2050/hotplug-error '/etc/xen/scripts/block-drbd failed; error detected.' backend/vbd/4/2050/hotplug-status error ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ log err '/etc/xen/scripts/block-drbd failed; error detected.' ++ local level=err ++ shift ++ logger -p daemon.err -- /etc/xen/scripts/block-drbd: '/etc/xen/scripts/block-drbd failed; error detected.' ++ exit 1 ++ exit 1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c11
--- Comment #11 from Mike Latimer
Okay so with this disk definition: disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ]
And the patch applied:
Ok, good starting point. ;)
+ case $t in + drbd_resource=/dev/drbd/by-res/test.root/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.swap/0 ++ /sbin/drbdadm role /dev/drbd/by-res/test.root/0 dev/drbd/by-res/test.swap/0 is not a valid number dev/drbd/by-res/test.root/0 is not a valid number + drbd_role= ++ sigerr
This 'sigerr' is in the place I am talking about (caused by drbdadm). However, it looks like this log is from an earlier configuration - while using the /dev/drbd/by-res... syntax. The drbd_resource should match what is in the drbd config above (e.g. test.root or test.swap). It could be that this block-drbd script was never executed after changing the syntax, or there could be something else strange going on. Can you clear the file by echoing "" into it, then try the test again (using the test.root syntax)? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c12
--- Comment #12 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c13
--- Comment #13 from Mike Latimer
Yeah block-drbd isn't executing with: disk = [ 'drbd:test.root,sda1,w', 'drbd:test.swap,sda2,w', ]
hotplug log is empty, block-drbd doesn't get called.
Interesting... I can see the same behavior (no block-drbd execution) if I use the drdb device as my root device. Going back to comment #10, I can see the same error reported in your logs:
libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.7.log
Can you post the contents of bootloader.7.log here? Also, you are saying this same configuration (of having a root device on a drbd device, with the 'drbd:test.root' syntax) was working previous to openSUSE 13.1? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c14
--- Comment #14 from Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c15
--- Comment #15 from Mike Latimer
After playing around with this a bit more, we should see what happens when we attach the device to dom0. Delete /var/log/xen/xen-hotplug.log, then execute the following command:
xl block-attach 0 'drbd:test.root,sda1,w'
Opps - change the sda1 to xvda1: xl block-attach 0 'drbd:test.root,xvda1,w' as this will create the specified device in dom0. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c17
--- Comment #17 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c18
--- Comment #18 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c19
--- Comment #19 from Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c20
Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c21
--- Comment #21 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c22
--- Comment #22 from Mike Latimer
Looking at the errors, if we use the drbd: directive in the config file, the block-drbd script is never called.
However, if we mount the drbd block device manually to dom0, the script is called and works correctly.
Therefore I think the problem is in the xl executable or the xl libraries.
Could be. Let's back up a bit and see if strace shows anything interesting. Can you do the following and send me the strace.log file? strace -s 500 -f -o strace.log xl create <domU> In the meantime, I'll keep looking through the xl code. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c23
--- Comment #23 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c24
Mike Latimer
Okay I've got the output, but it's 3500 lines long. How would you like me to submit this data?
You can just gzip it and attach it to this bug. However, the strace runs I've done here haven't been helpfuly, so it's probably safe to not bother submitting it. I do have some good news though, I think I've found the root of the problem. It turns out that the disk is being treated as a local device and is being passed straight through. There is a patch to run the appropriate block setup script first in this thread: http://lists.xen.org/archives/html/xen-devel/2014-02/msg01092.html. I should have some test code before too long. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c25
Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c26
--- Comment #26 from Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c27
--- Comment #27 from Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c28
--- Comment #28 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c29
--- Comment #29 from Mike Latimer
I've finished building the fix for this, but it might take another 1-2 hours to sync to the download area. When that completes (you'll get a 404 until then), can you please install the xen-tools RPM from my repo here:
http://download.opensuse.org/repositories/home:/mlatimer:/branches:/Virtuali...
I'm not sure why, but the publishing process still hasn't completed... Go ahead and grab the xen-tools rpm at the following link and let me know if this solves the problem: ftp://ftp.novell.com/outgoing/xen-tools-4.4.0_04-307.1.x86_64.rpm You might also need an updated xen-libs: ftp://ftp.novell.com/outgoing/xen-libs-4.4.0_04-307.1.x86_64.rpm -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c30
--- Comment #30 from Mike Latimer
(In reply to comment #25)
I've finished building the fix for this, but it might take another 1-2 hours to sync to the download area. When that completes (you'll get a 404 until then), can you please install the xen-tools RPM from my repo here:
http://download.opensuse.org/repositories/home:/mlatimer:/branches:/Virtuali...
I fixed the publishing problem on my end, so the above repo should work now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c31
--- Comment #31 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c32
Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c33
--- Comment #33 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c34
--- Comment #34 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c35
--- Comment #35 from Mike Latimer
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c36
Mike Latimer
xen-4.4.0_04-307.1.x86_64 xen-kmp-default-4.4.0_04_k3.11.10_7-307.1.x86_64 xen-libs-4.4.0_04-307.1.x86_64 xen-tools-4.4.0_04-307.1.x86_64
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c37
--- Comment #37 from Bill Weidman
https://bugzilla.novell.com/show_bug.cgi?id=865682
https://bugzilla.novell.com/show_bug.cgi?id=865682#c38
Mike Latimer
Yes everything is identical.
However, I tested again again to confirm and I can't reproduce. I migrated back-and-forth several times without errors.
In between the last test and this test, the xen dom0s were powered down. There may have been some deleted libraries or something that were held open until the reboot.
So at this point the reported error looks like it was due to the upgrade.
Thanks for the good news. I'll leave this open for a bit just in case, but I do expect to be able to close this soon. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
--- Comment #46 from Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
--- Comment #47 from Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
SMASH SMASH
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
--- Comment #48 from Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=865682
Swamp Workflow Management
participants (1)
-
bugzilla_noreply＠novell.com