[Bug 856973] New: Attempts to start domU virtual machines created in openSUSE 12.3 fail to work in openSUSE 13.1
https://bugzilla.novell.com/show_bug.cgi?id=856973 https://bugzilla.novell.com/show_bug.cgi?id=856973#c0 Summary: Attempts to start domU virtual machines created in openSUSE 12.3 fail to work in openSUSE 13.1 Classification: openSUSE Product: openSUSE 13.1 Version: Final Platform: x86-64 OS/Version: openSUSE 13.1 Status: NEW Severity: Normal Priority: P5 - None Component: Xen AssignedTo: jdouglas@suse.com ReportedBy: stanley@stronglg.demon.co.uk QAContact: qa-bugs@suse.de Found By: --- Blocker: --- Created an attachment (id=573020) --> (http://bugzilla.novell.com/attachment.cgi?id=573020) Snippet from /var/log/messages User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 Using YaST -> Virtualization -> Virtual Machine Manager, domU machines created in openSUSE 12.3 and earlier start up in openSUSE 13.1 but will fail after approximately 90 seconds with the following error: Error: Device 0 (vif) could not be connected. Hotplug scripts not working. Attempts to create a new virtual machine also fail after 90 seconds with the same error. Reproducible: Always Steps to Reproduce: 1. YaST -> Virtualization -> Virtual Machine Manager 2. start localhost(xen) and then double click on an existing domU 3. press start (and then un-pause) and watch virtual machine boot 4. either wait 90 seconds or attempt to log in and use 5. virtual machine will be shut down Actual Results: After approximately 90 seconds the virtual machine disappears and this error is displayed in a pop-up window Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 0 (vif) could not be connected. Hotplug scripts not working.') Expected Results: Old domU virtual machines should work under openSUSE 13.1 as they did in 12.3 I have read http://wiki.xenproject.org/wiki/Xen_Common_Problems where it suggests that This problem is often caused by not having "xen-netback" driver loaded in dom0 kernel. I am not sure if that is the netbk reported by lsmod. lsmod | egrep 'xen|virt' xen_scsibk 32068 0 [permanent] xenbus_be 13496 6 blktap,pciback,usbbk,xen_scsibk,blkbk,netbk xenblk 32254 0 cdrom 46652 2 sr_mod,xenblk xennet 42361 0 I have also attached the relevant section from /var/log/messages. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c1
James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c2
--- Comment #2 from James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c3
--- Comment #3 from Stanley Dziegiel
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c4
--- Comment #4 from James Fehlig
The virsh dumpxml dom-name > dom-name.xml worked for about half of the virtual machines that I had. The other half ended up with xml files of size 1, so I discarded those.
That is odd. Was there an error?
error: internal error: no supported architecture for os type 'hvm'
That error is indicative of some libvirtd configuration issue. What is the output of 'virsh capabilities'? Once we get to the bottom of this problem, you should be able to 'virsh define' your XML files. FYI, I wrote a blog post about the libvirt Xen drivers that you might find helpful too http://jfehlig.wordpress.com/2014/01/05/libvirt-support-for-xens-new-libxenl... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c5
--- Comment #5 from Stanley Dziegiel
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c6
--- Comment #6 from Stanley Dziegiel
(In reply to comment #3) ..
The virsh dumpxml dom-name error was: error: failed to get domain 'Fedora18' error: Domain not found: xenGetDomainDefForName Output of virsh capabilites attached in Comment 5. I can now create new virtual machines, so DSL works. Next step is reviving the existing VMs. Have read the blog post thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c7
James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c8
James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c9
--- Comment #9 from Stanley Dziegiel
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c10
--- Comment #10 from James Fehlig
As a matter of interest which file is actually used: the plain Debian6 or Debian6.xml?
If you were to use xl directly instead of libvirt, then Debian6 would be an appropriate xl configuration file (see 'man xl.cfg). libvirt domain configuration format is always libvirt domXML, so use the .xml file when using libvirt. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c11
--- Comment #11 from Stanley Dziegiel
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c12
--- Comment #12 from James Fehlig
Since the change from xm to xl (comment #3) I can only see the virtual machines that I have created afresh. VMM no longer displays the old virtual machines.
Right. As we've discussed, you have to 'virsh define vm.xml' to import the old machines into libvirt. Did you try that with Debian6.xml after making the change I suggested in #10?
I tried copying the amended Debian6.xml from /etc/xen/vm to /etc/libvirt/libxl where the new dsl13.1.xml got created but it isn't recognized by VMM and I am wary of randomly trying things.
The proper way to "copy" the .xml files to /etc/libvirt/libxl is to use 'virsh define vm.xml'. You can literally copy files to this location, but libvirtd must be restarted so they are read.
I am happy, indeed would prefer, to try using xl to try to recover the old *.xml files (with or without your suggested edit) but I will need some guidance as to what I need to type.
If you have the old xml files, then import them using 'virsh define vm.xml'. If they fail to import, then let's figure out why. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c13
Thomas Wagner
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c14
--- Comment #14 from Stanley Dziegiel
Right. As we've discussed, you have to 'virsh define vm.xml' to import the old machines into libvirt. Did you try that with Debian6.xml after making the change I suggested in #10?
Thank you, I have run virsh define Debian6.xml with driver name='qemu' and both restarted libvirtd and later rebooted but both attempts lead to this Traceback: Error starting domain: internal error: libxenlight failed to create new domain 'Debian6' Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1168, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 697, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: libxenlight failed to create new domain 'Debian6'
The proper way to "copy" the .xml files to /etc/libvirt/libxl is to use 'virsh define vm.xml'. You can literally copy files to this location, but libvirtd must be restarted so they are read.
If you have the old xml files, then import them using 'virsh define vm.xml'. If they fail to import, then let's figure out why.
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c15
--- Comment #15 from James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c16
--- Comment #16 from Stanley Dziegiel
Maybe there is something telling in /var/log/libvirt/libxl/Debian6.log
These 8 lines are printed to /var/log/libvirt/libxl/Debian6.log every time I click on the start button: libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x7f0090006240: create: how=(nil) callback=(nil) poller=0x7f00ac0009a0 libxl: error: libxl.c:321:libxl__domain_rename: domain with name "Debian6" already exists. libxl: error: libxl_create.c:644:initiate_domain_create: cannot make domain: -6 libxl: error: libxl_dm.c:1312:libxl__destroy_device_model: could not find device-model's pid for dom 3 libxl: error: libxl.c:1416:libxl__destroy_domid: libxl__destroy_device_model failed for 3 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7f0090006240: complete, rc=-3 libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7f0090006240: inprogress: poller=0x7f00ac0009a0, flags=ic libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7f0090006240: destroy -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c17
--- Comment #17 from James Fehlig
libxl: error: libxl.c:321:libxl__domain_rename: domain with name "Debian6" already exists.
Seems Debian6 is already running. Did you happen to previously start it with xl? Or maybe the libvirt libxl driver did not properly cleanup a previously failed start attempt, leaving behind a skeletal domain. Do you see Debian6 with 'xl list'? Is it running and accessible? Can you access its console or framebuffer (VNC)? If the domain is running and responds to shutdown requests, you can use 'xl shutdown Debian6' to gracefully shut it down, otherwise 'xl destroy Debian6' to destroy it. Then try starting Debian6 with libvirt (virsh) or virt-manager. BTW, it seems you succeeded importing your domains into libvirt with 'virsh define dom.xml', since the traceback in #14 shows that you are trying to start one with virt-manager. Is that correct? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c18
--- Comment #18 from Stanley Dziegiel
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c19
--- Comment #19 from Stanley Dziegiel
(In reply to comment #16)
libxl: error: libxl.c:321:libxl__domain_rename: domain with name "Debian6" already exists.
Seems Debian6 is already running. Did you happen to previously start it with xl? Or maybe the libvirt libxl driver did not properly cleanup a previously failed start attempt, leaving behind a skeletal domain.
I did not knowingly start it with xl. After the reboot I have always used YaST -> VMM to start Debian6, so if there is a skeletal domain it must persist across reboots.
Do you see Debian6 with 'xl list'? Is it running and accessible? Can you access its console or framebuffer (VNC)?
Yes: xl list Name ID Mem VCPUs State Time(s) Domain-0 0 6797 4 r----- 2991.3 Debian6 1 512 1 r----- 107951.1 I have forgotten how to access its console/frame buffer. What I want is for everything to work via VMM as it did before the upgrade to openSuSE 13.1.
If the domain is running and responds to shutdown requests, you can use 'xl shutdown Debian6' to gracefully shut it down, otherwise 'xl destroy Debian6' to destroy it. Then try starting Debian6 with libvirt (virsh) or virt-manager.
I ran xl shutdown Debian6 but it seemed to have no effect as xl list reports the same as above. xl destroy Debian6 did work as a subsequent xl list only displayed the Domain-0 entry. I then tried my preferred sequence of: YaST -> VMM -> localhost(xen) -> Debian6 -> <click start button> but I get the same Traceback as I usually do (posted in #14).
BTW, it seems you succeeded importing your domains into libvirt with 'virsh define dom.xml', since the traceback in #14 shows that you are trying to start one with virt-manager. Is that correct?
Yes I did succeed in importing the Debian6 domain (see first line of post #14). FYI: the log file now contains many more lines each time I click the start button. I am attaching that log (post #17: 123 lines). The only things that stand out are the error messages about xenbr0 as they do not tie up with the output of netstat -nr: Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 br0 0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br0 as that is using br0. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c20
--- Comment #20 from James Fehlig
The only things that stand out are the error messages about xenbr0 as they do not tie up with the output of netstat -nr:
Interesting. The default bridge name used if none is specified is defined low in the libxl stack, and for historical reasons is xenbr0. There is some compat code to look for other suitable bridges if xenbr0 is not found, and perhaps you have encountered a regression in that code. Does Debian6 start if you explicitly specify the bridge in the xml config? E.g. <interface type='bridge' model='rtl8139'> <source bridge='br0'/> <mac address='00:16:3e:60:ad:1c'/> <script path='/etc/xen/scripts/vif-bridge'/> </interface> You can use 'virsh edit Debian6' to do that or edit the xml file directly and "reimport" with 'virsh define'. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=856973
https://bugzilla.novell.com/show_bug.cgi?id=856973#c21
--- Comment #21 from Stanley Dziegiel
participants (1)
-
bugzilla_noreply@novell.com