[opensuse-virtual] after leap 42.3 -> 15.0 upgrades, Xen/EFI servers' multiple services fail with "failed with result 'dependency'"
Upgrading working OpenSUSE leap42.3 / Xen servers (5, atm) to leap15.0, after upgrade lsb_release -rd Description: openSUSE Leap 15.0 Release: 15.0 uname -rm 4.16.12-lp150.2.g39c7522-default x86_64 rpm -qa | egrep -i "xen|qemu|efi" | sort efibootmgr-14-lp150.2.2.x86_64 efivar-devel-31-lp150.1.11.x86_64 grub2-x86_64-efi-2.02-lp150.35.1.x86_64 grub2-x86_64-xen-2.02-lp150.35.1.x86_64 libefivar1-31-lp150.1.11.x86_64 qemu-2.12.0-lp150.453.1.x86_64 qemu-arm-2.12.0-lp150.453.1.x86_64 qemu-block-curl-2.12.0-lp150.453.1.x86_64 qemu-block-dmg-2.12.0-lp150.453.1.x86_64 qemu-extra-2.12.0-lp150.453.1.x86_64 qemu-ipxe-1.0.0-lp150.453.1.noarch qemu-ksm-2.12.0-lp150.453.1.x86_64 qemu-kvm-2.12.0-lp150.453.1.x86_64 qemu-lang-2.12.0-lp150.453.1.x86_64 qemu-linux-user-2.12.0-lp150.453.1.x86_64 qemu-ovmf-x86_64-2018+git1525854636.13e3f8c03339-lp150.75.1.noarch qemu-seabios-1.11.1-lp150.453.1.noarch qemu-sgabios-8-lp150.453.1.noarch qemu-tools-2.12.0-lp150.453.1.x86_64 qemu-vgabios-1.11.1-lp150.453.1.noarch qemu-x86-2.12.0-lp150.453.1.x86_64 xen-4.10.1_02-lp150.576.1.x86_64 xen-libs-4.10.1_02-lp150.576.1.x86_64 xen-tools-4.10.1_02-lp150.576.1.x86_64 No notable version changes, just the switch to lp150 packages. serial output @ boot ... Booting `OpenSUSE, with Xen hypervisor'Booting `OpenSUSE, with Xen hypervisor' Loading Xen 4.10.1_02-lp150.576 with Linux 4.16.12-lp150.2.g39c7522-default ...Loading Xen 4.10.1_02-lp150.576 with Linux 4.16.12-lp150.2.g39c7522-default ... /EndEntire /EndEntire file path: file path: /ACPI(a0341d0,0)/ACPI(a0341d0,0)/PCI(1,1c)/PCI(1,1c)/PCI(0,0)/PCI(0,0)/PCI(0,1)/PCI(0,1)/PCI(0,0)/PCI(0,0)/HardwareVendor (cf31fac5-c24e-11d2-85f3-00a0c93ec93b)[1: /HardwareVendor(cf31fac5-c24e-11d2-85f3-00a0c93ec93b)[1: 88 88 ]]/HD(2,1000,96000,3d7310980ce0754c ,2,2)/HD(2,1000,96000,3d7310980ce0754c,2,2)/File(\efi\opensuse) /File(\efi\opensuse)/File(xen-4.10.1_02-lp150.576.efi)/File(xen-4.10.1_02-lp150.576.efi)/EndEntire /EndEntire Xen 4.10.1_02-lp150.576 (c/s ) EFI loader Using configuration file 'xen-4.10.1_02-lp150.576.cfg' vmlinuz-4.16.12-lp150.2.g39c7522-default: 0x000000008b917000-0x000000008c073da8 initrd-4.16.12-lp150.2.g39c7522-default: 0x000000008a856000-0x000000008b916e60 0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0x928b9018 0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0x928b0018 0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0x92894018 __ __ \ \/ /___ _ __ \ // _ \ '_ \ / \ __/ | | | /_/\_\___|_| |_| _ _ _ ___ _ ___ ____ _ _ ____ ___ ____ _____ __ | || | / |/ _ \ / | / _ \___ \ | |_ __ / | ___| / _ \ | ___|___ / /_ | || |_ | | | | || | | | | |__) |__| | '_ \| |___ \| | | ||___ \ / / '_ \ |__ _|| | |_| || | | |_| / __/|__| | |_) | |___) | |_| | ___) |/ /| (_) | |_|(_)_|\___(_)_|___\___/_____| |_| .__/|_|____/ \___(_)____//_/ \___/ |_____| |_| (XEN) Xen version 4.10.1_02-lp150.576 (abuild@suse.de) (gcc (SUSE Linux) 8.1.1 20180523 [gcc-8-branch revision 260570]) debug=n Fri May 18 14:03:46 UTC 2018 (XEN) Latest ChangeSet: (XEN) Console output is synchronous. (XEN) Bootloader: EFI (XEN) Command line: dom0_mem=4016M,max:4096M bootscrub=false dom0_max_vcpus=4 vga=gfx-1920x1080x16 com1=115200,8n1,pci console=com1,vga console_timestamps console_to_ring conring_size=64 sc hed=credit2 reboot=acpi log_buf_len=16M loglvl=all guest_loglvl=all noreboot=false sync_console=true sched_debug iommu=verbose apic_verbosity=verbose ... dmesg ... [ 0.000000] Hypervisor detected: Xen PV ... But checking, xl list Name ID Mem VCPUs State Time(s) (null) 0 4016 4 r----- 329.6 Xenstore 1 31 1 -b---- 0.0 and dmesg | grep -i xen [ 0.000000] Command line: root=/dev/mapper/VG0 softlevel=xen rd.shell rd.debug=0 rd.udev.log_priority=info rd.auto=1 dolvm lvmwait=/dev/mapper/VG0 root=/dev/mapper/VG0 rootfstype=ext4 rootflags=journal_checksum noresume video=vesa:off video=efifb:1024x768 video=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=hvc0 elevator=deadline cpuidle cpufreq=xen:ondemand clocksource=xen net.ifnames=1 biosdevname=0 plymouth.enable=0 apparmor=0 pcie_aspm=off mce=off fsck.mode=skip fsck.repair=preen nomodeset nouveau.modeset=1 showopts noquiet log_buf_len=10M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=xen,keep [ 0.000000] Xen: [mem 0x0000000000000000-0x0000000000007fff] reserved ... [ 0.000000] Xen: [mem 0x000000016f2a8000-0x000000085dffffff] unusable [ 0.000000] Hypervisor detected: Xen PV [ 0.000000] Kernel/User page tables isolation: disabled on XEN PV. [ 0.000000] Setting APIC routing to Xen PV. [ 0.000000] Booting paravirtualized kernel on Xen [ 0.000000] Xen version: 4.10.1_02-lp150.576 (preserve-AD) [ 0.000000] xen: PV spinlocks enabled [ 0.000000] Kernel command line: root=/dev/mapper/VG0 softlevel=xen rd.shell rd.debug=0 rd.udev.log_priority=info rd.auto=1 dolvm lvmwait=/dev/mapper/VG0 root=/dev/mapper/VG0 rootfstype=ext4 rootflags=journal_checksum noresume video=vesa:off video=efifb:1024x768 video=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=hvc0 elevator=deadline cpuidle cpufreq=xen:ondemand clocksource=xen net.ifnames=1 biosdevname=0 plymouth.enable=0 apparmor=0 pcie_aspm=off mce=off fsck.mode=skip fsck.repair=preen nomodeset nouveau.modeset=1 showopts noquiet log_buf_len=10M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=xen,keep [ 0.004000] xen:events: Using FIFO-based ABI [ 0.004000] xen: --> pirq=1 -> irq=1 (gsi=1) ... [ 0.004000] xen: --> pirq=15 -> irq=15 (gsi=15) [ 0.004000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.004000] Xen: using vcpuop timer interface [ 0.004000] installing Xen timer for CPU 0 [ 0.192039] installing Xen timer for CPU 1 [ 0.232038] installing Xen timer for CPU 2 [ 0.272060] installing Xen timer for CPU 3 [ 0.360014] xen:grant_table: Grant tables using version 1 layout [ 0.464199] xen: registering gsi 9 triggering 0 polarity 0 ... [ 21.864462] xen: registering gsi 17 triggering 0 polarity 1 [ 27.472745] xen:xen_evtchn: Event-channel device installed [ 27.753699] xen_pciback: backend is vpci [ 27.820997] xen_acpi_processor: Uploading Xen processor PM info [ 59.589179] xen: registering gsi 19 triggering 0 polarity 1 [ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating. [ 117.961491] systemd[1]: Failed to start The Xen xenstore. [ 117.961541] systemd[1]: Dependency failed for Xendomains - start and stop guests on boot and shutdown. [ 117.961563] systemd[1]: xendomains.service: Job xendomains.service/start failed with result 'dependency'. [ 117.961577] systemd[1]: Dependency failed for qemu for xen dom0 disk backend. [ 117.961590] systemd[1]: Dependency failed for xencommons. [ 117.961605] systemd[1]: xencommons.service: Job xencommons.service/start failed with result 'dependency'. [ 117.961624] systemd[1]: xen-qemu-dom0-disk-backend.service: Job xen-qemu-dom0-disk-backend.service/start failed with result 'dependency'. [ 117.961628] systemd[1]: Dependency failed for Xenconsoled - handles logging from guest consoles and hypervisor. [ 117.961643] systemd[1]: xenconsoled.service: Job xenconsoled.service/start failed with result 'dependency'. systemctl status -l xen-qemu-dom0-disk-backend.service ● xen-qemu-dom0-disk-backend.service - qemu for xen dom0 disk backend Loaded: loaded (/usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service; enabled; vendor preset: disabled) Active: inactive (dead) systemctl list-unit-files | grep -i xen home-xen.mount generated proc-xen.mount static var-lib-xenstored.mount static xen-dom0-modules.service disabled xen-init-dom0.service enabled xen-qemu-dom0-disk-backend.service enabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service enabled xendomains.service enabled xenstored.service enabled cat /usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service [Unit] Description=qemu for xen dom0 disk backend Requires=proc-xen.mount xenstored.service After=proc-xen.mount xenstored.service xenconsoled.service Before=xendomains.service libvirtd.service libvirt-guests.service RefuseManualStop=true ConditionPathExists=/proc/xen/capabilities [Service] Type=simple PIDFile=/var/run/xen/qemu-dom0.pid ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities ExecStartPre=/bin/mkdir -p /var/run/xen ExecStart=/usr/bin/qemu-system-i386 -xen-domid 0 \ -xen-attach -name dom0 -nographic -M xenpv -daemonize \ -monitor /dev/null -serial /dev/null -parallel /dev/null \ -nodefaults -no-user-config \ -pidfile /var/run/xen/qemu-dom0.pid [Install] WantedBy=multi-user.target systemctl status -l xenstored.service ● xenstored.service - The Xen xenstore Loaded: loaded (/usr/lib/systemd/system/xenstored.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Wed 2018-05-30 10:37:08 PDT; 10min ago Main PID: 1391 (code=exited, status=0/SUCCESS) May 30 10:35:39 xserver launch-xenstore[1391]: Starting /usr/lib/xen/boot/xenstore-stubdom.gz... Not sure yet -- is this a new bug, or a config change, issue? -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
switching to my own repo's builds of qemu (which I haven't used in production on 42.3 for ages), in preference over d.o.o's Virtualization repo, cat LOC-qemu.repo [LOC-qemu] enabled=1 baseurl=https://download.opensuse.org/repositories/home:/pgnd:/Virtualization:/qemu/... priority=11 ... cat Virtualization.repo [Virtualization] enabled=1 priority=12 baseurl=https://download.opensuse.org/repositories/Virtualization/openSUSE_Leap_15.0 ... zypper dup --from LOC-qemu ... now, on reboot, all looks good dmesg | grep -i xen | egrep -i "fail" (empty) xl list Name ID Mem VCPUs State Time(s) Domain-0 0 4016 4 r----- 548.1 Xenstore 1 31 1 -b---- 0.0 systemctl status -l `systemctl list-unit-files | grep -i xen | grep enabled | awk '{print $1}'` ● xen-init-dom0.service - xen-init-dom0, initialise Dom0 configuration (xenstore nodes, JSON configuration stub) Loaded: loaded (/usr/lib/systemd/system/xen-init-dom0.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1552 ExecStart=/usr/lib/xen/bin/xen-init-dom0 (code=exited, status=0/SUCCESS) Process: 1549 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1552 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xen-init-dom0.service May 30 12:56:36 xen001 xen-init-dom0[1552]: Done setting up Dom0 ● xen-qemu-dom0-disk-backend.service - qemu for xen dom0 disk backend Loaded: loaded (/usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1556 ExecStartPre=/bin/mkdir -p /var/run/xen (code=exited, status=0/SUCCESS) Process: 1555 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1763 (qemu-system-i38) Tasks: 4 (limit: 9830) CGroup: /system.slice/xen-qemu-dom0-disk-backend.service └─1763 /usr/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null -parallel /dev/null -nodefaults -no-user> ● xencommons.service - xencommons Loaded: loaded (/usr/lib/systemd/system/xencommons.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2018-05-30 12:57:05 PDT; 10min ago Process: 1923 ExecStartPost=/bin/sh -c mv -vf /var/log/xen/xen-boot.log /var/log/xen/xen-boot.prev.log ; /usr/sbin/xl dmesg > /var/log/xen/xen-boot.log (code=exited, status=0/SUCCESS) Process: 1918 ExecStart=/usr/bin/xenstore-ls -f (code=exited, status=0/SUCCESS) Process: 1917 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1918 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xencommons.service May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/backends/vfb = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/backends/qnic = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/state = "running" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1 = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/domid = "1" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/name = "Xenstore" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory/target = "32768" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory/static-max = "331776" May 30 12:57:05 xen001.example.com sh[1923]: renamed '/var/log/xen/xen-boot.log' -> '/var/log/xen/xen-boot.prev.log' ● xenconsoled.service - Xenconsoled - handles logging from guest consoles and hypervisor Loaded: loaded (/usr/lib/systemd/system/xenconsoled.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1553 ExecStartPre=/bin/mkdir -p ${XENCONSOLED_LOG_DIR} (code=exited, status=0/SUCCESS) Process: 1550 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1554 (xenconsoled) Tasks: 2 (limit: 9830) CGroup: /system.slice/xenconsoled.service └─1554 /usr/sbin/xenconsoled -i --log=none --log-dir=/var/log/xen/console ● xendomains.service - Xendomains - start and stop guests on boot and shutdown Loaded: loaded (/usr/lib/systemd/system/xendomains.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:57:06 PDT; 10min ago Process: 1936 ExecStart=/usr/lib/xen/bin/xendomains start (code=exited, status=0/SUCCESS) Process: 1933 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1936 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xendomains.service May 30 12:57:06 xen001.example.com xendomains[1936]: ..done ● xenstored.service - The Xen xenstore Loaded: loaded (/usr/lib/systemd/system/xenstored.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1419 ExecStart=/etc/xen/scripts/launch-xenstore (code=exited, status=0/SUCCESS) Main PID: 1419 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xenstored.service May 30 12:56:36 xen001 launch-xenstore[1419]: Starting /usr/lib/xen/boot/xenstore-stubdom.gz... -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Wed, 2018-05-30 at 11:09 -0700, PGNet Dev wrote:
Upgrading working OpenSUSE leap42.3 / Xen servers (5, atm) to leap15.0, after upgrade
Well,
But checking,
xl list Name ID Mem VCPUs State Time(s) (null) 0 4016 4 r----- 329.6 Xenstore 1 31 1 -b---- 0.0
and
[...]
systemctl status -l xenstored.service ● xenstored.service - The Xen xenstore Loaded: loaded (/usr/lib/systemd/system/xenstored.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Wed 2018-05-30 10:37:08 PDT; 10min ago Main PID: 1391 (code=exited, status=0/SUCCESS)
May 30 10:35:39 xserver launch-xenstore[1391]: Starting /usr/lib/xen/boot/xenstore-stubdom.gz...
This 'failed' here is likely why you see '(null)' instead of 'Domain- 0'. It looks like, for some reason, xenstored is not actually starting. In fact, in your other email, where you report that things work, this succeeded. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
This 'failed' here is likely why you see '(null)' instead of 'Domain- 0'. It looks like, for some reason, xenstored is not actually starting.
In fact, in your other email, where you report that things work, this succeeded.
sure. the fact that xendstored isn't launching is clear. 'why not?' is the question. the 'other' email refers to using my own, rather that distro's, qemu instances; that's the only change in env what the difference is remains to be determined. -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Thu, 2018-05-31 at 09:22 -0700, PGNet Dev wrote:
In fact, in your other email, where you report that things work, this succeeded.
sure. the fact that xendstored isn't launching is clear.
'why not?' is the question.
Right.
the 'other' email refers to using my own, rather that distro's, qemu instances; that's the only change in env
Well, I guess we have to look for info in logs and such. It is a bit weird that qemu has an influence in having xenstore inside the xenstore domain starting or working properly. Does things work (with the distro packages) if you use a daemon instead of the stubdomain? Does HVM guests, which requires qemu, work with such configuration? I can try to have a look at a Leap15 and see if things work for me, but it may take a bit... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
Am Wed, 30 May 2018 11:09:17 -0700 schrieb PGNet Dev <pgnet.dev@gmail.com>:
[ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating.
Something during system-upgrade went wrong. I think this symptom was seen during SLE15 beta. Since another upgrade of the involved packages fixed the overall system-state, there is little info left what exactly the failure state is. Thanks for the report. Olaf
On 6/4/18 2:44 AM, Dario Faggioli wrote:
Well, I guess we have to look for info in logs and such. It is a bit weird that qemu has an influence in having xenstore inside the xenstore domain starting or working properly.
I'd thought it should be independent as well. On 6/4/18 4:04 AM, Olaf Hering wrote:
[ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating.
Something during system-upgrade went wrong.
Not sure how you know that from that message.
I think this symptom was seen during SLE15 beta.
Can you point to a reference/post/bug-report by any chance?
Since another upgrade of the involved packages fixed the overall system-state, there is little info left what exactly the failure state is.
Interesting. As I hadn't tried the force-reinstall from the original Virtualization: repo, let's see. Dropping BACK to qemu from Virtualization: -- i.e., replacing *my* qemu packages -- and rebooting ... ... DOES, in fact, now work. Appears to have done it! Thx. -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
participants (3)
-
Dario Faggioli -
Olaf Hering -
PGNet Dev