[opensuse-virtual] after leap 42.3 -> 15.0 upgrades, Xen/EFI servers' multiple services fail with "failed with result 'dependency'"
Upgrading working OpenSUSE leap42.3 / Xen servers (5, atm) to leap15.0, after upgrade lsb_release -rd Description: openSUSE Leap 15.0 Release: 15.0 uname -rm 4.16.12-lp150.2.g39c7522-default x86_64 rpm -qa | egrep -i "xen|qemu|efi" | sort efibootmgr-14-lp150.2.2.x86_64 efivar-devel-31-lp150.1.11.x86_64 grub2-x86_64-efi-2.02-lp150.35.1.x86_64 grub2-x86_64-xen-2.02-lp150.35.1.x86_64 libefivar1-31-lp150.1.11.x86_64 qemu-2.12.0-lp150.453.1.x86_64 qemu-arm-2.12.0-lp150.453.1.x86_64 qemu-block-curl-2.12.0-lp150.453.1.x86_64 qemu-block-dmg-2.12.0-lp150.453.1.x86_64 qemu-extra-2.12.0-lp150.453.1.x86_64 qemu-ipxe-1.0.0-lp150.453.1.noarch qemu-ksm-2.12.0-lp150.453.1.x86_64 qemu-kvm-2.12.0-lp150.453.1.x86_64 qemu-lang-2.12.0-lp150.453.1.x86_64 qemu-linux-user-2.12.0-lp150.453.1.x86_64 qemu-ovmf-x86_64-2018+git1525854636.13e3f8c03339-lp150.75.1.noarch qemu-seabios-1.11.1-lp150.453.1.noarch qemu-sgabios-8-lp150.453.1.noarch qemu-tools-2.12.0-lp150.453.1.x86_64 qemu-vgabios-1.11.1-lp150.453.1.noarch qemu-x86-2.12.0-lp150.453.1.x86_64 xen-4.10.1_02-lp150.576.1.x86_64 xen-libs-4.10.1_02-lp150.576.1.x86_64 xen-tools-4.10.1_02-lp150.576.1.x86_64 No notable version changes, just the switch to lp150 packages. serial output @ boot ... Booting `OpenSUSE, with Xen hypervisor'Booting `OpenSUSE, with Xen hypervisor' Loading Xen 4.10.1_02-lp150.576 with Linux 4.16.12-lp150.2.g39c7522-default ...Loading Xen 4.10.1_02-lp150.576 with Linux 4.16.12-lp150.2.g39c7522-default ... /EndEntire /EndEntire file path: file path: /ACPI(a0341d0,0)/ACPI(a0341d0,0)/PCI(1,1c)/PCI(1,1c)/PCI(0,0)/PCI(0,0)/PCI(0,1)/PCI(0,1)/PCI(0,0)/PCI(0,0)/HardwareVendor (cf31fac5-c24e-11d2-85f3-00a0c93ec93b)[1: /HardwareVendor(cf31fac5-c24e-11d2-85f3-00a0c93ec93b)[1: 88 88 ]]/HD(2,1000,96000,3d7310980ce0754c ,2,2)/HD(2,1000,96000,3d7310980ce0754c,2,2)/File(\efi\opensuse) /File(\efi\opensuse)/File(xen-4.10.1_02-lp150.576.efi)/File(xen-4.10.1_02-lp150.576.efi)/EndEntire /EndEntire Xen 4.10.1_02-lp150.576 (c/s ) EFI loader Using configuration file 'xen-4.10.1_02-lp150.576.cfg' vmlinuz-4.16.12-lp150.2.g39c7522-default: 0x000000008b917000-0x000000008c073da8 initrd-4.16.12-lp150.2.g39c7522-default: 0x000000008a856000-0x000000008b916e60 0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0x928b9018 0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0x928b0018 0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0x92894018 __ __ \ \/ /___ _ __ \ // _ \ '_ \ / \ __/ | | | /_/\_\___|_| |_| _ _ _ ___ _ ___ ____ _ _ ____ ___ ____ _____ __ | || | / |/ _ \ / | / _ \___ \ | |_ __ / | ___| / _ \ | ___|___ / /_ | || |_ | | | | || | | | | |__) |__| | '_ \| |___ \| | | ||___ \ / / '_ \ |__ _|| | |_| || | | |_| / __/|__| | |_) | |___) | |_| | ___) |/ /| (_) | |_|(_)_|\___(_)_|___\___/_____| |_| .__/|_|____/ \___(_)____//_/ \___/ |_____| |_| (XEN) Xen version 4.10.1_02-lp150.576 (abuild@suse.de) (gcc (SUSE Linux) 8.1.1 20180523 [gcc-8-branch revision 260570]) debug=n Fri May 18 14:03:46 UTC 2018 (XEN) Latest ChangeSet: (XEN) Console output is synchronous. (XEN) Bootloader: EFI (XEN) Command line: dom0_mem=4016M,max:4096M bootscrub=false dom0_max_vcpus=4 vga=gfx-1920x1080x16 com1=115200,8n1,pci console=com1,vga console_timestamps console_to_ring conring_size=64 sc hed=credit2 reboot=acpi log_buf_len=16M loglvl=all guest_loglvl=all noreboot=false sync_console=true sched_debug iommu=verbose apic_verbosity=verbose ... dmesg ... [ 0.000000] Hypervisor detected: Xen PV ... But checking, xl list Name ID Mem VCPUs State Time(s) (null) 0 4016 4 r----- 329.6 Xenstore 1 31 1 -b---- 0.0 and dmesg | grep -i xen [ 0.000000] Command line: root=/dev/mapper/VG0 softlevel=xen rd.shell rd.debug=0 rd.udev.log_priority=info rd.auto=1 dolvm lvmwait=/dev/mapper/VG0 root=/dev/mapper/VG0 rootfstype=ext4 rootflags=journal_checksum noresume video=vesa:off video=efifb:1024x768 video=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=hvc0 elevator=deadline cpuidle cpufreq=xen:ondemand clocksource=xen net.ifnames=1 biosdevname=0 plymouth.enable=0 apparmor=0 pcie_aspm=off mce=off fsck.mode=skip fsck.repair=preen nomodeset nouveau.modeset=1 showopts noquiet log_buf_len=10M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=xen,keep [ 0.000000] Xen: [mem 0x0000000000000000-0x0000000000007fff] reserved ... [ 0.000000] Xen: [mem 0x000000016f2a8000-0x000000085dffffff] unusable [ 0.000000] Hypervisor detected: Xen PV [ 0.000000] Kernel/User page tables isolation: disabled on XEN PV. [ 0.000000] Setting APIC routing to Xen PV. [ 0.000000] Booting paravirtualized kernel on Xen [ 0.000000] Xen version: 4.10.1_02-lp150.576 (preserve-AD) [ 0.000000] xen: PV spinlocks enabled [ 0.000000] Kernel command line: root=/dev/mapper/VG0 softlevel=xen rd.shell rd.debug=0 rd.udev.log_priority=info rd.auto=1 dolvm lvmwait=/dev/mapper/VG0 root=/dev/mapper/VG0 rootfstype=ext4 rootflags=journal_checksum noresume video=vesa:off video=efifb:1024x768 video=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=hvc0 elevator=deadline cpuidle cpufreq=xen:ondemand clocksource=xen net.ifnames=1 biosdevname=0 plymouth.enable=0 apparmor=0 pcie_aspm=off mce=off fsck.mode=skip fsck.repair=preen nomodeset nouveau.modeset=1 showopts noquiet log_buf_len=10M print_fatal_signals=1 systemd.log_level=info systemd.log_target=kmsg earlyprintk=xen,keep [ 0.004000] xen:events: Using FIFO-based ABI [ 0.004000] xen: --> pirq=1 -> irq=1 (gsi=1) ... [ 0.004000] xen: --> pirq=15 -> irq=15 (gsi=15) [ 0.004000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.004000] Xen: using vcpuop timer interface [ 0.004000] installing Xen timer for CPU 0 [ 0.192039] installing Xen timer for CPU 1 [ 0.232038] installing Xen timer for CPU 2 [ 0.272060] installing Xen timer for CPU 3 [ 0.360014] xen:grant_table: Grant tables using version 1 layout [ 0.464199] xen: registering gsi 9 triggering 0 polarity 0 ... [ 21.864462] xen: registering gsi 17 triggering 0 polarity 1 [ 27.472745] xen:xen_evtchn: Event-channel device installed [ 27.753699] xen_pciback: backend is vpci [ 27.820997] xen_acpi_processor: Uploading Xen processor PM info [ 59.589179] xen: registering gsi 19 triggering 0 polarity 1 [ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating. [ 117.961491] systemd[1]: Failed to start The Xen xenstore. [ 117.961541] systemd[1]: Dependency failed for Xendomains - start and stop guests on boot and shutdown. [ 117.961563] systemd[1]: xendomains.service: Job xendomains.service/start failed with result 'dependency'. [ 117.961577] systemd[1]: Dependency failed for qemu for xen dom0 disk backend. [ 117.961590] systemd[1]: Dependency failed for xencommons. [ 117.961605] systemd[1]: xencommons.service: Job xencommons.service/start failed with result 'dependency'. [ 117.961624] systemd[1]: xen-qemu-dom0-disk-backend.service: Job xen-qemu-dom0-disk-backend.service/start failed with result 'dependency'. [ 117.961628] systemd[1]: Dependency failed for Xenconsoled - handles logging from guest consoles and hypervisor. [ 117.961643] systemd[1]: xenconsoled.service: Job xenconsoled.service/start failed with result 'dependency'. systemctl status -l xen-qemu-dom0-disk-backend.service ● xen-qemu-dom0-disk-backend.service - qemu for xen dom0 disk backend Loaded: loaded (/usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service; enabled; vendor preset: disabled) Active: inactive (dead) systemctl list-unit-files | grep -i xen home-xen.mount generated proc-xen.mount static var-lib-xenstored.mount static xen-dom0-modules.service disabled xen-init-dom0.service enabled xen-qemu-dom0-disk-backend.service enabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service enabled xendomains.service enabled xenstored.service enabled cat /usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service [Unit] Description=qemu for xen dom0 disk backend Requires=proc-xen.mount xenstored.service After=proc-xen.mount xenstored.service xenconsoled.service Before=xendomains.service libvirtd.service libvirt-guests.service RefuseManualStop=true ConditionPathExists=/proc/xen/capabilities [Service] Type=simple PIDFile=/var/run/xen/qemu-dom0.pid ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities ExecStartPre=/bin/mkdir -p /var/run/xen ExecStart=/usr/bin/qemu-system-i386 -xen-domid 0 \ -xen-attach -name dom0 -nographic -M xenpv -daemonize \ -monitor /dev/null -serial /dev/null -parallel /dev/null \ -nodefaults -no-user-config \ -pidfile /var/run/xen/qemu-dom0.pid [Install] WantedBy=multi-user.target systemctl status -l xenstored.service ● xenstored.service - The Xen xenstore Loaded: loaded (/usr/lib/systemd/system/xenstored.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Wed 2018-05-30 10:37:08 PDT; 10min ago Main PID: 1391 (code=exited, status=0/SUCCESS) May 30 10:35:39 xserver launch-xenstore[1391]: Starting /usr/lib/xen/boot/xenstore-stubdom.gz... Not sure yet -- is this a new bug, or a config change, issue? -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
switching to my own repo's builds of qemu (which I haven't used in production on 42.3 for ages), in preference over d.o.o's Virtualization repo, cat LOC-qemu.repo [LOC-qemu] enabled=1 baseurl=https://download.opensuse.org/repositories/home:/pgnd:/Virtualization:/qemu/... priority=11 ... cat Virtualization.repo [Virtualization] enabled=1 priority=12 baseurl=https://download.opensuse.org/repositories/Virtualization/openSUSE_Leap_15.0 ... zypper dup --from LOC-qemu ... now, on reboot, all looks good dmesg | grep -i xen | egrep -i "fail" (empty) xl list Name ID Mem VCPUs State Time(s) Domain-0 0 4016 4 r----- 548.1 Xenstore 1 31 1 -b---- 0.0 systemctl status -l `systemctl list-unit-files | grep -i xen | grep enabled | awk '{print $1}'` ● xen-init-dom0.service - xen-init-dom0, initialise Dom0 configuration (xenstore nodes, JSON configuration stub) Loaded: loaded (/usr/lib/systemd/system/xen-init-dom0.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1552 ExecStart=/usr/lib/xen/bin/xen-init-dom0 (code=exited, status=0/SUCCESS) Process: 1549 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1552 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xen-init-dom0.service May 30 12:56:36 xen001 xen-init-dom0[1552]: Done setting up Dom0 ● xen-qemu-dom0-disk-backend.service - qemu for xen dom0 disk backend Loaded: loaded (/usr/lib/systemd/system/xen-qemu-dom0-disk-backend.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1556 ExecStartPre=/bin/mkdir -p /var/run/xen (code=exited, status=0/SUCCESS) Process: 1555 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1763 (qemu-system-i38) Tasks: 4 (limit: 9830) CGroup: /system.slice/xen-qemu-dom0-disk-backend.service └─1763 /usr/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null -parallel /dev/null -nodefaults -no-user> ● xencommons.service - xencommons Loaded: loaded (/usr/lib/systemd/system/xencommons.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2018-05-30 12:57:05 PDT; 10min ago Process: 1923 ExecStartPost=/bin/sh -c mv -vf /var/log/xen/xen-boot.log /var/log/xen/xen-boot.prev.log ; /usr/sbin/xl dmesg > /var/log/xen/xen-boot.log (code=exited, status=0/SUCCESS) Process: 1918 ExecStart=/usr/bin/xenstore-ls -f (code=exited, status=0/SUCCESS) Process: 1917 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1918 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xencommons.service May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/backends/vfb = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/backends/qnic = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/0/device-model/0/state = "running" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1 = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/domid = "1" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/name = "Xenstore" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory = "" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory/target = "32768" May 30 12:57:05 xen001.example.com xenstore-ls[1918]: /local/domain/1/memory/static-max = "331776" May 30 12:57:05 xen001.example.com sh[1923]: renamed '/var/log/xen/xen-boot.log' -> '/var/log/xen/xen-boot.prev.log' ● xenconsoled.service - Xenconsoled - handles logging from guest consoles and hypervisor Loaded: loaded (/usr/lib/systemd/system/xenconsoled.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1553 ExecStartPre=/bin/mkdir -p ${XENCONSOLED_LOG_DIR} (code=exited, status=0/SUCCESS) Process: 1550 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1554 (xenconsoled) Tasks: 2 (limit: 9830) CGroup: /system.slice/xenconsoled.service └─1554 /usr/sbin/xenconsoled -i --log=none --log-dir=/var/log/xen/console ● xendomains.service - Xendomains - start and stop guests on boot and shutdown Loaded: loaded (/usr/lib/systemd/system/xendomains.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:57:06 PDT; 10min ago Process: 1936 ExecStart=/usr/lib/xen/bin/xendomains start (code=exited, status=0/SUCCESS) Process: 1933 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS) Main PID: 1936 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xendomains.service May 30 12:57:06 xen001.example.com xendomains[1936]: ..done ● xenstored.service - The Xen xenstore Loaded: loaded (/usr/lib/systemd/system/xenstored.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2018-05-30 12:56:36 PDT; 10min ago Process: 1419 ExecStart=/etc/xen/scripts/launch-xenstore (code=exited, status=0/SUCCESS) Main PID: 1419 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 9830) CGroup: /system.slice/xenstored.service May 30 12:56:36 xen001 launch-xenstore[1419]: Starting /usr/lib/xen/boot/xenstore-stubdom.gz... -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Wed, 2018-05-30 at 11:09 -0700, PGNet Dev wrote:
Upgrading working OpenSUSE leap42.3 / Xen servers (5, atm) to leap15.0, after upgrade
Well,
This 'failed' here is likely why you see '(null)' instead of 'Domain- 0'. It looks like, for some reason, xenstored is not actually starting. In fact, in your other email, where you report that things work, this succeeded. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
sure. the fact that xendstored isn't launching is clear. 'why not?' is the question. the 'other' email refers to using my own, rather that distro's, qemu instances; that's the only change in env what the difference is remains to be determined. -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Thu, 2018-05-31 at 09:22 -0700, PGNet Dev wrote:
Right.
the 'other' email refers to using my own, rather that distro's, qemu instances; that's the only change in env
Well, I guess we have to look for info in logs and such. It is a bit weird that qemu has an influence in having xenstore inside the xenstore domain starting or working properly. Does things work (with the distro packages) if you use a daemon instead of the stubdomain? Does HVM guests, which requires qemu, work with such configuration? I can try to have a look at a Leap15 and see if things work for me, but it may take a bit... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Software Engineer @ SUSE https://www.suse.com/
Am Wed, 30 May 2018 11:09:17 -0700 schrieb PGNet Dev <pgnet.dev@gmail.com>:
[ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating.
Something during system-upgrade went wrong. I think this symptom was seen during SLE15 beta. Since another upgrade of the involved packages fixed the overall system-state, there is little info left what exactly the failure state is. Thanks for the report. Olaf
On 6/4/18 2:44 AM, Dario Faggioli wrote:
I'd thought it should be independent as well. On 6/4/18 4:04 AM, Olaf Hering wrote:
[ 117.961068] systemd[1]: xenstored.service: Start operation timed out. Terminating.
Something during system-upgrade went wrong.
Not sure how you know that from that message.
I think this symptom was seen during SLE15 beta.
Can you point to a reference/post/bug-report by any chance?
Since another upgrade of the involved packages fixed the overall system-state, there is little info left what exactly the failure state is.
Interesting. As I hadn't tried the force-reinstall from the original Virtualization: repo, let's see. Dropping BACK to qemu from Virtualization: -- i.e., replacing *my* qemu packages -- and rebooting ... ... DOES, in fact, now work. Appears to have done it! Thx. -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
participants (3)
-
Dario Faggioli -
Olaf Hering -
PGNet Dev