Mailinglist Archive: opensuse-bugs (4250 mails)

< Previous Next >
[Bug 1018741] New: after Xen 4.7 -> 4.8 upgrade, Xen PVHVM/UEFI guests fail to boot; hang @~ OVMF
  • From: bugzilla_noreply@xxxxxxxxxx
  • Date: Sun, 08 Jan 2017 00:43:32 +0000
  • Message-id: <bug-1018741-21960@http.bugzilla.opensuse.org/>
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741


Bug ID: 1018741
Summary: after Xen 4.7 -> 4.8 upgrade, Xen PVHVM/UEFI guests
fail to boot; hang @~ OVMF
Classification: openSUSE
Product: openSUSE Distribution
Version: Leap 42.2
Hardware: x86-64
OS: openSUSE 42.2
Status: NEW
Severity: Critical
Priority: P5 - None
Component: Bootloader
Assignee: jsrain@xxxxxxxx
Reporter: lists@xxxxxxxxxxxx
QA Contact: jsrain@xxxxxxxx
Found By: ---
Blocker: ---

I was running

Opensuse 42.2
Xen 4.7
Kernel-Stable

on X86_64.

This PVHVM, UEFI DomU guest, running Opensuse 42.2 + Kernel-Stable (as well as
multiple others) launched/operated OK,

cat /etc/xen/auto/opensuse.cfg
name = 'opensuse'
builder = 'hvm'
xen_platform_pci = 1
device_model_version="qemu-xen"
bios = 'ovmf'
bios_override = '/usr/share/qemu/ovmf-x86_64.bin'

maxmem = 1024
memory = 1024

boot = 'cd'
disk = [ 'phy:/dev/VG0/EFI,xvda,w', 'phy:/dev/VG0/BOOT,xvde,w',
'phy:/dev/VG0/SWAP,xvdf,w', 'phy:/dev/VG0/ROOT,xvdg,w',]
vif = [ 'mac=00:16:3E:50:00:01, model=e1000, bridge=br0,
vifname=vifT',]

acpi = 1
apic = 1
hap = 1
localtime = 0
nestedhvm = 0
nx = 1
pae = 1
tsc_mode = 'default'

sdl = 0
serial = 'pty'
vga = 'stdvga'
vnc = 1
vncdisplay = 1
vnclisten = '0.0.0.0'

on_crash = 'destroy'
on_reboot = 'restart'
on_shutdown = 'destroy'

After upgrade from

Xen 4.7 -> 4.8

specifically, atm, these packages @ Dom0

grub2-2.02~beta3-352.1.x86_64
grub2-branding-upstream-2.02~beta3-352.1.x86_64
grub2-i386-pc-2.02~beta3-352.1.x86_64
grub2-x86_64-efi-2.02~beta3-352.1.x86_64
grub2-x86_64-xen-2.02~beta3-352.1.x86_64
kernel-default-4.9.1-2.2.g1072b39.x86_64
kernel-default-devel-4.9.1-2.2.g1072b39.x86_64
kernel-devel-4.9.1-2.1.g1072b39.noarch
kernel-firmware-20161130-36.2.noarch
kernel-macros-4.9.1-2.1.g1072b39.noarch
kernel-source-4.9.1-2.1.g1072b39.noarch
kernel-syms-4.9.1-2.1.g1072b39.x86_64
xen-4.8.0_01-466.1.x86_64
xen-libs-4.8.0_01-466.1.x86_64
xen-tools-4.8.0_01-466.1.x86_64

no guests boot any longer.




fwiw,

I chrooted into the Guest env from the DomU, rebuilt the initrd there and
re-exec'd grub2-mkconfig.

No change restarting the Guest - same errors as above.

This morning I woke to find that the Guest had been restarting itself all night
-- apparently 200+ times!

xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 2048 1 r----- 2907.0
opensuse 225 1024 1 -b---- 5.7

That's new -- & clearly not good, esp since in the config I have

on_crash = 'destroy'
on_reboot = 'restart'
on_shutdown = 'destroy'


With Dom0 Xen cmdline containing

loglvl=all guest_loglvl=all

from exec of

xl create -c opensuse.cfg

this error appears in console


������������������������������������������������������������������������������Ŀ
� ERROR

�
�

�
� Verification failed: (15) Access Denied

�
�

�
�

�
�

�
�

�
�

�
�

�
� ����Ŀ

�
� � OK �

�
� �����

�
�

�
�

�
�

�
�

�
�

�
�

�
�

�
�

�
�

�
�

�

��������������������������������������������������������������������������������


here's the @Dom0 output of `journalctl -f`

Jan 07 10:23:12 xent.lanint systemd-udevd[918]: seq 3220 queued, 'add'
'xen-backend'
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: Validate module index
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: Check if link configuration
needs reloading.
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: seq 3220 forked new worker
[2464]
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: seq 3220 running
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: IMPORT builtin 'hwdb'
/usr/lib/udev/rules.d/50-udev-default.rules:15
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: IMPORT builtin 'hwdb'
returned non-zero
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: RUN 'kmod load
$env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: Execute 'load'
'xen-backend:vbd'
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: Inserted 'xen_blkback'
Jan 07 10:23:12 xent.lanint root[2508]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51712
Jan 07 10:23:12 xent.lanint root[2511]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51808
Jan 07 10:23:12 xent.lanint root[2509]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51776
Jan 07 10:23:12 xent.lanint root[2510]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51792
Jan 07 10:23:13 xent.lanint kernel: tun: Universal TUN/TAP device driver,
1.6
Jan 07 10:23:14 xent.lanint kernel: tun: (C) 1999-2004 Max Krasnyansky
<maxk@xxxxxxxxxxxx>
Jan 07 10:23:14 xent.lanint systemd-sysctl[2870]: Parsing
/usr/lib/sysctl.d/50-coredump.conf
Jan 07 10:23:14 xent.lanint systemd-sysctl[2870]: Parsing
/usr/lib/sysctl.d/50-default.conf
Jan 07 10:23:14 xent.lanint systemd-sysctl[2870]: Parsing
/etc/sysctl.d/99-sysctl.conf
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vif1.0\x2demu.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-devices-virtual-net-vif1.0\x2demu.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-subsystem-net-devices-vif1.0\x2demu.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-devices-virtual-net-vif1.0\x2demu.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd-sysctl[2876]: Parsing
/usr/lib/sysctl.d/50-coredump.conf
Jan 07 10:23:14 xent.lanint systemd-sysctl[2876]: Parsing
/usr/lib/sysctl.d/50-default.conf
Jan 07 10:23:14 xent.lanint systemd-sysctl[2876]: Parsing
/etc/sysctl.d/99-sysctl.conf
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vif1.0.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-devices-vif\x2d1\x2d0-net-vif1.0.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-subsystem-net-devices-vif1.0.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-devices-vif\x2d1\x2d0-net-vif1.0.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint kernel: vif vif-1-0 vifT: renamed from vif1.0
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vif1.0.device: Dev
sys-subsystem-net-devices-vif1.0.device appeared twice with different sysfs
paths /sys/devices/vif-1-0/net/vif1.0 and /sys/devices/vif-1-0/net/vifT
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vifT.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[2028]:
sys-devices-vif\x2d1\x2d0-net-vifT.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-subsystem-net-devices-vif1.0.device: Dev
sys-subsystem-net-devices-vif1.0.device appeared twice with different sysfs
paths /sys/devices/vif-1-0/net/vif1.0 and /sys/devices/vif-1-0/net/vifT
Jan 07 10:23:14 xent.lanint systemd[1]:
sys-subsystem-net-devices-vifT.device: Changed dead -> plugged
Jan 07 10:23:14 xent.lanint kernel: br0: port 2(vifT) entered blocking
state
Jan 07 10:23:15 xent.lanint kernel: br0: port 2(vifT) entered disabled
state
Jan 07 10:23:15 xent.lanint kernel: device vifT entered promiscuous mode
Jan 07 10:23:15 xent.lanint kernel: IPv6: ADDRCONF(NETDEV_UP): vifT: link
is not ready
Jan 07 10:23:15 xent.lanint kernel: vifT-emu: renamed from vif1.0-emu
Jan 07 10:23:15 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vif1.0\x2demu.device: Dev
sys-subsystem-net-devices-vif1.0\x2demu.device appeared twice with different
sysfs paths /sys/devices/virtual/net/vif1.0-emu and
/sys/devices/virtual/net/vifT-emu
Jan 07 10:23:15 xent.lanint systemd[2028]:
sys-subsystem-net-devices-vifT\x2demu.device: Changed dead -> plugged
Jan 07 10:23:15 xent.lanint systemd[2028]:
sys-devices-virtual-net-vifT\x2demu.device: Changed dead -> plugged
Jan 07 10:23:15 xent.lanint kernel: br0: port 3(vifT-emu) entered blocking
state
Jan 07 10:23:15 xent.lanint kernel: br0: port 3(vifT-emu) entered disabled
state
Jan 07 10:23:15 xent.lanint kernel: device vifT-emu entered promiscuous
mode
Jan 07 10:23:15 xent.lanint kernel: br0: port 3(vifT-emu) entered blocking
state
Jan 07 10:23:15 xent.lanint kernel: br0: port 3(vifT-emu) entered listening
state
Jan 07 10:23:12 xent.lanint root[2649]: /etc/xen/scripts/block: Writing
backend/vbd/1/51808/physical-device fe:10 to xenstore.
Jan 07 10:23:12 xent.lanint root[2651]: /etc/xen/scripts/block: Writing
backend/vbd/1/51808/physical-device-path /dev/dm-16 to xenstore.
Jan 07 10:23:12 xent.lanint root[2653]: /etc/xen/scripts/block: Writing
backend/vbd/1/51808/hotplug-status connected to xenstore.
Jan 07 10:23:12 xent.lanint root[2719]: /etc/xen/scripts/block: Writing
backend/vbd/1/51712/physical-device fe:d to xenstore.
Jan 07 10:23:12 xent.lanint root[2721]: /etc/xen/scripts/block: Writing
backend/vbd/1/51712/physical-device-path /dev/dm-13 to xenstore.
Jan 07 10:23:12 xent.lanint root[2723]: /etc/xen/scripts/block: Writing
backend/vbd/1/51712/hotplug-status connected to xenstore.
Jan 07 10:23:13 xent.lanint root[2787]: /etc/xen/scripts/block: Writing
backend/vbd/1/51776/physical-device fe:e to xenstore.
Jan 07 10:23:13 xent.lanint root[2789]: /etc/xen/scripts/block: Writing
backend/vbd/1/51776/physical-device-path /dev/dm-14 to xenstore.
Jan 07 10:23:13 xent.lanint root[2791]: /etc/xen/scripts/block: Writing
backend/vbd/1/51776/hotplug-status connected to xenstore.
Jan 07 10:23:13 xent.lanint root[2853]: /etc/xen/scripts/block: Writing
backend/vbd/1/51792/physical-device fe:f to xenstore.
Jan 07 10:23:13 xent.lanint root[2855]: /etc/xen/scripts/block: Writing
backend/vbd/1/51792/physical-device-path /dev/dm-15 to xenstore.
Jan 07 10:23:13 xent.lanint root[2857]: /etc/xen/scripts/block: Writing
backend/vbd/1/51792/hotplug-status connected to xenstore.
Jan 07 10:23:13 xent.lanint root[2888]: /etc/xen/scripts/vif-bridge: online
type_if=vif XENBUS_PATH=backend/vif/1/0
Jan 07 10:23:13 xent.lanint root[2916]: /etc/xen/scripts/vif-bridge:
Successful vif-bridge online for vifT, bridge br0.
Jan 07 10:23:13 xent.lanint root[2917]: /etc/xen/scripts/vif-bridge:
Writing backend/vif/1/0/hotplug-status connected to xenstore.
Jan 07 10:23:13 xent.lanint root[2929]: /etc/xen/scripts/vif-bridge: add
type_if=tap XENBUS_PATH=backend/vif/1/0
Jan 07 10:23:13 xent.lanint root[2957]: /etc/xen/scripts/vif-bridge:
Successful vif-bridge add for vifT-emu, bridge br0.
Jan 07 10:23:16 xent.lanint kernel: xen-blkback: backend/vbd/1/51712: using
1 queues, protocol 1 (x86_64-abi)
Jan 07 10:23:16 xent.lanint kernel: xen-blkback: backend/vbd/1/51776: using
1 queues, protocol 1 (x86_64-abi)
Jan 07 10:23:16 xent.lanint kernel: xen-blkback: backend/vbd/1/51792: using
1 queues, protocol 1 (x86_64-abi)
Jan 07 10:23:16 xent.lanint kernel: xen-blkback: backend/vbd/1/51808: using
1 queues, protocol 1 (x86_64-abi)
Jan 07 10:23:17 xent.lanint kernel: br0: port 3(vifT-emu) entered learning
state
Jan 07 10:23:22 xent.lanint kernel: br0: port 3(vifT-emu) entered
forwarding state
Jan 07 10:23:22 xent.lanint kernel: br0: topology change detected,
propagating

and here's the output @ Dom0 serial console

(XEN) [2017-01-07 18:30:27] HVM3 save: CPU
(XEN) [2017-01-07 18:30:27] HVM3 save: PIC
(XEN) [2017-01-07 18:30:27] HVM3 save: IOAPIC
(XEN) [2017-01-07 18:30:27] HVM3 save: LAPIC
(XEN) [2017-01-07 18:30:27] HVM3 save: LAPIC_REGS
(XEN) [2017-01-07 18:30:27] HVM3 save: PCI_IRQ
(XEN) [2017-01-07 18:30:27] HVM3 save: ISA_IRQ
(XEN) [2017-01-07 18:30:27] HVM3 save: PCI_LINK
(XEN) [2017-01-07 18:30:27] HVM3 save: PIT
(XEN) [2017-01-07 18:30:27] HVM3 save: RTC
(XEN) [2017-01-07 18:30:27] HVM3 save: HPET
(XEN) [2017-01-07 18:30:27] HVM3 save: PMTIMER
(XEN) [2017-01-07 18:30:27] HVM3 save: MTRR
(XEN) [2017-01-07 18:30:27] HVM3 save: VIRIDIAN_DOMAIN
(XEN) [2017-01-07 18:30:27] HVM3 save: CPU_XSAVE
(XEN) [2017-01-07 18:30:27] HVM3 save: VIRIDIAN_VCPU
(XEN) [2017-01-07 18:30:27] HVM3 save: VMCE_VCPU
(XEN) [2017-01-07 18:30:27] HVM3 save: TSC_ADJUST
(XEN) [2017-01-07 18:30:27] HVM3 restore: CPU 0
(d3) [2017-01-07 18:30:28] HVM Loader
(d3) [2017-01-07 18:30:28] Detected Xen v4.8.0_01-466
(d3) [2017-01-07 18:30:28] Xenbus rings @0xfeffc000, event channel 1
(d3) [2017-01-07 18:30:28] System requested OVMF
(d3) [2017-01-07 18:30:28] CPU speed is 3093 MHz
(d3) [2017-01-07 18:30:28] Relocating guest memory for lowmem MMIO space
disabled
(d3) [2017-01-07 18:30:28] PCI-ISA link 0 routed to IRQ5
(d3) [2017-01-07 18:30:28] PCI-ISA link 1 routed to IRQ10
(d3) [2017-01-07 18:30:28] PCI-ISA link 2 routed to IRQ11
(d3) [2017-01-07 18:30:28] PCI-ISA link 3 routed to IRQ5
(d3) [2017-01-07 18:30:28] pci dev 01:3 INTA->IRQ10
(d3) [2017-01-07 18:30:28] pci dev 02:0 INTA->IRQ11
(d3) [2017-01-07 18:30:28] pci dev 04:0 INTA->IRQ5
(d3) [2017-01-07 18:30:28] No RAM in high memory; setting high_mem resource
base to 100000000
(d3) [2017-01-07 18:30:28] pci dev 02:0 bar 14 size 001000000: 0f0000008
(d3) [2017-01-07 18:30:28] pci dev 03:0 bar 10 size 001000000: 0f1000008
(d3) [2017-01-07 18:30:28] pci dev 04:0 bar 30 size 000040000: 0f2000000
(d3) [2017-01-07 18:30:29] pci dev 04:0 bar 10 size 000020000: 0f2040000
(d3) [2017-01-07 18:30:29] pci dev 03:0 bar 30 size 000010000: 0f2060000
(d3) [2017-01-07 18:30:29] pci dev 03:0 bar 18 size 000001000: 0f2070000
(d3) [2017-01-07 18:30:29] pci dev 02:0 bar 10 size 000000100: 00000c001
(d3) [2017-01-07 18:30:29] pci dev 04:0 bar 14 size 000000040: 00000c101
(d3) [2017-01-07 18:30:29] pci dev 01:1 bar 20 size 000000010: 00000c141
(d3) [2017-01-07 18:30:29] Multiprocessor initialisation:
(d3) [2017-01-07 18:30:29] - CPU0 ... 39-bit phys ... fixed MTRRs ... var
MTRRs [1/8] ... do
ne.
(d3) [2017-01-07 18:30:29] Writing SMBIOS tables ...
(d3) [2017-01-07 18:30:29] Loading OVMF ...
(XEN) [2017-01-07 18:30:29] d3v0 Over-allocation for domain 3: 524545 >
524544
(d3) [2017-01-07 18:30:29] Loading ACPI ...
(d3) [2017-01-07 18:30:29] vm86 TSS at fc00a400
(d3) [2017-01-07 18:30:29] BIOS map:
(d3) [2017-01-07 18:30:29] ffe00000-ffffffff: Main BIOS
(d3) [2017-01-07 18:30:29] E820 table:
(d3) [2017-01-07 18:30:29] [00]: 00000000:00000000 - 00000000:000a0000:
RAM
(d3) [2017-01-07 18:30:29] HOLE: 00000000:000a0000 - 00000000:000f0000
(d3) [2017-01-07 18:30:29] [01]: 00000000:000f0000 - 00000000:00100000:
RESERVED
(d3) [2017-01-07 18:30:29] [02]: 00000000:00100000 - 00000000:7eeb5000:
RAM
(d3) [2017-01-07 18:30:29] HOLE: 00000000:7eeb5000 - 00000000:fc000000
(d3) [2017-01-07 18:30:29] [03]: 00000000:fc000000 - 00000001:00000000:
RESERVED
(d3) [2017-01-07 18:30:29] Invoking OVMF ...

it stops there.


From inside the chrooted DomU guest, re-installing grub, without overwriting
the physical, motherboard efi-related nvram,

grub2-install -v --target=x86_64-efi --efi-directory=/boot/efi --no-nvram
...
grub2-install: info: copying `/boot/grub2/x86_64-efi/core.efi' ->
`/boot/efi/EFI/opensuse/grubx64.efi'.
Installation finished. No error reported.

Ensuring the startup boots the right loader

echo "fs0:\EFI\opensuse\grubx64.efi" > /boot/efi/startup.nsh

What's installed at this point is

tree -D /boot/efi
/boot/efi
├── [root 512 Jun 21 2016] EFI
│ ├── [root 512 Jan 7 9:47] boot
│ │ ├── [root 1164376 Jan 7 9:47] bootx64.efi
│ │ ├── [root 72240 Jan 4 6:37] fallback.efi
│ │ ├── [root 120 Jan 7 9:47] grub.cfg
│ │ ├── [root 1008720 Jan 7 9:47] grub.efi
│ │ └── [root 1166552 Jan 7 9:47]
MokManager.efi
│ └── [root 512 Jun 21 2016] opensuse
│ ├── [root 58 Jan 4 6:37] boot.csv
│ ├── [root 120 Jan 4 6:37] grub.cfg
│ ├── [root 1008720 Jan 4 6:37] grub.efi
│ ├── [root 129024 Jan 7 13:53] grubx64.efi
│ ├── [root 1166552 Jan 4 6:37] MokManager.efi
│ └── [root 1164376 Jan 4 6:37] shim.efi
└── [root 30 Jan 7 13:53] startup.nsh

exit the chroot, then create the DomU

xl create -c /etc/xen/vm/opensuse.cfg

It still fails, 1st providing

Trust openSUSE Certificate �
�

�
� Do you agree to use the built-in openSUSE certificate

�
� to verify boot loaders and kernels?

�
�

�
� �����Ŀ

�
� � No �

�
� � Yes �

�1
� �������


If I select NO, I get the same error,

������������������������������������������������������������������������������Ŀ
� ERROR

�
�

�
� Verification failed: (15) Access Denied

�
�

�
� ����Ŀ

�
� � OK �

�
� �����


If I select yes, it returns


������������������������������������������������������������������������������Ŀ
� ERROR

�
�

�
� Could not install security protocol: (2) Invalid Parameter

�
�

�
� ����Ŀ

�
� � OK �

�
� �����

�

then OK, and this again.


������������������������������������������������������������������������������Ŀ
� ERROR

�
�

�
� Verification failed: (15) Access Denied

�
�

�
�

�
�

�
�

�
�

�
�

�
� ����Ŀ

�
� � OK �

�
� �����



After a re-grub + reboot, then FAIL, if I destroy the guest again

xl destroy opensuse

re-enter chroot

I now note the creation/addition of

tree -D /boot/efi
/boot/efi
...
├── [root 12673 Jan 7 13:55] NvVars
...

Checking that

strings /boot/efi/NvVars | head 10
Washington1
Redmond1
Microsoft Corporation1;09
2Microsoft Corporation Third Party Marketplace Root0
110624204129Z
260624205129Z0
Washington1
Redmond1
Microsoft Corporation1*0(
!Microsoft Corporation KEK CA 20110

Clearly the Guest's BIOS (tianocore/ovmf) at least 'touches' the EFI partition
-- i.e. it's aware of the Guest.

Why SecureBoot is involved is unclear.

Is there some new Xen mechanism, or config param, to ensure it's DISABLED for
Guests ?

--
You are receiving this mail because:
You are on the CC list for the bug.
< Previous Next >