[Bug 1018741] New: after Xen 4.7 -> 4.8 upgrade, Xen PVHVM/UEFI guests fail to boot; hang @~ OVMF
From inside the chrooted DomU guest, re-installing grub, without overwriting
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
Bug ID: 1018741
Summary: after Xen 4.7 -> 4.8 upgrade, Xen PVHVM/UEFI guests
fail to boot; hang @~ OVMF
Classification: openSUSE
Product: openSUSE Distribution
Version: Leap 42.2
Hardware: x86-64
OS: openSUSE 42.2
Status: NEW
Severity: Critical
Priority: P5 - None
Component: Bootloader
Assignee: jsrain@suse.com
Reporter: lists@ssl-mail.com
QA Contact: jsrain@suse.com
Found By: ---
Blocker: ---
I was running
Opensuse 42.2
Xen 4.7
Kernel-Stable
on X86_64.
This PVHVM, UEFI DomU guest, running Opensuse 42.2 + Kernel-Stable (as well as
multiple others) launched/operated OK,
cat /etc/xen/auto/opensuse.cfg
name = 'opensuse'
builder = 'hvm'
xen_platform_pci = 1
device_model_version="qemu-xen"
bios = 'ovmf'
bios_override = '/usr/share/qemu/ovmf-x86_64.bin'
maxmem = 1024
memory = 1024
boot = 'cd'
disk = [ 'phy:/dev/VG0/EFI,xvda,w', 'phy:/dev/VG0/BOOT,xvde,w',
'phy:/dev/VG0/SWAP,xvdf,w', 'phy:/dev/VG0/ROOT,xvdg,w',]
vif = [ 'mac=00:16:3E:50:00:01, model=e1000, bridge=br0,
vifname=vifT',]
acpi = 1
apic = 1
hap = 1
localtime = 0
nestedhvm = 0
nx = 1
pae = 1
tsc_mode = 'default'
sdl = 0
serial = 'pty'
vga = 'stdvga'
vnc = 1
vncdisplay = 1
vnclisten = '0.0.0.0'
on_crash = 'destroy'
on_reboot = 'restart'
on_shutdown = 'destroy'
After upgrade from
Xen 4.7 -> 4.8
specifically, atm, these packages @ Dom0
grub2-2.02~beta3-352.1.x86_64
grub2-branding-upstream-2.02~beta3-352.1.x86_64
grub2-i386-pc-2.02~beta3-352.1.x86_64
grub2-x86_64-efi-2.02~beta3-352.1.x86_64
grub2-x86_64-xen-2.02~beta3-352.1.x86_64
kernel-default-4.9.1-2.2.g1072b39.x86_64
kernel-default-devel-4.9.1-2.2.g1072b39.x86_64
kernel-devel-4.9.1-2.1.g1072b39.noarch
kernel-firmware-20161130-36.2.noarch
kernel-macros-4.9.1-2.1.g1072b39.noarch
kernel-source-4.9.1-2.1.g1072b39.noarch
kernel-syms-4.9.1-2.1.g1072b39.x86_64
xen-4.8.0_01-466.1.x86_64
xen-libs-4.8.0_01-466.1.x86_64
xen-tools-4.8.0_01-466.1.x86_64
no guests boot any longer.
fwiw,
I chrooted into the Guest env from the DomU, rebuilt the initrd there and
re-exec'd grub2-mkconfig.
No change restarting the Guest - same errors as above.
This morning I woke to find that the Guest had been restarting itself all night
-- apparently 200+ times!
xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 2048 1 r----- 2907.0
opensuse 225 1024 1 -b---- 5.7
That's new -- & clearly not good, esp since in the config I have
on_crash = 'destroy'
on_reboot = 'restart'
on_shutdown = 'destroy'
With Dom0 Xen cmdline containing
loglvl=all guest_loglvl=all
from exec of
xl create -c opensuse.cfg
this error appears in console
������������������������������������������������������������������������������Ŀ
� ERROR
�
�
�
� Verification failed: (15) Access Denied
�
�
�
�
�
�
�
�
�
�
�
�
�
� ����Ŀ
�
� � OK �
�
� �����
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
��������������������������������������������������������������������������������
here's the @Dom0 output of `journalctl -f`
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: seq 3220 queued, 'add'
'xen-backend'
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: Validate module index
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: Check if link configuration
needs reloading.
Jan 07 10:23:12 xent.lanint systemd-udevd[918]: seq 3220 forked new worker
[2464]
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: seq 3220 running
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: IMPORT builtin 'hwdb'
/usr/lib/udev/rules.d/50-udev-default.rules:15
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: IMPORT builtin 'hwdb'
returned non-zero
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: RUN 'kmod load
$env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: Execute 'load'
'xen-backend:vbd'
Jan 07 10:23:12 xent.lanint systemd-udevd[2464]: Inserted 'xen_blkback'
Jan 07 10:23:12 xent.lanint root[2508]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51712
Jan 07 10:23:12 xent.lanint root[2511]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51808
Jan 07 10:23:12 xent.lanint root[2509]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51776
Jan 07 10:23:12 xent.lanint root[2510]: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/1/51792
Jan 07 10:23:13 xent.lanint kernel: tun: Universal TUN/TAP device driver,
1.6
Jan 07 10:23:14 xent.lanint kernel: tun: (C) 1999-2004 Max Krasnyansky
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c1
--- Comment #1 from lists dev
You could try switching to ovmf-x86_64-{opensuse|suse|ms}.bin, just to see if one of the other ones behaves better. If there is no difference, we'll probably have to wait for an expert in this area to respond.
For all four cases bios_override = '/usr/share/qemu/ovmf-x86_64.bin' bios_override = '/usr/share/qemu/ovmf-x86_64-opensuse.bin' bios_override = '/usr/share/qemu/ovmf-x86_64-suse.bin' bios_override = '/usr/share/qemu/ovmf-x86_64-ms.bin the result's the same: in shell console, ERROR Verification failed: (15) Access Denied and serial output stops at (d8) [2017-01-08 02:12:52] Invoking OVMF ... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c2
Gary Ching-Pang Lin
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c3
--- Comment #3 from lists dev
When the TianoCore logo showed (in the very beginning)
I do not see the TianoCore logo when booting my DomU clients. The only time I can *recall* seeing it was "way back" when I installed the DomU. Which "beginning" are you referring to? Can I get BACK into the config menu for this existing Guest ?
I guess the guest booted with ovmf-x86_64-ms before so the MS keys were written into NvVars. I'll try to reproduce the issue with my machine.
Wouldn't the guest only boot with ovmf-x86_64-ms if it was SPECIFIED in the .cfg? My .cfg has always had bios_override = '/usr/share/qemu/ovmf-x86_64.bin' specified. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c4
--- Comment #4 from Gary Ching-Pang Lin
When the TianoCore logo showed (in the very beginning)
I do not see the TianoCore logo when booting my DomU clients.
The only time I can *recall* seeing it was "way back" when I installed the DomU.
Which "beginning" are you referring to?
Can I get BACK into the config menu for this existing Guest ?
It's easier to see the logo when booting the guest with sdl or vnc, and the logo will be gone quickly. Another possible way is to clear NvVars. Don't do it in the guest since ovmf will write the current configuration later. Mount the disk image directly and remove NvVars. If there is no NvVars, ovmf will just use the default values.
I guess the guest booted with ovmf-x86_64-ms before so the MS keys were written into NvVars. I'll try to reproduce the issue with my machine.
Wouldn't the guest only boot with ovmf-x86_64-ms if it was SPECIFIED in the .cfg?
My .cfg has always had
bios_override = '/usr/share/qemu/ovmf-x86_64.bin'
specified. By default, xen in openSUSE use ovmf-x86_64-ms.bin. But if the guest always booted with ovmf-x86_64.bin, it's very strange to see the MS keys existed in NvVars.
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c5
--- Comment #5 from lists dev
It's easier to see the logo when booting the guest with sdl or vnc, and the logo will be gone quickly.
Booting to SDL did the trick. In the TianoCore config, it says SecureBoot current state is "Enabled". I DE-selected "Attempt SecureBoot", SAVED and restarted. It still boots to the error message. If I repeat config entry, it still says "Enabled", even thought the check-box remains DE_selected. In the TianoCore config's 'Boot Manager Menu' the options are [opensuse-secureboot] UEFI floppy UEFI floppy 2 UEFI QEMU HARDDISK QM00001 UEFI Misc Device UEFI Misc Device 2 UEFI Misc Device 3 UEFI Misc Device 4 EFI Internal Shell where [opensuse-secureboot] is selected -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c6
--- Comment #6 from Gary Ching-Pang Lin
It's easier to see the logo when booting the guest with sdl or vnc, and the logo will be gone quickly.
Booting to SDL did the trick.
In the TianoCore config, it says SecureBoot current state is "Enabled".
I DE-selected "Attempt SecureBoot", SAVED and restarted.
It still boots to the error message. If I repeat config entry, it still says "Enabled", even thought the check-box remains DE_selected.
Sorry, I forgot to mention that the firmware has to be reset in a proper way. Press "esc" two times to back to the main menu and choose "continue" and then a prompt will hint that the system is going to reset. For KVM with pflash, merely f10 save should be fine. The firmware can write the data immediately to the flash file. However, xen doesn't pflash yet, so it only writes back the data to NvVars (an emulated flash) in certain conditions, i.e. a proper shutdown/reset. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c7
--- Comment #7 from lists dev
Press "esc" two times to back to the main menu and choose "continue" and then a prompt will hint that the system is going to reset.
That's the procedure I used ... esc + esc + continue at which point the error msg Verification failed: (15) Access Denied simply appears. I see no reset message. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c8
--- Comment #8 from Gary Ching-Pang Lin
Press "esc" two times to back to the main menu and choose "continue" and then a prompt will hint that the system is going to reset.
That's the procedure I used ... esc + esc + continue
at which point the error msg
Verification failed: (15) Access Denied
simply appears. I see no reset message.
Hmmm, changing the secure boot related variables will need a reset, and a prompt should show after "continue". It's not normal that the prompt didn't show. BTW, could you provide the version of qemu-ovmf-x86_64? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c9
--- Comment #9 from lists dev
simply appears. I see no reset message.
Hmmm, changing the secure boot related variables will need a reset, and a prompt should show after "continue". It's not normal that the prompt didn't show.
Sure. Here tho all I see is that error.
BTW, could you provide the version of qemu-ovmf-x86_64?
(In reply to Gary Ching-Pang Lin from comment #8)
BTW, could you provide the version of qemu-ovmf-x86_64?
rpm -qa | grep -i ovmf ovmf-2017+git1480394913.2b2efe3-40.1.x86_64 qemu-ovmf-x86_64-2017+git1480394913.2b2efe3-40.1.noarch from http://download.opensuse.org/repositories/Virtualization/openSUSE_Leap_42.2 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c10
--- Comment #10 from lists dev
Hmmm, changing the secure boot related variables will need a reset, and a prompt should show after "continue". It's not normal that the prompt didn't show.
I force reinstalled @ Dom0 kernel-default, ovmf, ovmf-tools, qemu-ovmf, xen, xen-libs, xen-tools Now, after toggling the TianoCore config to [ ] Attempt Secure Boot then Esc - Esc - Config, I *do* get a prompt: "Configuration changed. Reset to apply it Now. Press ENTER to reset." hit ENTER, it restarts, I see the logo, and ... immediately get the "Verification failed" error again. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c11
--- Comment #11 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c12
--- Comment #12 from Gary Ching-Pang Lin
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c13
--- Comment #13 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c14
--- Comment #14 from Gary Ching-Pang Lin
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c15
Gary Ching-Pang Lin
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c16
--- Comment #16 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c20
--- Comment #20 from Gary Ching-Pang Lin
[Bds]Booting opensuse-secureboot FSOpen: Open '\EFI\opensuse\shim.efi' Success [Bds] DevicePath expand: HD(1,GPT,12A830DE-7E2F-4DC6-B64F-B886FB80E77C,0x800,0x3F7DF)/ \EFI\opensuse\shim.efi -> PciRoot(0x0)/Pci(0x1,0x1)/Ata(Primary,Master,0x0)/HD(1,GPT,12A830DE-7E2F- 4DC6-B64F-B886FB80E77C,
it's still booting in 'secureboot' mode.
Rebooting, & checking in TianoCore config, however, it reports
Secure Boot Configuration Current Secure Boot State Disabled Attempt Secure Boot [ ] Secure Boot Mode <standard Mode>
so it thinks it should be in NON-SecureBoot mode.
So, why isn't it is the question.
To simplify the boot path, openSUSE always boots from shim.efi. shim.efi will detect the SecureBoot variable and decide whether the signature verification should be applied or not. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c23
Charles Arnold
Hi Charles,
Could you help to check if the option "bios_override" was changed to "bios_path_override"?
On the mailing list there was a discussion about what it should be called. https://lists.xen.org/archives/html/xen-devel/2016-03/msg01928.html The patch that was posted there had "bios_override". The actual patch that was committed to git called it "bios_path_override". So bios_path_override is the correct name. (see commit d7dd8737ae98b6cc2af22fcaa9a3f3a8480cb709) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c24
lists dev
(In reply to Gary Ching-Pang Lin from comment #20) opensuse ignores the presence/use of it?
Just putting some context around that question, since -- from within the chroot'd Guest -- apparently I (1) can't manage to write to the OVMF bios boot menu (2) startup.nsh is ignored If I were to create a standalone .efi, e.g., grub2-mkstandalone \ -d /usr/lib/grub2/x86_64-efi/ \ -O x86_64-efi \ --modules="..." \ --fonts="..." \ --locales="..." \ --themes="..." \ -o "/boot/efi/EFI/opensuse/grub-standalone-test.efi" \ "boot/grub2/grub.cfg=/boot/grub2/grub.cfg" how would I convince opensuse xen/ovmf to use that^ on boot if startup.nsh is not used ? Would I have to rename it as grub.efi or grubx64.efi? That doesn't seem a good idea. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c25
--- Comment #25 from lists dev
I guess the guest booted with ovmf-x86_64-ms before so the MS keys were written into NvVars. I'll try to reproduce the issue with my machine.
In current state, [case 1] edit guest.cfg ... bios = 'ovmf' # bios_path_override = ... chroot into guest rm -f /boot/efi/NvVars exit chroot xl create guest.cfg screen (1) TianoCore logo screen (2) dialog: Trust openSUSE Certificate Do you agree to use the built-in openSUSE certificate to verify boot loaders and kernels? [No] [Yes] select/enter [Yes] screen (3) quick flash of "System bootloader not found ... ..." then ERROR Verification failed: (15) Access Denied [OK] xl destroy guest chroot into guest note 'NvVars' created ls -al /boot/efi total 18K drwxr-xr-x 3 root root 512 Dec 31 1969 ./ drwxr-xr-x 5 root root 4.0K Jan 10 08:44 ../ drwxr-xr-x 4 root root 512 Jun 21 2016 EFI/ -rwxr-xr-x 1 root root 13K Jan 10 16:41 NvVars* exit chroot xl create guest screen (1) TianoCore logo screen (2) ERROR Verification failed: (15) Access Denied [OK] [case 2] edit guest.cfg ... bios = 'ovmf' bios_path_override = '/usr/share/qemu/ovmf-x86_64.bin' ... chroot into guest rm -f /boot/efi/NvVars exit chroot xl create guest.cfg screen (1) TianoCore logo screen (2) GNU GRUB version 2.02^beta3 OpenSUSE Advanced options for OpenSUSE [PVHVM] boots 'PVHVM' automatically ... stalls, as reported [case 3] edit guest.cfg ... bios = 'ovmf' bios_path_override = '/usr/share/qemu/ovmf-x86_64-ms.bin' ... chroot into guest rm -f /boot/efi/NvVars exit chroot xl create guest.cfg screen (1) TianoCore logo screen (2) dialog: Trust openSUSE Certificate Do you agree to use the built-in openSUSE certificate to verify boot loaders and kernels? [No] [Yes] select/enter [Yes] screen (3) quick flash of "System bootloader not found ... ..." then ERROR Verification failed: (15) Access Denied [OK] -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c26
Gary Ching-Pang Lin
(In reply to Gary Ching-Pang Lin from comment #20)
To simplify the boot path, openSUSE always boots from shim.efi. shim.efi will detect the SecureBoot variable and decide whether the signature verification should be applied or not.
IIUC, the use of /boot/efi/startup.nsh, which here contains:
echo "fs0:\EFI\opensuse\grubx64.efi" > /boot/efi/startup.nsh
is supposed to simplify the boot path.
opensuse ignores the presence/use of it? No, it's not relevant to the boot path.
For UEFI, the priority of the boot loader is decided by BDS in the firmware. For OS, after installation, it should create a boot option and set the priority of the boot option, and then BDS chooses one boot option from the existed ones. To simplify the boot path, openSUSE only creates one boot option which points to shim.efi instead of creating another one for grub.efi. startup.nsh is for UEFI shell, and it only works when BDS boots UEFI shell. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c27
--- Comment #27 from Gary Ching-Pang Lin
(In reply to Gary Ching-Pang Lin from comment #20) opensuse ignores the presence/use of it?
Just putting some context around that question,
since -- from within the chroot'd Guest -- apparently I
(1) can't manage to write to the OVMF bios boot menu (2) startup.nsh is ignored
If I were to create a standalone .efi, e.g.,
grub2-mkstandalone \ -d /usr/lib/grub2/x86_64-efi/ \ -O x86_64-efi \ --modules="..." \ --fonts="..." \ --locales="..." \ --themes="..." \ -o "/boot/efi/EFI/opensuse/grub-standalone-test.efi" \ "boot/grub2/grub.cfg=/boot/grub2/grub.cfg"
how would I convince opensuse xen/ovmf to use that^ on boot if startup.nsh is not used ?
Would I have to rename it as grub.efi or grubx64.efi? That doesn't seem a good idea.
For the one-time boot, enter the "Boot Maintenance Manager" and choose "Boot
From File" to find the specific efi image. To create a permanent boot option, choose "Boot Options" -> "Add Boot Option". You can change the boot order with "Boot Options" -> "Change Boot Order". But it seems your guest has problem to save NvVars correctly, I'm not sure if the boot option will be saved.
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c28
--- Comment #28 from lists dev
startup.nsh is for UEFI shell, and it only works when BDS boots UEFI shell. ... For the one-time boot, enter the "Boot Maintenance Manager" and choose "Boot From File" to find the specific efi image. To create a permanent boot option, choose "Boot Options" -> "Add Boot Option". You can change the boot order with "Boot Options" -> "Change Boot Order". But it seems your guest has problem to save NvVars correctly, I'm not sure if the boot option will be saved.
In Dom0, I can (still) easily set 'EFI Shell' to Boot Order == 0000. With startup.nsh in place, on boot it does -- as you explain -- fall through to startup.nsh after ~4 sec delay. Works well if needed. In the Guest, I am not able to get the boot option to stick. Is it possible, from within chroot, to write a correct set of NvVars directly, and get the guest-boot process to use them? I can certainly use efibootmgr to write to file, but am not clear if that's sufficient, or what the recommended options should be. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c29
--- Comment #29 from lists dev
For the one-time boot, enter the "Boot Maintenance Manager" and choose "Boot From File" to find the specific efi image.
In the chroot'd guest, I set echo "fs0:\EFI\opensuse\grubx64.efi" > /boot/efi/startup.nsh exit chroot restart guest enter OVMF config select Boot Manager select one-time EFI Shell Shell appears, times out & falls through to startup.nsh and fails at Shell> fs0:\EFI\opensuse\grubx64.efi Script Error Status: Security Violation (line number 1) Shell> -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c30
--- Comment #30 from Gary Ching-Pang Lin
startup.nsh is for UEFI shell, and it only works when BDS boots UEFI shell. ... For the one-time boot, enter the "Boot Maintenance Manager" and choose "Boot From File" to find the specific efi image. To create a permanent boot option, choose "Boot Options" -> "Add Boot Option". You can change the boot order with "Boot Options" -> "Change Boot Order". But it seems your guest has problem to save NvVars correctly, I'm not sure if the boot option will be saved.
In Dom0, I can (still) easily set 'EFI Shell' to Boot Order == 0000.
With startup.nsh in place, on boot it does -- as you explain -- fall through to startup.nsh after ~4 sec delay. Works well if needed.
In the Guest, I am not able to get the boot option to stick.
Is it possible, from within chroot, to write a correct set of NvVars directly, and get the guest-boot process to use them?
AFAIK, there is no such tool.
I can certainly use efibootmgr to write to file, but am not clear if that's sufficient, or what the recommended options should be.
If the guest can boot to OS, efibootmgr is recommended to change the boot options and boot order. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c31
--- Comment #31 from Gary Ching-Pang Lin
For the one-time boot, enter the "Boot Maintenance Manager" and choose "Boot From File" to find the specific efi image.
In the chroot'd guest, I set
echo "fs0:\EFI\opensuse\grubx64.efi" > /boot/efi/startup.nsh
exit chroot restart guest enter OVMF config select Boot Manager select one-time EFI Shell
Shell appears, times out & falls through to startup.nsh
and fails at
Shell> fs0:\EFI\opensuse\grubx64.efi Script Error Status: Security Violation (line number 1) Shell>
This means Secure Boot is enabled. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c32
--- Comment #32 from lists dev
If the guest can boot to OS, efibootmgr is recommended to change the boot options and boot order.
Shell> fs0:\EFI\opensuse\grubx64.efi Script Error Status: Security Violation (line number 1) Shell>
This means Secure Boot is enabled.
I'm out of obvious config options to try. I can't get out of SecureBoot, can't boot the Guest, and can't change its vars persistently. Any further suggestions? Or is there a bug to be fixed? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c33
--- Comment #33 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c34
--- Comment #34 from Gary Ching-Pang Lin
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c35
lssl lssl
While I'm still finding a workable UEFI xen machine, I've built the latest OVMF snapshot:
https://build.opensuse.org/package/show/home:gary_lin:branches: Virtualization/ovmf
Maybe you can give a try.
installing your ovmf-tools-2017+git1484111680.7c14bc8769-44.1.x86_64.rpm qemu-ovmf-x86_64-2017+git1484111680.7c14bc8769-44.1.noarch.rpm ovmf-2017+git1484111680.7c14bc8769-44.1.x86_64.rpm cat guest.cfg ... bios = 'ovmf' bios_path_override = '/usr/share/qemu/ovmf-x86_64.bin' ... launch guest enter OVMH config select EFI shell grub menu appears, launches grub config stalls no access to dracut emerg shell; UNable to access guest via console destroy guest edit guest.cfg ... bios = 'ovmf' - bios_path_override = '/usr/share/qemu/ovmf-x86_64.bin' + #bios_path_override = '/usr/share/qemu/ovmf-x86_64.bin' ... grub menu appears, launches grub config dialog ERROR Verification failed: [OK] cat debug.log, "Step 1" ===========> http://pastebin.com/S1LHkYEq click OK dialog ERROR Verification failed: [OK] cat debug.log (same link), "Step 2" ===========> http://pastebin.com/S1LHkYEq -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c36
--- Comment #36 from Gary Ching-Pang Lin
From my perspective, the real problem is 2) since it's using the default setup of the bootloader and kernel.
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c37
--- Comment #37 from lists dev
However, with bios_path_override, the Secure Boot issue can be worked around.
Yes, can boot with SecureBoot DISabled.
2) The system failed at initrd. Per comment#16, the system dropped into the dracut emergency shell. Does Leap 42.2 DVD work? Just wonder whether there is anything missing in initrd.
Launching it as a Guest? I'll have to give that a try.
3) Customized grub2 Per comment#24, a customized grub2 was created. In this case, "ovmf-x86_64-ms.bin" should not be used since the new grub.efi cannot be verified by shim.efi.
A standalone grub-standalone-test.efi is NOT being currently used
(Anything else?)
From my perspective, the real problem is 2) since it's using the default setup of the bootloader and kernel.
Without the ability to boot into the Guest, isn't `mkinitrd` in a chrooted Guest env generally sufficient? Perhaps parameterless mkinitrd isn't ... and passing specific params to it? I'll take a look; if you have a specific recipe that should be tried, pls post. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c38
--- Comment #38 from lists dev
2) The system failed at initrd. Per comment#16, the system dropped into the dracut emergency shell. Does Leap 42.2 DVD work? Just wonder whether there is anything missing in initrd.
Starting with cat guest.cfg ... bios = 'ovmf' bios_path_override = '/usr/share/qemu/ovmf-x86_64.bin' ... in order to DISABLE secureboot I enetered Guest chroot exec'd mkinitrd -v noted ... I: Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force /boot/initrd-4.1.12-1-default 4.1.12-1-default ... version 4.1.12? That exists neither on Dom0 or in DomU. Is that hardcoded? In any case, it completes ... and the Guest is unbootable, as before exec'ing instead K="4.9.2-2.g2d3c294" R="/dev/mapper/VG0_T_ROOT1" mkinitrd -v -A \ -k /boot/vmlinuz-${K}-default \ -i /boot/initrd-${K}-default \ -M /boot/System.map-${K}-default \ -d ${R} ls -al /boot/ ... lrwxrwxrwx 1 root root 31 Jan 10 08:34 initrd -> initrd-4.9.2-2.g2d3c294-default -rw------- 1 root root 33M Jan 12 06:34 initrd-4.9.2-2.g2d3c294-default ... exited the chroot launched the Guest Guest boot is now successful. Guest is up and running. Repeating, but without the 'monster' init - mkinitrd -v -A \ + mkinitrd -v \ after chroot exit, guest launch FAILs, never completing. Sanity check repeat with - mkinitrd -v \ + mkinitrd -v -A \ , Guest is up and running again. Your suspicion that the initrd's "missing something" sounds about right ... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c39
--- Comment #39 from lists dev
Your suspicion that the initrd's "missing something" sounds about right ...
Here's the diff of lsinitrd output for (1) mkinitrd -v (2) mkinitrd -v -A ... =========> https://www.diffchecker.com/QBSbkfPq -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c40
--- Comment #40 from Mike Latimer
Without the ability to boot into the Guest, isn't `mkinitrd` in a chrooted Guest env generally sufficient?
Before you chrooted into the guest filesystem, did you bind mount the /proc, /sys, and /dev filesystems to the new chroot environment? This process is very similar to Rescue Mode, and without having those system directories available to the chroot environment, mkinitrd is not able to build a fully functional initrd. (FYI - This TID talks about bind mounting in rescue environments: https://www.suse.com/support/kb/doc?id=7018126) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c41
--- Comment #41 from lists dev
(In reply to lists dev from comment #37)
Without the ability to boot into the Guest, isn't `mkinitrd` in a chrooted Guest env generally sufficient?
Before you chrooted into the guest filesystem, did you bind mount the /proc, /sys, and /dev filesystems to the new chroot environment?
yes, always. standard procedure ... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c42
--- Comment #42 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c43
--- Comment #43 from lists dev
then just
mkinitrd etc etc
with those^ adds, guest boot still fails, but @ xl console, I *do* now drop to the dracut shell
with "those^ adds" still in place, exec'ing kust mkinitrd -A gets me to a booted Guest. repeating mkinitrd fails again. repeating mkinitrd -A REMOVING the additions in dracut config also ends up in a failed boot with both mkinitrd mkinitrd -A -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c44
--- Comment #44 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c45
Mike Latimer
for all opensuse GUESTS
enter Guest chroot add changes^ to dracut.conf.d/99-xen-domu.conf mkinitrd -vA exit chroot `xl create` Guest
all opensuse guests boot successfully
Can you confirm whether the above steps are required on all openSUSE guests, or only those guests which encountered the OVMF issues (and have been hacked on during troubleshooting)? I'm not sure why I didn't see this on my test VM, but I also haven't yet seen the OVMF issue. (I do plan on doing additional testing.) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c46
lists dev
Can you confirm whether the above steps are required on all openSUSE guests, or only those guests which encountered the OVMF issues (and have been hacked on during troubleshooting)?
On this server, I've 5 OpenSUSE guests, up to date with latest KernelStable, Grub, etc. as of immediately prior to the Xen 4.7 -> 4.8 upgrade of the Host. I've only 'hacked' on the one Guest; this^ thread has been all on/with that one Guest. Well, until the last getting-it-working-again bits ... I'd bet that there's some shades-of-gray less that I could've gotten to work; I didn't do the necessary 'hacking' on those Guests, instead choosing to simply replicate what worked on the hacked-guest. And, yes, all those steps were apparently needed for all 4 of those OpenSUSE guests.
I'm not sure why I didn't see this on my test VM, but I also haven't yet seen the OVMF issue. (I do plan on doing additional testing.)
What I can say is that the `mkinitrd -A` "monster" initrd seems to be necessary; at least in the chroot environment. I refuse to believe that ALL of it is required, but haven't found the nominal delta that works, yet. I also haven't started playing again with the booted Guests, to see if "just" a `mkinitrd` (no -A) from inside the fully-virtualized Guest env will kill boot again ... And, fwiw, currently, all seems (m_oO_m) to be working having dropped back to using OVMF from the Virtualization repo, instead of glin's custom build. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c47
--- Comment #47 from lists dev
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c48
--- Comment #48 from Mike Latimer
There's something obviously unhealthy about in-chroot, 'just' `mkinitrd`.
I believe this is likely due to the logic in mkinitrd which detects the driver used by the root disk. In the chroot environment, this is probably failing, and then failing to add xen-blkfront into the initrd, or failing to setup the root device properly. The `mkinitrd -A` just adds everything and it gets around the problem. I think the 'mkinitrd in chroot' problem is a secondary issue as there really should be no reason to do that step (under normal circumstances). We need to fix the problem before it gets to that point. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c49
--- Comment #49 from lists dev
I think the 'mkinitrd in chroot' problem is a secondary issue as there really should be no reason to do that step (under normal circumstances). We need to fix the problem before it gets to that point.
Yes, and there should be a well-understood procedure for when-not-if a Xen DomU &/or UEFI "lose their mind" again. Which happens more frequently that it should, simply because there are still too many new AND moving parts involved. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741
http://bugzilla.opensuse.org/show_bug.cgi?id=1018741#c50
Gary Ching-Pang Lin
participants (1)
-
bugzilla_noreply＠novell.com