http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 Michal Suchanek changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |msuchanek@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c2 Neil Rickert changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nwr10cst-oslnx@yahoo.com --- Comment #2 from Neil Rickert --- I have seen two cases, but only indirectly. ZB reports on his case in a forum thread, and in this bug report. His problem was with Leap 15.2 And Michal S reports on the factory mailing list, and that was for Tumbleweed. I have come across (indirectly) firmware that ignores NVRAM entries and always uses "\EFI\boot\bootx64.efi" to boot the system. I have come across those via forum posts. It looks as if the two cases of this bug have that kind of firmware. I'll note, also, that ZB reports that he has gone back to the previous shim and kernel, and his system boots that way. My tests indicate that the old shim will boot the new kernel but the new shim will not boot the old kernel. I've tested this with both Leap 15.2 and Tumbleweed. So the problem is related to the changes to shim. I am unable to reproduce the problem. I can force the use of "\EFI\boot\bootx64.efi". But when I do that, it loads "fallback.efi" and from there I get a successful boot. It somehow looks as if "fallback.efi" is not being run on the two problem systems, but I do not know why. And a further note: If you were to copy "grub.efi", "MokManager.efi" and "grub.cfg" from "\EFI\opensuse" to "\EFI\boot", I am guessing that would work. Perhaps that is worth trying. In my experience, those files are not used in "fallback.efi" is present. But the reported errors suggest that those files would be used on the two problem systems. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c3 Andrei Borzenkov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arvidjaar@gmail.com, | |glin@suse.com --- Comment #3 from Andrei Borzenkov --- I do not see how booting from \EFI\Boot is supposed to work in presence of pending MOK requests. shim.c:efi_main() efi_status = import_mok_state(image_handle); if (!secure_mode() && efi_status == EFI_INVALID_PARAMETER) { ... } else if (EFI_ERROR(efi_status)) { die: console_print(L"Something has gone seriously wrong: %s: %r\n", msgs[msg], efi_status); msleep(5000000); gRT->ResetSystem(EfiResetShutdown, EFI_SECURITY_VIOLATION, 0, NULL); } ... /* * Hand over control to the second stage bootloader */ efi_status = init_grub(image_handle); We have import_mok_state() - check_mok_request() - start_image(image_handle, MOK_MANAGER) which returns Not found because MokManager is not present in \EFI\Boot. And shim immediately resets. fallback.efi would have been attempted in init_grub() but we never reach it. IOW - fallback booting by definition does not work if we have pending MOK request. I do not see any easy solution beyond integrating MokManager into shim binary to ensure it is always available ... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 Christian Hägele changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christian_haegele@web.de -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c6 --- Comment #6 from Andrei Borzenkov --- (In reply to Christian Hägele from comment #5)

I have the same problem with a ASUS PRIME X-570-PRO mainboard (if that matters) with Secure-Boot enabled.

Since the metioned update the bootloader does not load and I ony see a black screen

How can it be the same problem? In this problem you get clear error message, not black screen. Mixing unrelated issues neither helps to troubleshoot original problem nor to solve your problem. Granted, this bug report summary is misleading, bootloader was not "killed" actually.

I finally found a workaround by copying shim.efi and momanager.efi from openSUSE 15.2 installation-mediuem (old version) manually to the EFI partition

Where exactly did you copy them (ESP has directories)? What is output of "efibootmgr -v"? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c8 Andrei Borzenkov changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #8 from Andrei Borzenkov --- (In reply to Neil Rickert from comment #7)

I then copied "shim.efi" and "fallback.efi" [from 15.1] to "\EFI\boot", and renamed "shim.efi" to "bootx64.efi".

I created a MokManager request. And I deleted the boot entry, so that it would go to the fallback.

On reboot, I got the MokManager screen (to delete a key). And then the system booted. So somehow this all works with the old shim but not with the new shim.

Yes, shim 14 had efi_status = measure_mok(); if (efi_status != EFI_SUCCESS && efi_status != EFI_NOT_FOUND) { i.e. it ignored missing MokManager, continued with fallback which called shim from \EFI\opensuse which called MokManager (because request was still pending). This is upstream change, commit 4181a16f62951c6078802bdea735947c69999acd, which removed check for EFI_NOT_FOUND without as much as a single word of explanation (commit message talks about something entirely different). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c10 --- Comment #10 from Z B --- This problem was fixed by a bios update on a dell precision 7530. Bios version 1.5.2 results in failed boot with the new shim. Bios 1.13.1 boots successfully. I do not know what other bios versions might be affected by this. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c12 --- Comment #12 from Gary Ching-Pang Lin --- (In reply to Andrei Borzenkov from comment #3)

I do not see how booting from \EFI\Boot is supposed to work in presence of pending MOK requests.

shim.c:efi_main()

efi_status = import_mok_state(image_handle);

if (!secure_mode() && efi_status == EFI_INVALID_PARAMETER) { ... } else if (EFI_ERROR(efi_status)) { die: console_print(L"Something has gone seriously wrong: %s: %r\n", msgs[msg], efi_status); msleep(5000000); gRT->ResetSystem(EfiResetShutdown, EFI_SECURITY_VIOLATION, 0, NULL); } ... /* * Hand over control to the second stage bootloader */ efi_status = init_grub(image_handle);

We have import_mok_state() - check_mok_request() - start_image(image_handle, MOK_MANAGER) which returns Not found because MokManager is not present in \EFI\Boot. And shim immediately resets. fallback.efi would have been attempted in init_grub() but we never reach it.

Thanks for the analysis! I can reproduce the error by boothing bootx64.efi with a pending MOK request.

IOW - fallback booting by definition does not work if we have pending MOK request. I do not see any easy solution beyond integrating MokManager into shim binary to ensure it is always available ...

I'd rather not integrate more code into shim since it needs to be signed by UEFI CA and the signing process could take a long while. An easy fix is to modify shim-install to install MokManager.efi to \EFI\boot to guarantee that all MOK request will be processed. Will work on that. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c20 --- Comment #20 from Michal Suchanek --- After reboot: # efibootmgr BootCurrent: 0000 Timeout: 0 seconds BootOrder: 0004 Boot0000* Notebook Hard Drive Boot0001* Notebook Ethernet Boot0002* Notebook Ethernet Boot0003* opensuse-secureboot Boot0004* opensuse-secureboot -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c21 --- Comment #21 from Neil Rickert --- Reply to comment c#20 It seems strange that, after the reinit, there is only one entry on the boot order list. I suspect that your firmware is making changes to the boot order on reboot. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c23 --- Comment #23 from Martin Wilck --- Am I understanding correctly that this problem would only manifest itself with secure boot disabled? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c25 --- Comment #25 from Andrei Borzenkov --- (In reply to Martin Wilck from comment #24)

More precisely, have we understood which firmware properties trigger this error?

Which error do you mean precisely? There are two related issues here 1. Current shim requires presence of MokManager, otherwise it does not allow further booting and simply turns off system. This is unrelated to firmware. 2. Some firmware is known to arbitrary reorder boot options or completely ignore some of them. There is no specific property (at least I am aware of anybody doing systematical study) that can be used to detect it. So issue 2 could mean users were actually using \EFI\Boot\bootx64.efi even without knowing it and with shim update that triggered issue 1. I an not aware of any way to detect issue 2 in advance. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c27 --- Comment #27 from Gary Ching-Pang Lin --- I'm working on the shim update to install MokManager to \EFI\Boot. Will apply the change to 15.1, 15.2, and Tumbleweed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c30 --- Comment #30 from John Shaw --- (In reply to Martin Wilck from comment #29)

(In reply to Gary Ching-Pang Lin from comment #27)

...

I'm working on the shim update to install MokManager to \EFI\Boot.

Just in case someone else is wondering, too: I asked myself what effect this might have on dual-boot or multi-boot systems, where \EFI\Boot is shared between OSes. I talked to Gary about that, and he explained to me that the shim-install script will only touch \EFI\Boot if it detects an (open)SUSE shim.efi (by looking at the CA cert). So it shouldn't be an issue.

I just installed the new shim package with this fix. I have kernel 5.3.18-lp152.36 as the default, and this still does not boot with SecureBoot enabled in the BIOS (at least it boots if its disabled;). I verified that /boot/efi/efi/boot contains: bootx64.efi fallback.efi MokManager.efi Reading some other comments, it looks like this really should have worked. The newer kernel was in place before the new shim package was installed. Do I have to do something else like (with Yast) reinstall the boot loader or clear the old certificates from the BIOS? I am running OpenSuSE leap 15.2 on an Asus Z170-A MB (latest BIOS). Yast still offers the old version 14 of the shim package. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c33 --- Comment #33 from Gary Ching-Pang Lin --- (In reply to Gary Ching-Pang Lin from comment #32)

(In reply to John Shaw from comment #30)

...

(In reply to Martin Wilck from comment #29)

...

(In reply to Gary Ching-Pang Lin from comment #27)

...

I'm working on the shim update to install MokManager to \EFI\Boot.

Just in case someone else is wondering, too: I asked myself what effect this might have on dual-boot or multi-boot systems, where \EFI\Boot is shared between OSes. I talked to Gary about that, and he explained to me that the shim-install script will only touch \EFI\Boot if it detects an (open)SUSE shim.efi (by looking at the CA cert). So it shouldn't be an issue.

I just installed the new shim package with this fix. I have kernel 5.3.18-lp152.36 as the default, and this still does not boot with SecureBoot enabled in the BIOS (at least it boots if its disabled;). I verified that /boot/efi/efi/boot contains: bootx64.efi fallback.efi MokManager.efi

Could you elaborate the failure? Was there any message? Did the grub2 menu show?

...

Reading some other comments, it looks like this really should have worked. The newer kernel was in place before the new shim package was installed. Do I have to do something else like (with Yast) reinstall the boot loader or clear the old certificates from the BIOS? I am running OpenSuSE leap 15.2 on an Asus Z170-A MB (latest BIOS). Yast still offers the old version 14 of the shim package.

Could you post the output of the following commands?

$ mokutil --list-new

$ pesign -S -i /boot/efi/EFI/boot/bootx64.efi $ pesign -S -i /boot/efi/EFI/boot/MokManager.efi $ pesign -S -i /boot/efi/EFI/boot/fallback.efi $ pesign -S -i /boot/efi/EFI/opensuse/shim.efi

Forgot to mention grub2: $ pesign -S -i /boot/efi/EFI/opensuse/grub.efi -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c35 --- Comment #35 from John Shaw --- With regard to my reply in comment 34, I do not get to the GRUB menu. If SecureBoot is enabled, failure is an immediate blank screen(s) as the BIOS exits POST. Nothing reboots the machine (eg, cntl-alt-del) other than physically pressing the reboot button. Other info: In my Asus BIOS, I can effectively disable SecureBoot by selecting "Other OS" vs "Windows UEFI" under the SecureBoot section. This still displays a line stating that SecureBoot is enabled, but the full linux boot sequence works, and mokutil will confirm that it's disabled. I have read that clearing the BIOS keys also disables SecureBoot, and that the BIOS will state so. I have not tried that method. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c37 --- Comment #37 from John Shaw --- (In reply to Martin Wilck from comment #36)

(In reply to John Shaw from comment #35)

...

With regard to my reply in comment 34, I do not get to the GRUB menu. If SecureBoot is enabled, failure is an immediate blank screen(s) as the BIOS exits POST. Nothing reboots the machine (eg, cntl-alt-del) other than physically pressing the reboot button.

The question is if the BIOS fails to verify the shim, or shim fails to verify any of the follow-up binaries. Can you check this e.g. with the UEFI shell?

I don't know how to check this in the UEFI shell. Can you explain? However, I managed to get a version of sbverify to work under suse 15.2. How correct this version is is unclear. I got the binary from a ubuntu repo, and I had to jury-rig the libcrypto.1.0.0 library dependency with a symlink to /usr/lib64/libcrypto.so.43. Anyway, it apparently works for the --list option. efi/boot% sudo sbverify --list bootx64.efi warning: data remaining[1172696 vs 1336112]: gaps between PE/COFF sections? signature 1 image signature issuers: - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011 image signature certificates: - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher issuer: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011 issuer: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root signature 2 image signature issuers: - /CN=openSUSE Secure Boot CA/C=DE/L=Nuremberg/O=openSUSE Project/emailAddress=build@opensuse.org image signature certificates: - subject: /CN=openSUSE Secure Boot Signkey/C=DE/L=Nuremberg/O=openSUSE Project/emailAddress=build@opensuse.org issuer: /CN=openSUSE Secure Boot CA/C=DE/L=Nuremberg/O=openSUSE Project/emailAddress=build@opensuse.org I tested every efi file and the vmlinux images in both /efi/boot and /efi/opensuse, and all BUT /efi/opensuse/grubx64.efi appear to have valid signature certificates. For this I get: % sudo sbverify --list grubx64.efi No signature table present Should grubx86.efi be signed or not? As to using sbverify to actually verify the signatures, I either don't understand what to use for the x509 certificate (for the --cert option), or the utility is broken (as it was not meant for Suse). Its looking for a file here. I tried to save the for keysets in the bios (db, dbx, KEK, PK) and these do seem to be somesort of certificate files, but they don't work in this context. One must be able to get the appropriate (original) certificate from the BIOS, but I don't know how to get it in the correct x509 format.

...

Other info: In my Asus BIOS, I can effectively disable SecureBoot by selecting "Other OS" vs "Windows UEFI" under the SecureBoot section. This still displays a line stating that SecureBoot is enabled, but the full linux boot sequence works, and mokutil will confirm that it's disabled.

Sounds like broken BIOS.

its the only one I got (well, the most recent one anyway;). Asus does not define this very well. I interpreted "Other OS" as anything that does not use UEFI but clearly my system is not booting from an MBR.

...

I have read that clearing the BIOS keys also disables SecureBoot, and that the BIOS will state so. I have not tried that method.

I wouldn't recommend that, unless you're certain you can undo it.

Yes, I don't really know what I am doing here. I have the keys backed up, but I can see really screwing this up. I can restore the keys to the orginal factory condition easily as well. I am wondering if its possible that the keys that are in the BIOS are somehow invalid. Still I would expect the BIOS to put put a big red screen and indicate it will not boot. That is difference behavior to hanging the 1st stage boot loader.

Btw the pesign -S output is a pain. No fingerprint or otherwise useful data, just the common name, which could mean anything. And why does it *always* print "Content is detached; signature cannot be verified."?

see my comment above regarding the use of sbverify. Can I use it to try and verify the signitures of the efi files? I just don't know how to supply the correct certificates with it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c39 Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |meissner@suse.com Flags| |needinfo?(meissner@suse.com | |) --- Comment #39 from Martin Wilck --- (In reply to John Shaw from comment #37)

see my comment above regarding the use of sbverify. Can I use it to try and verify the signitures of the efi files? I just don't know how to supply the correct certificates with it.

We have this tool actually (package sbsigntools in factory), but build is disabled for anything but factory itself. Quoting from https://build.opensuse.org/request/show/798204: "ok, but we should prefer to use our own tool pesign , as discussed by email" @Markus, what's the background of this statement? At least for signature verification, sbverify seems to be far more powerful then pesign. I reckon we should provide this package for SLE and Leap rather sooner than later. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c41 Marcus Meissner changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(meissner@suse.com | |) | --- Comment #41 from Marcus Meissner --- The background was my initial reaction to this "not invented here" addition and tool duplication, amidst the complex secure boot topic. I admit that sbverify is easier usable than what pesign offers. I do not have an issue with adding it to Leap 15.1 and 15.2 directly, but for SLE we still have the 1 tool / 1 purpose approach. That said, this bug hwere is not secure boot signature related. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c44 --- Comment #44 from Neil Rickert ---

Its unclear what to enrole.

The keys that you need should be in "/etc/uefi/certs". The key fingerprint is part of the file name. You can use: mokutil -l to list currently enrolled keys. Check their fingerprints against what is in "/etc/uefi/certs". If you are missing a key, try enrolling that. If you are having nvidia problems even when secure-boot is disabled, then that is probably not related to this bug. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c46 Gary Ching-Pang Lin changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(johngshaw@twc.com | |) --- Comment #46 from Gary Ching-Pang Lin --- BTW, I still would like to know whether "mokutil --set-verbosity true" makes shim print more messages before the blank screen. I hope it could provide some hints to the blank screen. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c48 --- Comment #48 from Martin Wilck --- (In reply to Gary Ching-Pang Lin from comment #45)

I guess the update of kernel triggered the enrollment of new signkey and the deletion of the old signkey. But somehow MokNew was gone, so MokManager thought that there was a reset.

How would MokNew go away? The scriptlets in the kernel rpm are written so that the keys won't be deleted as long as the system still has one kernel installed that needs them. Why do we have different signkeys for Leap 15.2, anyway?

~ # rpm -qlp /var/cache/zypp/packages/repo-oss/x86_64/kernel-default-5.3.18-lp152.19.2.x86_64.rpm | grep crt /etc/uefi/certs/F1C08E27.crt ~ # rpm -qlp /var/cache/zypp/packages/repo-update/x86_64/kernel-default-5.3.18-lp152.33.1.x86_64.rpm | grep crt /etc/uefi/certs/F1C08E27.crt ~ # rpm -qlp /var/cache/zypp/packages/repo-update/x86_64/kernel-default-5.3.18-lp152.33.1.x86_64.rpm /var/cache/zypp/packages/repo-update/x86_64/kernel-default-5.3.18-lp152.36.1.x86_64.rpm | grep crt /etc/uefi/certs/F1C08E27.crt /etc/uefi/certs/33CEA71B.crt

We've got two signkeys here with the same subject and slightly different serial number and validity. I guess it depends on the OBS project the kernel's built in, but it's intransparent and confusing for users. == /etc/uefi/certs/33CEA71B.crt == issuer=CN = openSUSE Secure Boot CA, C = DE, L = Nuremberg, O = openSUSE Project, emailAddress = build@opensuse.org serial=FABED8BF409A5E63 subject=CN = openSUSE Secure Boot Signkey, C = DE, L = Nuremberg, O = openSUSE Project, emailAddress = build@opensuse.org notBefore=Aug 3 12:35:39 2020 GMT notAfter=Jun 12 12:35:39 2030 GMT SHA1 Fingerprint=33:CE:A7:1B:A1:6A:67:BA:D8:6B:93:CB:DA:2E:19:A7:E9:DF:95:66 X509v3 Subject Key Identifier: C8:BD:C7:AC:1A:1D:85:96:62:17:FD:93:EB:FC:14:F4:A2:00:B8:14 == /etc/uefi/certs/F1C08E27.crt == issuer=CN = openSUSE Secure Boot CA, C = DE, L = Nuremberg, O = openSUSE Project, emailAddress = build@opensuse.org serial=FABED8BF409A5E60 subject=CN = openSUSE Secure Boot Signkey, C = DE, L = Nuremberg, O = openSUSE Project, emailAddress = build@opensuse.org notBefore=Jan 8 16:25:54 2020 GMT notAfter=Nov 16 16:25:54 2029 GMT SHA1 Fingerprint=F1:C0:8E:27:F3:5F:3F:39:C5:5C:CB:F8:58:18:C6:27:DF:49:4E:34 X509v3 Subject Key Identifier: 03:32:FA:9C:BF:0D:88:BF:21:92:4B:0D:E8:2A:09:A5:4D:5D:EF:C8

...

I forced Yast to reinstall them, rebooted, and its now working (not SecureBoot).

The forced reinstall may have caused harm. When you reinstall (and the version number of the nvidia-gfxG05-kmp-default package is the same as before), the installation procedure will generate a new key with the same file name, overwriting the previous one. Because of rpm scriptlet ordering (%postun is executed after %post), this might lead to the old key remaining present and the new one not being enrolled. I'm not 100% certain about this, but I guess it could happen. It's safer to delete the package and install it again. I'm not saying this is the user's fault. It's rather a corner case that we may have overlooked. It does not apply to the kernel packages.

Reinstalling OS won't change MokList. The most reliable way is to reset the firmware to factory default. You'll need to restore the boot entries after that.

Isn't this is an overkill recommendation? It should be only a last resort measure. Perhaps MokManager should have an option to reset the MokList only. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c51 --- Comment #51 from Martin Wilck --- (In reply to Martin Wilck from comment #48)

The forced reinstall may have caused harm. When you reinstall (and the version number of the nvidia-gfxG05-kmp-default package is the same as before), the installation procedure will generate a new key with the same file name, overwriting the previous one. Because of rpm scriptlet ordering (%postun is executed after %post), this might lead to the old key remaining present and the new one not being enrolled.

Probably not, see bug 1176146. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c53 --- Comment #53 from Gary Ching-Pang Lin --- (In reply to Martin Wilck from comment #48)

(In reply to Gary Ching-Pang Lin from comment #45)

...

I guess the update of kernel triggered the enrollment of new signkey and the deletion of the old signkey. But somehow MokNew was gone, so MokManager thought that there was a reset.

How would MokNew go away? The scriptlets in the kernel rpm are written so that the keys won't be deleted as long as the system still has one kernel installed that needs them.

MokManager deletes the MokNew after loading it. This is a bug should be fixed. It seems to me that MokManager was loaded but got stuck or just failed to draw the screen. John, Have you seen the MokManager screen with Secure Boot enabled before? I wonder if it's a drawing problem. [snip]

...

...

I forced Yast to reinstall them, rebooted, and its now working (not SecureBoot).

The forced reinstall may have caused harm. When you reinstall (and the version number of the nvidia-gfxG05-kmp-default package is the same as before), the installation procedure will generate a new key with the same file name, overwriting the previous one. Because of rpm scriptlet ordering (%postun is executed after %post), this might lead to the old key remaining present and the new one not being enrolled. I'm not 100% certain about this, but I guess it could happen. It's safer to delete the package and install it again.

I'm not saying this is the user's fault. It's rather a corner case that we may have overlooked. It does not apply to the kernel packages.

...

Reinstalling OS won't change MokList. The most reliable way is to reset the firmware to factory default. You'll need to restore the boot entries after that.

Isn't this is an overkill recommendation? It should be only a last resort measure. Perhaps MokManager should have an option to reset the MokList only.

Although mokutil can create a reset request, it seems MokManager cannot be launched reliably, and that's why I recommend a firmware reset. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c55 --- Comment #55 from Martin Wilck --- (In reply to Neil Rickert from comment #52)

That cannot actually work. I might have more than one openSUSE system installed on my computer, and those systems all share the same list of enrolled keys. The rpm scripts can only check what is needed for the particular system where it is running.

Good point. But solving this correctly is tough. See comment 49, there are cases where we *want* to be sure to purge keys from the MokList. We can't ensure that, and at the same time keep all keys that other installations would want to have around. I guess the only way to solve it would be to have per-installation MokList variables, which would be a major implementation effort. Meanwhile Gary explained to me that for booting the *kernel*, having the keys in the MokList is actually not necessary, because shim has these sign keys built-in anyway. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c57 --- Comment #57 from Martin Wilck --- (In reply to John Shaw from comment #43)

This seemed to be working on my machine after the last round of grub2 updates went through, then this morning a new nVidia driver update came (along with about 20 other updates). On reboot with SecureBoot enabled, the machine hung after post with a black screen. I rebooted with SecureBoot disabled (ie, selected OtherOS in the ASUS bios), and it brought up the MOK Manager. However, unlike before the options are now different. There is no "enrole MOK", just: Continue, Reset MOK, Delete MOK, and enrole key and enrole hash from disk.

I think I understand this now, at least partially. The explanation is similar to what I wrote in comment 48. Can you confirm that the *version* of the Nvidia driver was different, with only the *release* differing? Because the driver package was using only the *version* (and not the release) in the certificate file name, the package update would create both an import and a deletion request for the same (new!) key at the same time, rather than a delete request for the old and an import request for the new one. Not sure how MokManager would deal with that, but your comment shows that it offered the key only for deletion. The latest Nvidia package fixes this by adding the release to the certificate file name. If it's only that, you should be able to recover from the problem by booting into a non-graphical environment, uninstalling the nvidia-gfx0$X package, and installing it again. I don't understand why any of this would cause your GRUB menu not to be shown. Normally these keys wouldn't be used until user space tries to load the nvidia driver. By that time, you should be past grub and past lots of boot messages from the kernel. The only explanation I have for that is that mokmanager might crash while trying to deal with the "contradictory" mok requests. (In reply to John Shaw from comment #37)

(In reply to Martin Wilck from comment #36)

...

The question is if the BIOS fails to verify the shim, or shim fails to verify any of the follow-up binaries. Can you check this e.g. with the UEFI shell?

I don't know how to check this in the UEFI shell. Can you explain?

If you don't have an existing UEFI shell boot entry already, grab one. See e.g.

https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#O...

Next you need to copy this to your EFI boot partition, and find a way to make your BIOS boot it. Either the BIOS has a way to boot a specified EFI binary directly, or you could use efibootmgr to craft a suitable entry for it. Once in the shell, you can move around and run commands a bit like on good old DOS. You may have to enter a "drive" first, which would look like "fs0:". The "map" command can be used to see drives that the shell has recognized. The test would look roughly like this:

fs0: cd \EFI\BOOT bootx64.efi # or whatever else you want to boot

The advantage is that you would see some error messages in this environment, e.g. if the binary's signature can't be verified by the BIOS. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c59 --- Comment #59 from John Shaw --- (In reply to Martin Wilck from comment #57)

(In reply to John Shaw from comment #43)

...

This seemed to be working on my machine after the last round of grub2 updates went through, then this morning a new nVidia driver update came (along with about 20 other updates). On reboot with SecureBoot enabled, the machine hung after post with a black screen. I rebooted with SecureBoot disabled (ie, selected OtherOS in the ASUS bios), and it brought up the MOK Manager. However, unlike before the options are now different. There is no "enrole MOK", just: Continue, Reset MOK, Delete MOK, and enrole key and enrole hash from disk.

I think I understand this now, at least partially. The explanation is similar to what I wrote in comment 48. Can you confirm that the *version* of the Nvidia driver was different, with only the *release* differing? Because the driver package was using only the *version* (and not the release) in the certificate file name, the package update would create both an import and a deletion request for the same (new!) key at the same time, rather than a delete request for the old and an import request for the new one. Not sure how MokManager would deal with that, but your comment shows that it offered the key only for deletion. The latest Nvidia package fixes this by adding the release to the certificate file name.

If it's only that, you should be able to recover from the problem by booting into a non-graphical environment, uninstalling the nvidia-gfx0$X package, and installing it again.

I don't understand why any of this would cause your GRUB menu not to be shown. Normally these keys wouldn't be used until user space tries to load the nvidia driver. By that time, you should be past grub and past lots of boot messages from the kernel. The only explanation I have for that is that mokmanager might crash while trying to deal with the "contradictory" mok requests.

(In reply to John Shaw from comment #37)

...

(In reply to Martin Wilck from comment #36)

...

The question is if the BIOS fails to verify the shim, or shim fails to verify any of the follow-up binaries. Can you check this e.g. with the UEFI shell?

I don't know how to check this in the UEFI shell. Can you explain?

If you don't have an existing UEFI shell boot entry already, grab one. See e.g.

...

https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#O...

Next you need to copy this to your EFI boot partition, and find a way to make your BIOS boot it. Either the BIOS has a way to boot a specified EFI binary directly, or you could use efibootmgr to craft a suitable entry for it.

Once in the shell, you can move around and run commands a bit like on good old DOS. You may have to enter a "drive" first, which would look like "fs0:". The "map" command can be used to see drives that the shell has recognized. The test would look roughly like this:

...

fs0: cd \EFI\BOOT bootx64.efi # or whatever else you want to boot

The advantage is that you would see some error messages in this environment, e.g. if the binary's signature can't be verified by the BIOS.

Unfortunately I can't really confirm what the exact version was for the nVidia update that came through. I tried hunting thoughout the saved snapshots, but only succeeded in hanging the yast snapshot tool after looking for differences between saved pairs. This is the first time that has happened, it may be that there were a LOT of differences in the pair I was examining. The current version for the nVidia driver is 450.66, but I doubt that helps much. I finally found compiled efi shell (from the Arch Linux repos) and copied it to by UEFI partition. I should be able to get into to from the BIOS. I can try to boot the opensusi shim.efi file and see what happens. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c61 --- Comment #61 from John Shaw --- (In reply to Martin Wilck from comment #60)

(In reply to John Shaw from comment #59)

...

Unfortunately I can't really confirm what the exact version was for the nVidia update that came through. I tried hunting thoughout the saved snapshots, but only succeeded in hanging the yast snapshot tool after looking for differences between saved pairs. This is the first time that has happened, it may be that there were a LOT of differences in the pair I was examining. The current version for the nVidia driver is 450.66, but I doubt that helps much.

You chose quite a complex method to figure this out ;-) I'd just run "grep -i nvidia /var/log/zypp/history"; possibly prepend /.snapshots/$X/snapshot with the snapshot where the problem first occured.

...

I finally found compiled efi shell (from the Arch Linux repos) and copied it to by UEFI partition. I should be able to get into to from the BIOS. I can try to boot the opensusi shim.efi file and see what happens.

Thanks a lot for your cooperation!

From what I can tell, it looks like the driver went from 450.57 to 450.66 which would be a version change.

The current state of the machine (which I need for work, so I can't afford to screw it up too much;) is that if I enable SecureBoot, it DOES now flash what is likely a MOK message (what else could it be?). This is almost certainly due to the nVidia driver. However, flash is the operative word. I cannot read it and I cannot catch it with a key press. I tried a couple of times, but the message is up for maybe 1/10th of a second. What this thing needs is to pause for a few seconds. The boot process at least gets past this and drops into grub. If I boot suse, it fails to load the nvidia driver. I tried to reinstall the driver, but I can't find a way to get MOK to enroll it. Booting with SecureBoot disabled still works. I could not boot the shellx86.efi from my /boot/efi partition. The bios gives the option to boot it from a USB drive, so I guess I can try that. As you suggest, I could try and add a boot entry with efibootmgr, but the USB option is safer. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c63 --- Comment #63 from John Shaw ---

Be careful with "reinstalling", as noted above that procedure is a bit flawed in the current release. It's better to uninstall and install again.

Please provide output of mokutil --list-enrolled, mokutil --list-new, mokutil --list-delete.

All are empty. I shut my machine off every night and reboot in the morning with SecureBoot disabled. Updates come through in that mode, so I wonder if they don't try to setup the certificates/keys. Although I can sometimes boot with SecureBoot (I tried again a couple of days ago), I cannot get the nVidia driver to be enrolled. This is not an option for me as I write Cuda-based codes. I count not care less about having SecureBoot (why would anyone?), but I figured I would try to help debug this issue. The certificate file is available for the nvidia driver, so I can be imported with mokutil again. This may force MOK to pause and let me enroll it. If there is nothing to enroll, what is MOK Manager supposed to do on a boot? Is its supposed to: a) do nothing and pass control to grub; b) flash MOC screen and move on; or c) show MOC screen and wait a few seconds?

...

Booting with SecureBoot disabled still works.

I could not boot the shellx86.efi from my /boot/efi partition. The bios gives the option to boot it from a USB drive, so I guess I can try that. As you suggest, I could try and add a boot entry with efibootmgr, but the USB option is safer.

I should probably get out of X11 and full uninstall the drivers, and reinstall them.

Sorry that this is such a hassle for you. Using efibootmgr for this is actually not so dangerous because it will create a new boot entry, and if that one can't be booted, the EFI bootloader will fall back to the next working one. But I can understand that you're wary after your recent experience.

I'd do it roughly like this (note that you need to adapt disk (-d), partition (-p) and path (-l)):

...

efibootmgr -c -L 'UEFI Shell' -d /dev/sda -p 1 -l '\EFI\shell\shellx64.efi'

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c65 --- Comment #65 from Martin Wilck --- (In reply to John Shaw from comment #63)

All are empty. I shut my machine off every night and reboot in the morning with SecureBoot disabled. Updates come through in that mode, so I wonder if they don't try to setup the certificates/keys.

This is how it's designed, yes. If the nvidia driver is installed with SB disabled, it will not try to either generate or enroll any keys. You need to uninstall the drivers, enable SB, reboot, and install them again.

Although I can sometimes boot with SecureBoot (I tried again a couple of days ago), I cannot get the nVidia driver to be enrolled.

That makes no sense to me. With SB, a driver can't be sometimes loadable and sometimes not. If the key hasn't been enrolled, it's not loadable, period. Unless you are booting a Leap 15.1 kernel, that is.

This is not an option for me as I write Cuda-based codes.

I count not care less about having SecureBoot (why would anyone?), but I figured I would try to help debug this issue. The certificate file is available for the nvidia driver, so I can be imported with mokutil again.

Hm. If the driver was installed without SB as you wrote above, it wouldn't generate a key. If there is still a key file around, it's likely from a previous installed version of the driver and won't work with the current drivers you have.

This may force MOK to pause and let me enroll it. If there is nothing to enroll, what is MOK Manager supposed to do on a boot? Is its supposed to: a) do nothing and pass control to grub; b) flash MOC screen and move on; or c) show MOC screen and wait a few seconds?

It's a). Otherwise we'd see the mok screen on every boot. (In reply to John Shaw from comment #64)

I imported the nVidia certificate for the latest driver, so it shows up in with --list-new. This should force MOK manager to attempt to enroll it. If I boot with SecureBoot enabled, it hangs BUT the first time I tried I had a mostly black screen, with green "noise" along the top. This was the first time I saw this phenomena, and it looks like the video driver (?) crashes when trying to display the MOK screen. Repeating the boot resulted in a pure black screen, so I guess the green noise is a random thing.

At the time MokManager is running, the EFI environment of your BIOS is responsible to display the screen. It could be that this fails. MokManager uses EFI calls to switch the console to *from graphics to text mode*, actually; perhaps your EFI BIOS has an issue with text mode? It wouldn't be the first BIOS that fails at simple text mode. Whatever goes wrong here, I don't see how it could be related to the fact that you installed the Linux nvidia drivers and enrolled a certificate for them. https://github.com/rhboot/shim/blob/master/lib/console.c https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_final.pdf (§12) If you have that black/green screen, you don't know what's going on of course. Normally Mokmanager would wait for a key press for some time, then continue normal boot without enrolling anything. But it deletes any certs in MokNew IIRC, even if they weren't enrolled, so it's normal behavior that at subsequent boots the MokManager screen wouldn't show up again.

My machine has two nVidia cards in it. A GTX 950 drives two monitors, and a Quadro GV100 is used for Cuda development. This is not "normal" for a home computer (the GV100 costs ~$8.5k).

Both cards using the same driver?

This may be the difference causing the problem, but I cannot imaging why, as the BIOS display clearly has no problem with it and it has nothing to do with nVidia drivers. Maybe the way MOK Manager writes or initializes the display has an issue if multiple graphics cards are present. If this is the case, maybe this is not worth pursuing at your end.

If it's no big hassle for you, you might try if the problem goes away if you remove or disable the GV100 card.

Anyway, if I then boot with SecureBoot disabled, MOK Manager comes up, but it does not offer to enroll anything (ie, the nVidia certificate).

Probably the cert has been removed from MokNew by that time. (Does mokutil --list-new still show them?).

I can continue, reset MOK, or import (?) keys or hashes for files. In either of the last two cases, I cannot get to the certificate file (.der) for the nVidia driver, but I can navigate the UEFI directory. I guess I could try to import hashes for some of the \efi\opensuse\*.bin files.

You could try copying the nvidia .der files to some path below \EFI aka /boot/efi/EFI. (doesn't have to be \EFI\opensuse).

Presumably I need at least shim.efi, grubx64.efi, and MokManager.efi. Anything else? Clearly none of this matters with SecureBoot disabled.

No certificates should be needed for any of these. shim is signed by Microsoft, and has built-in certificates for verifying the further steps in the openSUSE boot chain. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1175626 http://bugzilla.opensuse.org/show_bug.cgi?id=1175626#c67 Martin Wilck changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(glin@suse.com) --- Comment #67 from Martin Wilck --- (In reply to John Shaw from comment #66)

The current certificate is MOK-nvidia-gfxG05-450.66-default.der, which is the same version as the driver. Presumably the actual keys would be different from install to install

yes.

...

At the time MokManager is running, the EFI environment of your BIOS is responsible to display the screen. It could be that this fails. MokManager uses EFI calls to switch the console to *from graphics to text mode*, actually; perhaps your EFI BIOS has an issue with text mode? It wouldn't be the first BIOS that fails at simple text mode. Whatever goes wrong here, I don't see how it could be related to the fact that you installed the Linux nvidia drivers and enrolled a certificate for them.

If its the underlying EFI BIOS that buggy, there is nothing to be done. I have the latest BIOS for my MB, but it was issued about 2 years ago. Its unlikely ASUS will issue any more updates for it, but I will keep watch for one. People running Windows have to occasionally start in safe mode, or bring up the ntldr boot menu for whatever reason. That would seem to be a screen in simple text mode, and by this point many many people would have reported if it did not work with this MB (ASUS z170A).

Is it possible that there is some simple bug in console.c that could account for this? (eg, a timing issue?) Just a thought.

Possible? Certainly. Likely? I doubt it, but I have to say my knowledge about these APIs is close to 0. It's also possible that MokManager dies and leaves the screen in an unhealthy condition (like old DOS games did when they crashed). Gary?

...

Both cards using the same driver?

yes, nVidia used to have separate drivers for consumer and pro products, but now there is just a single unified driver. 450.66 is the current long-lived stable one. Its officially supports both cards, so there is nothing funny here.

Well it COULD be that changing the graphics mode might confuse the BIOS somehow if there are multiple graphics cards. Just a wild guess, and I wouldn't know what to do about it.

...

If it's no big hassle for you, you might try if the problem goes away if you remove or disable the GV100 card.

Let me think about it (ie, its a hassle;). I don't really believe this has anything to do with anything, but I thought I would mention it. The theory about a bug in the BIOS being able to switch to text only mode makes the most sense to me.

Just strange that that happens only with SB on. In non-SB mode, MokManager seems to start just fine.

There seems to be no way to show a "simple" bios in text-only mode anymore. In the past, I have been able to run memtest86, which looks like it to has a text interface. I can try to boot that and make sure it still works.

I don't think the standard version works with UEFI. You need to get a special version of it.

Out of curiosity, why does MokManager come up if SB is disabled? Presumably there is nothing meaningful for it to do.

Excellent question. I guess it *can* enroll certificates in this mode (after all, MokList is just a regular boot-service UEFI variable). They just won't be used. From a paranoid point of view, I guess the MOK variables would need to be reset and everything re-enrolled after switching SB off and on. After all, with SB off, any untrusted EFI program could change the list at will.

For users who disable SB, this is just a confusing screen page. I would suggest that MokManager kick out immediately and pass control to grub if it determines SB is is not in effect. I can confirm the Linux believes SB is disabled when I disable it.

No dispute over the lousy and confusing UI. Gary, please comment. -- You are receiving this mail because: You are on the CC list for the bug.