Build 20230505 Kernel LockDown
I just updated from 20230413 to 20230505 and after rebooting vmware kernel modules ( after recompiling for kernel 6.3.1-1 ) would not load. modprobe: ERROR: could not insert 'vmmon': Key was rejected by service modprobe: ERROR: could not insert 'vmnet': Key was rejected by service I have not see an announcement regarding kernel lockdown being enabled again, but those messages seem to indicate it was. Where should I be looking for that kind of announcement ? From my notes when kernel lockdown was previously enabled I had: cat /sys/kernel/security/lockdown but from the return value it does not appear to be locked down ??? [none] integrity confidentiality How to check if secondary mok keys are allowed zcat /proc/config.gz | grep CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY but from that return value it also appears that the kernel does not have support for user signed keys # CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY is not set Since the mobprobe error seems to indicate that the modules could not be loaded because they were not signed, I went ahead Verified secure boot is enabled via mokutil --sb-state which confirms "SecureBoot enabled" Compiled the vmware modules ( just to double check that there were not compile issues ) Generated a signing key Signed both modules modutil --import *.der file reboot Perform MOK Management / Enroll MOK / Enroll the key I added reboot mokutil --list-enrolled now shows my key is enrolled systemctl start vmware now starts the vmware service and vmware is working. So it definitely appears that kernel lockdown was enabled ( how did I miss that announcement ??? ) but why do both of these seem to indicate that it is not ??? cat /sys/kernel/security/lockdown zcat /proc/config.gz | grep CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY What's different this time when it was enabled ??? -- Regards, Joe
I do not know what changed but got the same modprobe failure with NVIDIA modules. As a workaround, I disabled Secure Boot in the BIOS because I do not absolutely need it. On 5/7/23 20:25, Joe Salmeri wrote:
I just updated from 20230413 to 20230505 and after rebooting vmware kernel modules ( after recompiling for kernel 6.3.1-1 ) would not load.
modprobe: ERROR: could not insert 'vmmon': Key was rejected by service modprobe: ERROR: could not insert 'vmnet': Key was rejected by service
Hi Michael, On 5/7/23 16:21, Michael Pujos wrote:
I do not know what changed but got the same modprobe failure with NVIDIA modules. As a workaround, I disabled Secure Boot in the BIOS because I do not absolutely need it.
It "appears" to me that they resolved the issues with self signed certificates which existed last time they turned on kernel lockdown, but I don't yet understand why the things I used to check to see if lockdown was enabled still say no. To use self signed modules with Secure Boot you must Create your own certificates ( using openssl ) Sign the modules with those certificates ( using the appropriate sign-file script for the kernel ) mokutil --import <your-cert>.der reboot When prompted during boot you need to Perform MOK Management Enroll MOK Enroll the key reboot mokutil --list-enrolled # should now show your key enrolled modules should have now loaded without the key error Next time there is a kernel update you can just sign the modules with your certifcate and then they should load and you don't have to do the other steps because you are using the same certificate. NOTE: I don't use Nvidia drivers but I would have expected distro to have signed them with their key Regards, Joe
On 07.05.2023 21:25, Joe Salmeri wrote:
I just updated from 20230413 to 20230505 and after rebooting vmware kernel modules ( after recompiling for kernel 6.3.1-1 ) would not load.
modprobe: ERROR: could not insert 'vmmon': Key was rejected by service modprobe: ERROR: could not insert 'vmnet': Key was rejected by service
Post kernel messages (dmesg output) at the time when you see this error. Even better in such cases is to provide full dmesg output.
I have not see an announcement regarding kernel lockdown being enabled
Where do you see any indication of kernel lockdown?
again, but those messages seem to indicate it was.
Where should I be looking for that kind of announcement ?
From my notes when kernel lockdown was previously enabled I had:
cat /sys/kernel/security/lockdown
but from the return value it does not appear to be locked down ???
[none] integrity confidentiality
How to check if secondary mok keys are allowed
zcat /proc/config.gz | grep CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY
but from that return value it also appears that the kernel does not have support for user signed keys
# CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY is not set
Since the mobprobe error seems to indicate that the modules could not be loaded because they were not signed, I went ahead
Verified secure boot is enabled via mokutil --sb-state which confirms "SecureBoot enabled"
Compiled the vmware modules ( just to double check that there were not compile issues )
Generated a signing key
Signed both modules
modutil --import *.der file
reboot
Perform MOK Management / Enroll MOK / Enroll the key I added
reboot
mokutil --list-enrolled now shows my key is enrolled
systemctl start vmware now starts the vmware service and vmware is working.
So it definitely appears that kernel lockdown was enabled ( how did I miss that announcement ??? ) but why do both of these seem to indicate that it is not ???
cat /sys/kernel/security/lockdown
zcat /proc/config.gz | grep CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY
What's different this time when it was enabled ???
HI Andrei On 5/8/23 00:50, Andrei Borzenkov wrote:
On 07.05.2023 21:25, Joe Salmeri wrote:
I just updated from 20230413 to 20230505 and after rebooting vmware kernel modules ( after recompiling for kernel 6.3.1-1 ) would not load.
modprobe: ERROR: could not insert 'vmmon': Key was rejected by service modprobe: ERROR: could not insert 'vmnet': Key was rejected by service
Post kernel messages (dmesg output) at the time when you see this error. Even better in such cases is to provide full dmesg output.
I interpreted the 2 modprobe messages to indicate that since the 2 vmware modules were not signed they were rejected. The same errors were thrown in my test environment and in my main desktop. As a test for you, I recompiled both modules again in my test environment, but this time I did NOT sign them. I ran 'journalctl -k -xef' in one konsole session and in another konsole session I ran 'systemctl start vmware' The only 2 messages that were displayed were May 08 10:17:56 Test-VM: Loading of unsigned module is rejected May 08 10:17:56 Test-VM: Loading of unsigned module is rejected So that seems to indicate that the kernel was changed back to requiring signed modules. If I boot back to the previous kernel which was 6.2.10-1, then the unsigned vmware modules load fine. If I follow the steps in my original message and sign the vmware modules, then kernel 6.3.1.1 loads the modules, the service starts and vmware works.
I have not see an announcement regarding kernel lockdown being enabled
Where do you see any indication of kernel lockdown?
Everything I had noted from the last time that kernel lockdown was enabled, indicates that it is NOT locked down with 6.3.1-1. I don't understand why everything indicates that lockdown is NOT enabled, however, the actual messages from the journal/dmesg indicate that unsigned modules can no longer be loaded with the 6.3.1-1 kernel and signing them allows them to load so clearly something changed. Maybe they found a way to deal with the previous issues that occurred back with the 6.2 kernel where lockdown would not allow signing with secondary keys and loading the modules? Hopefully Jira,will see these messages and chime in.... -- Regards, Joe
On 08.05.2023 17:29, Joe Salmeri wrote:
HI Andrei
On 5/8/23 00:50, Andrei Borzenkov wrote:
On 07.05.2023 21:25, Joe Salmeri wrote:
I just updated from 20230413 to 20230505 and after rebooting vmware kernel modules ( after recompiling for kernel 6.3.1-1 ) would not load.
modprobe: ERROR: could not insert 'vmmon': Key was rejected by service modprobe: ERROR: could not insert 'vmnet': Key was rejected by service
Post kernel messages (dmesg output) at the time when you see this error. Even better in such cases is to provide full dmesg output.
I interpreted the 2 modprobe messages to indicate that since the 2 vmware modules were not signed they were rejected.
The same errors were thrown in my test environment and in my main desktop.
As a test for you, I recompiled both modules again in my test environment, but this time I did NOT sign them.
I ran 'journalctl -k -xef' in one konsole session and in another konsole session I ran 'systemctl start vmware'
The only 2 messages that were displayed were
May 08 10:17:56 Test-VM: Loading of unsigned module is rejected May 08 10:17:56 Test-VM: Loading of unsigned module is rejected
So that seems to indicate that the kernel was changed back to requiring signed modules.
Yes, although it probably is not exactly lockdown. It is due to CONFIG_IMA_ARCH_POLICY=y which was set by this commit: commit 90a46594a115a4abf9381bd4c327fd875ac0da0b Author: Lee, Chun-Yi <jlee@suse.com> Date: Thu Mar 9 13:25:10 2023 +0800 Update config files. Add the following config to x86_64, arm64 and i386. CONFIG_IMA_ARCH_POLICY=y CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT=y This config be used to detect secure boot. (bsc#1209006) There is no way to override it at run time (except disabling secure boot). Cc kernel.
If I boot back to the previous kernel which was 6.2.10-1, then the unsigned vmware modules load fine.
If I follow the steps in my original message and sign the vmware modules, then kernel 6.3.1.1 loads the modules, the service starts and vmware works.
I have not see an announcement regarding kernel lockdown being enabled
Where do you see any indication of kernel lockdown?
Everything I had noted from the last time that kernel lockdown was enabled, indicates that it is NOT locked down with 6.3.1-1.
I don't understand why everything indicates that lockdown is NOT enabled, however, the actual messages from the journal/dmesg indicate that unsigned modules can no longer be loaded with the 6.3.1-1 kernel and signing them allows them to load so clearly something changed.
Maybe they found a way to deal with the previous issues that occurred back with the 6.2 kernel where lockdown would not allow signing with secondary keys and loading the modules?
Hopefully Jira,will see these messages and chime in....
As a test for you, I recompiled both modules again in my test
environment, but this time I did NOT sign them.
I ran 'journalctl -k -xef' in one konsole session and in another konsole session I ran 'systemctl start vmware'
The only 2 messages that were displayed were
May 08 10:17:56 Test-VM: Loading of unsigned module is rejected May 08 10:17:56 Test-VM: Loading of unsigned module is rejected
So that seems to indicate that the kernel was changed back to requiring signed modules.
Yes, although it probably is not exactly lockdown.
It is due to CONFIG_IMA_ARCH_POLICY=y which was set by this commit:
commit 90a46594a115a4abf9381bd4c327fd875ac0da0b Author: Lee, Chun-Yi <jlee@suse.com> Date: Thu Mar 9 13:25:10 2023 +0800
Update config files.
Add the following config to x86_64, arm64 and i386.
CONFIG_IMA_ARCH_POLICY=y CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT=y
This config be used to detect secure boot. (bsc#1209006)
There is no way to override it at run time (except disabling secure boot).
Ok, so that explains why the indicators I was checking regarding lockdown still seem to say it is NOT locked down. I prefer to leave secure boot turned on. Fortunately signing the vmware kernel modules seems to do the trick in both my test vm machine and also my desktop, but I still wonder if/where there was any notice of this new requirement as I couldn't find one. Regards, Joe
On 08. 05. 23, 19:52, Joe Salmeri wrote:
As a test for you, I recompiled both modules again in my test
environment, but this time I did NOT sign them.
I ran 'journalctl -k -xef' in one konsole session and in another konsole session I ran 'systemctl start vmware'
The only 2 messages that were displayed were
May 08 10:17:56 Test-VM: Loading of unsigned module is rejected May 08 10:17:56 Test-VM: Loading of unsigned module is rejected
So that seems to indicate that the kernel was changed back to requiring signed modules.
Yes, although it probably is not exactly lockdown.
It is due to CONFIG_IMA_ARCH_POLICY=y which was set by this commit:
commit 90a46594a115a4abf9381bd4c327fd875ac0da0b Author: Lee, Chun-Yi <jlee@suse.com> Date: Thu Mar 9 13:25:10 2023 +0800
Update config files.
Add the following config to x86_64, arm64 and i386.
CONFIG_IMA_ARCH_POLICY=y CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT=y
This config be used to detect secure boot. (bsc#1209006)
There is no way to override it at run time (except disabling secure boot).
Ok, so that explains why the indicators I was checking regarding lockdown still seem to say it is NOT locked down.
I prefer to leave secure boot turned on.
Fortunately signing the vmware kernel modules seems to do the trick in both my test vm machine and also my desktop, but I still wonder if/where there was any notice of this new requirement as I couldn't find one.
No, this went in unnoticed. Now reverted. At the very least, we still miss a documentation how to deal with this. I.e. how to sign a module and insert the key into MOK (answering people these questions in random threads on this list is not the right way™). thanks, -- js suse labs
Hi Jira, On 5/9/23 06:25, Jiri Slaby wrote:
Yes, although it probably is not exactly lockdown.
It is due to CONFIG_IMA_ARCH_POLICY=y which was set by this commit:
commit 90a46594a115a4abf9381bd4c327fd875ac0da0b Author: Lee, Chun-Yi <jlee@suse.com> Date: Thu Mar 9 13:25:10 2023 +0800
Update config files.
Add the following config to x86_64, arm64 and i386.
CONFIG_IMA_ARCH_POLICY=y CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT=y
This config be used to detect secure boot. (bsc#1209006)
There is no way to override it at run time (except disabling secure boot).
Ok, so that explains why the indicators I was checking regarding lockdown still seem to say it is NOT locked down.
I prefer to leave secure boot turned on.
Fortunately signing the vmware kernel modules seems to do the trick in both my test vm machine and also my desktop, but I still wonder if/where there was any notice of this new requirement as I couldn't find one.
No, this went in unnoticed. Now reverted.
Thanks for chiming in. Which TW build will have the change reverted ?
At the very least, we still miss a documentation how to deal with this. I.e. how to sign a module and insert the key into MOK (answering people these questions in random threads on this list is not the right way™).
I agree it needs to be formally documented somewhere. FWIW, I just followed the normal steps for generating self signing keys, signing both vmware modules, and inserting the key to the MOK and after rebooting the vmware modules successfully loaded and worked. This is with kernel 6.3.1-1-default. It is unclear to me if the full lockdown features were enabled but at least with what was enabled this time, self signed modules for vmware works. Regards, Joe
On 09. 05. 23, 15:55, Joe Salmeri wrote:
No, this went in unnoticed. Now reverted.
Thanks for chiming in. Which TW build will have the change reverted ?
Some time after the SR: https://build.opensuse.org/request/show/1085678 and QA of the staging succeeds: https://openqa.opensuse.org/tests?hoursfresh=24&match=Staging:I Sorry, I don't have exact date. -- js suse labs
participants (4)
-
Andrei Borzenkov -
Jiri Slaby -
Joe Salmeri -
Michael Pujos