I'm working on an issue with my System76 system not being able to apply the firmware update that updates the UEFI dbx to version 371 (the version that resolves the issue resulting from the discovery of the BlackLotus UEFI bootkit). The issue I'm having is that efivarfs is running out of space. I'm being advised to increase the size of my UEFI partition - which is 4 GB and has 158 MB free - for a firmware update that is 21.2 KB in size, and the output from fwupdmgr is telling me that efivarfs doesn't have enough free space (it has 20KB free). There's nothing in the messages that suggests that I need additional space in the UEFI partition on my system - the specific message I get is: failed to write data to efivarfs: Error writing to file descriptor: No space left on device There's no reason to be messing about with my UEFI partition at all, is there? (They had also suggested using efibootmgr to remove some entries, and that does clear out some of the Boot0* files in /sys/firmware/efi/ efivars - but not enough). The issue is with the constraints in the nvram where this data is stored itself, and not the UEFI boot partition on my SSD, correct? Just want to make sure I'm completely understanding the message I'm seeing. (As a "bonus question", messing around with just deleting stuff like the dbx files in efivarfs could potentially brick the system, right?) -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 20.03.2024 19:41, Jim Henderson wrote:
I'm working on an issue with my System76 system not being able to apply the firmware update that updates the UEFI dbx to version 371 (the version that resolves the issue resulting from the discovery of the BlackLotus UEFI bootkit).
The issue I'm having is that efivarfs is running out of space.
I'm being advised to increase the size of my UEFI partition - which is 4 GB and has 158 MB free - for a firmware update that is 21.2 KB in size, and the output from fwupdmgr is telling me that efivarfs doesn't have enough free space (it has 20KB free).
Oh, I was not aware that it can show it. Do you have anything in dmesg? Storage space in UEFI is not large, so several KB does make a difference.
There's nothing in the messages that suggests that I need additional space in the UEFI partition on my system - the specific message I get is:
failed to write data to efivarfs: Error writing to file descriptor: No space left on device
There's no reason to be messing about with my UEFI partition at all, is there? (They had also suggested using efibootmgr to remove some entries, and that does clear out some of the Boot0* files in /sys/firmware/efi/ efivars - but not enough).
The issue is with the constraints in the nvram where this data is stored itself, and not the UEFI boot partition on my SSD, correct?
Yes.
Just want to make sure I'm completely understanding the message I'm seeing.
(As a "bonus question", messing around with just deleting stuff like the dbx files in efivarfs could potentially brick the system, right?)
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows. What says df -h /sys/firmware/efi/efivars/
On Wed, 20 Mar 2024 20:23:29 +0300, Andrei Borzenkov wrote:
I'm being advised to increase the size of my UEFI partition - which is 4 GB and has 158 MB free - for a firmware update that is 21.2 KB in size, and the output from fwupdmgr is telling me that efivarfs doesn't have enough free space (it has 20KB free).
Oh, I was not aware that it can show it. Do you have anything in dmesg?
Nothing in dmesg. I determined the free space using: [jhenderson@TheEarth ~]$ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 128K 104K 20K 85% /sys/firmware/efi/efivars
Storage space in UEFI is not large, so several KB does make a difference.
Indeed it does.
The issue is with the constraints in the nvram where this data is stored itself, and not the UEFI boot partition on my SSD, correct?
Yes.
Thank you for the confirmation. I was pretty sure that was the case, but after going back and forth several times with the suggestion to "increase the size of the UEFI partition", I thought I might be missing something.
(As a "bonus question", messing around with just deleting stuff like the dbx files in efivarfs could potentially brick the system, right?)
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows.
What says
df -h /sys/firmware/efi/efivars/
(See above) I had seen something on another type of system (a Lenovo laptop) about there maybe being an option in the UEFI settings to restore factory keys - and if that option exists, that might be an approach that works. (https://github.com/fwupd/fwupd/issues/5603 is where that comment is) -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 20.03.2024 21:12, Jim Henderson wrote:
On Wed, 20 Mar 2024 20:23:29 +0300, Andrei Borzenkov wrote:
I'm being advised to increase the size of my UEFI partition - which is 4 GB and has 158 MB free - for a firmware update that is 21.2 KB in size, and the output from fwupdmgr is telling me that efivarfs doesn't have enough free space (it has 20KB free).
Oh, I was not aware that it can show it. Do you have anything in dmesg?
Nothing in dmesg. I determined the free space using:
[jhenderson@TheEarth ~]$ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 128K 104K 20K 85% /sys/firmware/efi/efivars
(Current) Linux kernel keeps safety margin of 5K. You may force it into using all available space with efi_no_storage_paranoia kernel parameter, but please keep in mind that this parameter was introduced after actual case of bricking laptops. https://mjg59.dreamwidth.org/22855.html It was made less aggressive as initially it reserved 50%, but still you never know what happens if you drive firmware to the extreme.
Storage space in UEFI is not large, so several KB does make a difference.
Indeed it does.
The issue is with the constraints in the nvram where this data is stored itself, and not the UEFI boot partition on my SSD, correct?
Yes.
Thank you for the confirmation. I was pretty sure that was the case, but after going back and forth several times with the suggestion to "increase the size of the UEFI partition", I thought I might be missing something.
(As a "bonus question", messing around with just deleting stuff like the dbx files in efivarfs could potentially brick the system, right?)
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows.
Actually you should not be able to remove dbx, as for the others ... https://www.reddit.com/r/linux/comments/43ls7j/no_post_after_rm_rf_systemd_u...
What says
df -h /sys/firmware/efi/efivars/
(See above)
I had seen something on another type of system (a Lenovo laptop) about there maybe being an option in the UEFI settings to restore factory keys - and if that option exists, that might be an approach that works.
(https://github.com/fwupd/fwupd/issues/5603 is where that comment is)
Yes, this may help, but it means you will lose all EFI configuration (boot entries, shim certificates etc).
Mr H: The joy of System76 is that they do have support for the life of their products, so you could open a ticket with them to get their response on it . . . "care and feeding" etc. I had a **similar** type of "you are running out of space" type of message from my Gaz15 awhile back, took a bit of back and forth with them to figure out it had to do with the /efi/boot partition filling up . . . because Pop! uses that partition to for the kernel . . . so it doesn't take too many of them to fill up that partition . . . which might then "bleed" into other functions?? I think I posted this info previously on this list or on the forum . . . may or may not relate . . .
On Wed, 20 Mar 2024 19:45:14 -0000, Fritz Hudnut wrote:
Mr H:
The joy of System76 is that they do have support for the life of their products, so you could open a ticket with them to get their response on it . . . "care and feeding" etc.
Indeed, I have a ticket open with them, and it's their recommendation that I resize my UEFI partition (on my hard drive) that's led to my question here, because that suggestion makes no sense for a 21.2K update, when the partition (a) isn't where the 'out of space' message is from, and (b) has 158 MB free on it already (I could free up about 3 GB by moving the Pop OS recovery image to another location temporarily, but I find it hard to believe that a 21.2 KB update needs multiple GB of space somewhere that's not identified by the error message from fwupdmgr. ;) )
I had a **similar** type of "you are running out of space" type of message from my Gaz15 awhile back, took a bit of back and forth with them to figure out it had to do with the /efi/boot partition filling up . . . because Pop! uses that partition to for the kernel . . . so it doesn't take too many of them to fill up that partition . . . which might then "bleed" into other functions??
I think I posted this info previously on this list or on the forum . . . may or may not relate . . .
I think I saw that one as well. This issue is different, because it's the nvram efivarfs space, not the efi boot partition. I think their support engineer is perhaps conflating the two spaces in the filesystem. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
OK, it does seem like you have to bounce back and forth to get the ticket "escalated" to players that have chops . . . . I just re-read the first post and I had read it as "update" . . . I did run a firmware update a couple months back that apparently was waiting since September . . . didn't hit any errors on it . . . . I could check whatever directories you are having problems with and see what's showing in my Pop install . . . if that would give you any help?? My system is still "jammy" based . . . . I ran your command . . . . [CODE] $ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 64K 6.6K 53K 12% /sys/firmware/efi/efivars [/CODE]
On Thu, 21 Mar 2024 00:59:54 -0000, Fritz Hudnut wrote:
OK, it does seem like you have to bounce back and forth to get the ticket "escalated" to players that have chops . . . .
That has been an ongoing thing. I've found their support to generally be good, and it's not a system-breaking issue for me at this point. More a minor annoyance that I get messages saying there's firmware that needs to be updated.
I just re-read the first post and I had read it as "update" . . . I did run a firmware update a couple months back that apparently was waiting since September . . . didn't hit any errors on it . . . .
I could check whatever directories you are having problems with and see what's showing in my Pop install . . . if that would give you any help?? My system is still "jammy" based . . . .
I ran your command . . . .
[CODE] $ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 64K 6.6K 53K 12% /sys/firmware/efi/efivars [/CODE]
Unless you're on a Thelio Major B3, the amount of free space is probably going to be different, as our firmwares are different. Based on that your efivars filesystem space is 64K and mine is 128K, it looks like we're on different systems. The filesystem in question is nvram, so the OS won't make a difference. (I can boot into Pop_OS - without GUI because that somehow got broken on my installation, and since I run TW as my daily driver, I haven't felt the need to try to fix the Pop installation). It's something in the firmware nvram storage that's the problem, and ultimately, just a free space issue. Partially seems to have to do with how fwupdmgr protects the system by not overwriting the existing files, but creating new ones, verifying the write was successful (it seems), and then deleting the old ones. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On Wed, 20 Mar 2024 22:05:15 +0300, Andrei Borzenkov wrote:
(Current) Linux kernel keeps safety margin of 5K. You may force it into using all available space with efi_no_storage_paranoia kernel parameter, but please keep in mind that this parameter was introduced after actual case of bricking laptops.
https://mjg59.dreamwidth.org/22855.html
It was made less aggressive as initially it reserved 50%, but still you never know what happens if you drive firmware to the extreme.
That's good to know. I'll keep that in my back pocket for further discussion with them.
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows.
Actually you should not be able to remove dbx, as for the others ...
https://www.reddit.com/r/linux/comments/43ls7j/ no_post_after_rm_rf_systemd_uefi/
Yeah, I saw reports of that kind of thing, which is why I hesitate to just start nuking things without knowing what I'm doing. ;)
I had seen something on another type of system (a Lenovo laptop) about there maybe being an option in the UEFI settings to restore factory keys - and if that option exists, that might be an approach that works.
(https://github.com/fwupd/fwupd/issues/5603 is where that comment is)
Yes, this may help, but it means you will lose all EFI configuration (boot entries, shim certificates etc).
As I don't have secure boot enabled at present, the shim certificates are probably less important, though, right? -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 20.03.2024 23:54, Jim Henderson wrote:
As I don't have secure boot enabled at present, the shim certificates are probably less important, though, right?
I wonder what takes up so much space then. What is the total size of all variables? Are there any obvious outliers? ls -l /sys/firmware/efi/efivars/ would be interesting.
On Thu, 21 Mar 2024 06:55:08 +0300, Andrei Borzenkov wrote:
On 20.03.2024 23:54, Jim Henderson wrote:
As I don't have secure boot enabled at present, the shim certificates are probably less important, though, right?
I wonder what takes up so much space then. What is the total size of all variables? Are there any obvious outliers?
ls -l /sys/firmware/efi/efivars/
would be interesting.
The dbx files are the largest, it seems: [jhenderson@TheEarth ~]$ ls -l /sys/firmware/efi/efivars/ total 0 -rw-r--r-- 1 root root 5 Mar 19 13:16 1GPageTable-356471b1-b483-42ae-b6e7-3b2ebab14e15 -rw-r--r-- 1 root root 28 Mar 19 13:16 AMITCGPPIVAR-a8a2093b-fefa-43c1-8e62-ce526847265e -rw-r--r-- 1 root root 20 Mar 19 13:16 AsusRomLayout-2e0585e9-2b5e-4f1e-bbeb-e632c5ef44b8 -rw-r--r-- 1 root root 170 Mar 19 13:16 BiosEventLog-4034591c-48ea-4cdc-864f-e7cb61cfd0f2 -rw-r--r-- 1 root root 134 Mar 19 13:16 Boot000D-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 124 Mar 19 13:16 Boot000E-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 330 Mar 19 13:16 Boot000F-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 363 Mar 19 13:16 Boot0010-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 332 Mar 19 13:16 Boot0011-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 365 Mar 19 13:16 Boot0012-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 6 Mar 19 13:16 BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 5 Mar 19 13:16 BootFromUSB-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- 1 root root 8 Mar 19 13:16 BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 16 Mar 19 13:16 BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 6 Mar 19 13:16 CMOSfailflag-c89dc9c7-5105-472c-a743-b1621e142b41 -rw-r--r-- 1 root root 160 Mar 19 13:16 ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 160 Mar 19 13:16 ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 40 Mar 19 13:16 ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 40 Mar 19 13:16 ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 5 Mar 19 13:16 CurrentPolicy-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 6326 Mar 19 13:16 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- 1 root root 6326 Mar 19 13:16 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 13144 Mar 19 13:16 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- 1 root root 3728 Mar 19 13:16 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 16 Mar 19 13:16 DefaultBootOrder-45cf35f6-0d6e-4d04-856a-0370a5b16f53 -rw-r--r-- 1 root root 5 Mar 19 13:16 DeploymentModeNv-97e8965f-c761-4f48-b6e4-9ffa9cb2a2d6 -rw-r--r-- 1 root root 68 Mar 19 13:16 EnWpData-cbab171f-f356-4009-baaa-6628353a0a29 -rw-r--r-- 1 root root 36 Mar 19 13:16 Ep-73dad563-8f27-42af-918f-8651eb0a93ef -rw-r--r-- 1 root root 40 Mar 19 13:16 ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 40 Mar 19 13:16 ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 24 Mar 19 13:16 FanCalibrateData-948343ac-a700-4fab-91c1-ac27c16a0cc3 -rw-r--r-- 1 root root 5 Mar 19 13:16 FanCalibrateFlag-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- 1 root root 5 Mar 19 13:16 FirstBootFlag-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- 1 root root 14 Mar 19 13:16 FPLayoutOrder-4db88a62-6721-47a0-9082-280b00323594 -rw-r--r-- 1 root root 5 Mar 19 13:16 FTMActiveFlag-4034591c-48ea-4cdc-864f-e7cb61cfd0f2 -rw-r--r-- 1 root root 12 Mar 19 13:16 HiiDB-1b838190-4625-4ead-abc9-cd5e6af18fe0 -rw-r--r-- 1 root root 3577 Mar 19 13:16 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 3577 Mar 19 13:16 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_ATPSiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_EntRevokeSiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_RvkSiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_SiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_SkuSiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 8 Mar 19 13:16 Kernel_WinSiStatus-77fa9abd-0359-4d32-bd60-28f4e78f784b -rw-r--r-- 1 root root 78 Mar 19 13:16 LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f -rw-r--r-- 1 root root 26 Mar 19 13:16 LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f -rw------- 1 root root 516 Mar 19 13:16 LoaderSystemToken-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f -rw-r--r-- 1 root root 6 Mar 19 13:16 MaximumTableSize-4b3082a3-80c6-4d7e-9cd0-583917265df1 -rw-r--r-- 1 root root 1858 Mar 19 13:16 MemoryConfig0-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 1736 Mar 19 13:16 MemoryConfig1-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 244 Mar 19 13:16 MemoryConfig2-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 1620 Mar 19 13:16 MemoryConfig3-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 1784 Mar 19 13:16 MemoryConfig4-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 298 Mar 19 13:16 MemoryConfig5-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 229 Mar 19 13:16 MemoryConfig6-80dbd530-b74c-4f11-8c03-418665532831 -rw-r--r-- 1 root root 5 Mar 19 13:16 MemoryOverwriteRequestControl-e20939be-32d4-41be-a150-897f85d49829 -rw-r--r-- 1 root root 5 Mar 19 13:16 MemoryOverwriteRequestControlLock-bb983ccf-151d-40e1-a07b-4a17be168292 -rw-r--r-- 1 root root 8 Mar 19 13:16 MonotonicCounter-01368881-c4ad-4b1d-b631-d57a8ec8db6b -rw-r--r-- 1 root root 280 Mar 19 13:16 MyFav-4034591c-48ea-4cdc-864f-e7cb61cfd0f2 -rw-r--r-- 1 root root 6 Mar 19 13:16 NVRAM_Verify-15a9dd61-e4f8-4a99-80db-353b13d76490 -rw-r--r-- 1 root root 32 Mar 19 13:16 OCMR_CPU_FACTORY_RATIO_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- 1 root root 12 Mar 19 13:16 OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 12 Mar 19 13:16 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 890 Mar 19 13:16 PK-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 890 Mar 19 13:16 PKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 10 Mar 19 13:16 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 57 Mar 19 13:16 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 12 Mar 19 13:16 RsdpAddr-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- 1 root root 5 Mar 19 13:16 RstOptaneConfig-4da4f952-2516-4d06-8975-65036403a8c7 -rw-r--r-- 1 root root 5 Mar 19 13:16 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 6 Mar 19 13:16 SetupAPMFeatures-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- 1 root root 8 Mar 19 13:16 SetUpdateCountVar-81c76078-bfde-4368-9790-570914c01a65 -rw-r--r-- 1 root root 20 Mar 19 13:16 SetupHWMFeatures-69fc7103-285c-4d8b-9a08-7b53ef90765c -rw-r--r-- 1 root root 5 Mar 19 13:16 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 148 Mar 19 13:16 SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 12 Mar 19 13:16 SmbiosEntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1 -rw-r--r-- 1 root root 12 Mar 19 13:16 SmbiosEntryPointTableF000-4b3082a3-80c6-4d7e-9cd0-583917265df1 -rw-r--r-- 1 root root 12 Mar 19 13:16 SmbiosScratchBuffer-4b3082a3-80c6-4d7e-9cd0-583917265df1 -rw-r--r-- 1 root root 12 Mar 19 13:16 SmbiosV3EntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1 -rw-r--r-- 1 root root 12 Mar 19 13:16 TbtHRStatusVar-ba1d893b-803e-4b26-a3de-585703ff7bd6 -rw-r--r-- 1 root root 6 Mar 19 13:16 Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 10 Mar 19 13:16 TPMPERBIOSFLAGS-7d3dceee-cbce-4ea7-8709-6e552f1edbde -rw-r--r-- 1 root root 12 Mar 19 13:16 TpmServFlags-7d3dceee-cbce-4ea7-8709-6e552f1edbde -rw-r--r-- 1 root root 5 Mar 19 13:16 VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- 1 root root 8 Mar 19 13:16 WpBufAddr-cba83c4a-a5fc-48a8-b3a6-d33636166544 -rw-r--r-- 1 root root 68 Mar 19 13:16 WriteOnceStatus-4b3082a3-80c6-4d7e-9cd0-583917265df1 [jhenderson@TheEarth ~]$ -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On Thu, Mar 21, 2024 at 8:04 PM Jim Henderson <hendersj@gmail.com> wrote:
[jhenderson@TheEarth ~]$ ls -l /sys/firmware/efi/efivars/
Well, the total size is about 45K. Of course, we do not know the implementation details and overhead, but still it is very unlikely that firmware needs additional 80K to manage 45K of data. It sounds more like either firmware needs to be tricked into garbage collection or it leaks variable storage. In both cases reinitializing settings may actually bring available space up to the reasonable size.
On Fri, 22 Mar 2024 14:50:03 +0300, Andrei Borzenkov wrote:
On Thu, Mar 21, 2024 at 8:04 PM Jim Henderson <hendersj@gmail.com> wrote:
[jhenderson@TheEarth ~]$ ls -l /sys/firmware/efi/efivars/
Well, the total size is about 45K. Of course, we do not know the implementation details and overhead, but still it is very unlikely that firmware needs additional 80K to manage 45K of data. It sounds more like either firmware needs to be tricked into garbage collection or it leaks variable storage. In both cases reinitializing settings may actually bring available space up to the reasonable size.
Good point and catch. I hadn't actually thought to add up the size of all the nodes in the filesystem. I'll mention that when I talk to support next. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/22/24 13:23, Jim Henderson wrote:
On Fri, 22 Mar 2024 14:50:03 +0300, Andrei Borzenkov wrote:
[jhenderson@TheEarth ~]$ ls -l /sys/firmware/efi/efivars/ Well, the total size is about 45K. Of course, we do not know the implementation details and overhead, but still it is very unlikely that firmware needs additional 80K to manage 45K of data. It sounds more like either firmware needs to be tricked into garbage collection or it leaks variable storage. In both cases reinitializing settings may actually bring available space up to the reasonable size. Good point and catch. I hadn't actually thought to add up the size of all
On Thu, Mar 21, 2024 at 8:04 PM Jim Henderson <hendersj@gmail.com> wrote: the nodes in the filesystem. I'll mention that when I talk to support next.
Hi Jim, Any idea how we "trick" the firmware into garbage collection ? I have rebooted and that didn't change anything the amount used. I also summed up the sizes and it is quite smaller than the used amount. df -h /sys/firmware/efi/efivars Filesystem Size Used Avail Use% Mounted on efivarfs 192K 167K 21K 89% /sys/firmware/efi/efivars FYI, if you have pwsh installed on Linux is is really easy to sum up the values: ( dir /sys/firmware/efi/efivars | Measure-Object -Sum Length ).Sum 125253 125253 is only 122.3K quite different from the 167K that df is reporting as used. I will be interested to hear what you find out after talking to support. Feel free to email me directly.... Thanks -- Regards, Joe
On Wed, Mar 20, 2024 at 06:12:18PM -0000, Jim Henderson wrote:
On Wed, 20 Mar 2024 20:23:29 +0300, Andrei Borzenkov wrote:
I'm being advised to increase the size of my UEFI partition - which is 4 GB and has 158 MB free - for a firmware update that is 21.2 KB in size, and the output from fwupdmgr is telling me that efivarfs doesn't have enough free space (it has 20KB free).
Oh, I was not aware that it can show it. Do you have anything in dmesg?
Nothing in dmesg. I determined the free space using:
[jhenderson@TheEarth ~]$ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 128K 104K 20K 85% /sys/firmware/efi/efivars
This function be introduced to Linux kernel since v6.5-rc1 commit d86ff3333cb1d5f42d8898fb5fdb304e143c0237 Author: Anisse Astier <an.astier@criteo.com> Date: Wed May 17 17:38:12 2023 +0200 efivarfs: expose used and total size
Storage space in UEFI is not large, so several KB does make a difference.
Indeed it does.
The issue is with the constraints in the nvram where this data is stored itself, and not the UEFI boot partition on my SSD, correct?
Yes.
Thank you for the confirmation. I was pretty sure that was the case, but after going back and forth several times with the suggestion to "increase the size of the UEFI partition", I thought I might be missing something.
(As a "bonus question", messing around with just deleting stuff like the dbx files in efivarfs could potentially brick the system, right?)
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows.
As I remember that dbx is only effective when secure boot is enabled. If you do not enable secure boot, then you don't need dbx.
What says
df -h /sys/firmware/efi/efivars/
(See above)
I had seen something on another type of system (a Lenovo laptop) about there maybe being an option in the UEFI settings to restore factory keys - and if that option exists, that might be an approach that works.
(https://github.com/fwupd/fwupd/issues/5603 is where that comment is)
The nvram space is a hardware limitation. Hardware manufacturer should handles the dbx update because they should check nvram space first. The UEFI dbx is still growing. Normally 32kb nvram space be reserved for dbx: UEFI shim bootloader secure boot lifecycle improvements https://github.com/rhboot/shim/blob/sbat/SBAT.md Some machine provides > 32kb space for dbx. But it's not in spec. The nvram and dbx space limitation is a problem. No good answer currently. Regards Joey Lee
[QUOTE]Unless you're on a Thelio Major B3, the amount of free space is probably going to be different, as our firmwares are different. Based on that your efivars filesystem space is 64K and mine is 128K, it looks like we're on different systems. The filesystem in question is nvram, so the OS won't make a difference. (I can boot into Pop_OS - without GUI because that somehow got broken on my installation, and since I run TW as my daily driver, I haven't felt the need to try to fix the Pop installation) [/QUOTE] Hmm . . . [off topic question] How do you like your Thelio?? : - ) I had the thought that seems to be on the same lines as Joey's, that it's a "hardware based" issue . . . the and "I lost mt Pop GUI" comment **might** be at the heart of the issue. I know that technically a "firmware update" could be handled by TW, but in Pop when there is a frimware update, it's sort of "like a Mac" in that it seems to be "proprietary" . . . . In my Gazelle to run a firmware update it either reboots, or gets to its own window to run it through, it's not the GUI console or a TTY . . . it runs through and then it reboots back out . . . . My gut feeling (questionable validity) would be to try to fix the Pop GUI and then see if the firmware notification will show up in that fixed GUI environment??
On Thu, 21 Mar 2024 15:02:04 -0000, Fritz Hudnut wrote:
Hmm . . . [off topic question] How do you like your Thelio?? : - )
I love it. Had a few issues with it when I first got it (apparent NMI errors related to memory), but they actually cleared up while trying to troubleshoot them, and haven't happened since.
I had the thought that seems to be on the same lines as Joey's, that it's a "hardware based" issue . . . the and "I lost mt Pop GUI" comment **might** be at the heart of the issue. I know that technically a "firmware update" could be handled by TW, but in Pop when there is a frimware update, it's sort of "like a Mac" in that it seems to be "proprietary" . . . . In my Gazelle to run a firmware update it either reboots, or gets to its own window to run it through, it's not the GUI console or a TTY . . . it runs through and then it reboots back out . . . .
Oh, the "Pop GUI" issue is unrelated - guaranteed. That's a long-standing issue since I installed TW on it the week I got it (about a year and a half ago IIRC). My understanding is that it has something to do with how Pop's bootloader works - it's apparently not entirely compatible with how we do it in openSUSE (I forget the details).
My gut feeling (questionable validity) would be to try to fix the Pop GUI and then see if the firmware notification will show up in that fixed GUI environment??
I tried doing it from a flash drive running Pop at System76's suggestion, and that didn't work either - so it doesn't seem to be the OS that's the issue here. It's just a space issue with the nvram. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On Thu, 21 Mar 2024 13:57:00 +0800, joeyli via openSUSE Factory wrote:
Nothing in dmesg. I determined the free space using:
[jhenderson@TheEarth ~]$ df -h /sys/firmware/efi/efivars/ Filesystem Size Used Avail Use% Mounted on efivarfs 128K 104K 20K 85% /sys/firmware/efi/efivars
This function be introduced to Linux kernel since v6.5-rc1
commit d86ff3333cb1d5f42d8898fb5fdb304e143c0237 Author: Anisse Astier <an.astier@criteo.com> Date: Wed May 17 17:38:12 2023 +0200 efivarfs: expose used and total size
Good to know.
I suppose, removing some variables whose existence is taken for granted by firmware may well do it. I vaguely remember having seen something like this. dbx should not be essential, but how knows.
As I remember that dbx is only effective when secure boot is enabled. If you do not enable secure boot, then you don't need dbx.
I suspected that might be the case, but given the nature of what's stored there, mucking about with it could be bad (and indeed, others pointed out instances where systems were bricked by deleting the wrong thing).
(https://github.com/fwupd/fwupd/issues/5603 is where that comment is)
The nvram space is a hardware limitation. Hardware manufacturer should handles the dbx update because they should check nvram space first. The UEFI dbx is still growing. Normally 32kb nvram space be reserved for dbx:
UEFI shim bootloader secure boot lifecycle improvements https://github.com/rhboot/shim/blob/sbat/SBAT.md
Some machine provides > 32kb space for dbx. But it's not in spec. The nvram and dbx space limitation is a problem. No good answer currently.
That's good to hear. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/21/24 13:02, Jim Henderson wrote:
I suspected that might be the case, but given the nature of what's stored there, mucking about with it could be bad (and indeed, others pointed out instances where systems were bricked by deleting the wrong thing).
Hi Jim, After reading all the comments in this discussion, I wondering if there is a bigger issue going on here. When I first saw your message, I did a df -h /sys/firmware/efi/efivars to see what mine was and found that the size is 192k. While I did not save the output the percent used was like 27% and I thought good, I've got plenty of room of room. Then I did a fwupdmgr get-updates and had one for the dbx for the MS issue. Then I did a fwupdmgr update and after that I did the df -h /sys/firmware/efi/efivars Filesystem Size Used Avail Use% Mounted on efivarfs 192K 167K 21K 89% /sys/firmware/efi/efivars Now it is 89% Used with only 21K still available ! Here's a list of all the things that are > 1k in size: -rw-r--r-- root root 3/21/2024 13:15 1,438 SaSetup-72c5e28c-7783-43a1-8767-fad73fccafa4 -rw-r--r-- root root 3/21/2024 13:15 2,042 MokListRT-605dab50-e046-4300-abb6-3dd810dd8b23 -rw-r--r-- root root 3/21/2024 13:15 2,069 PchSetup-4570b7f1-ade8-4943-8dc3-406472842384 -rw-r--r-- root root 3/21/2024 13:15 3,400 Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- root root 3/21/2024 13:15 3,577 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 3,577 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 3,714 BiosSettingMappingTableV2-b57086d5-c2e5-4654-9e3a-0b55830fbb32 -rw-r--r-- root root 3/21/2024 13:15 6,326 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 6,326 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ? Regards, Joe
On 3/21/24 17:37, Joe Salmeri wrote:
On 3/21/24 13:02, Jim Henderson wrote:
I suspected that might be the case, but given the nature of what's stored there, mucking about with it could be bad (and indeed, others pointed out instances where systems were bricked by deleting the wrong thing).
Hi Jim,
After reading all the comments in this discussion, I wondering if there is a bigger issue going on here.
When I first saw your message, I did a df -h /sys/firmware/efi/efivars to see what mine was and found that the size is 192k.
While I did not save the output the percent used was like 27% and I thought good, I've got plenty of room of room.
Then I did a fwupdmgr get-updates and had one for the dbx for the MS issue. Then I did a fwupdmgr update and after that I did the
df -h /sys/firmware/efi/efivars
Filesystem Size Used Avail Use% Mounted on efivarfs 192K 167K 21K 89% /sys/firmware/efi/efivars Now it is 89% Used with only 21K still available !
Here's a list of all the things that are > 1k in size: -rw-r--r-- root root 3/21/2024 13:15 1,438 SaSetup-72c5e28c-7783-43a1-8767-fad73fccafa4 -rw-r--r-- root root 3/21/2024 13:15 2,042 MokListRT-605dab50-e046-4300-abb6-3dd810dd8b23 -rw-r--r-- root root 3/21/2024 13:15 2,069 PchSetup-4570b7f1-ade8-4943-8dc3-406472842384 -rw-r--r-- root root 3/21/2024 13:15 3,400 Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9 -rw-r--r-- root root 3/21/2024 13:15 3,577 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 3,577 KEK-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 3,714 BiosSettingMappingTableV2-b57086d5-c2e5-4654-9e3a-0b55830fbb32 -rw-r--r-- root root 3/21/2024 13:15 6,326 db-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 6,326 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
Also forgot to mention My EFI partition is 2 GB in size and only 1% used as I'm only using 5.9 MB right now. When I built this system last year I made the partition 2 GB in case I switched to systemd-boot so that I didn't have to adjust the size later to hold the kernels.
On Thu, 21 Mar 2024 17:41:19 -0400, Joe Salmeri wrote:
Also forgot to mention
My EFI partition is 2 GB in size and only 1% used as I'm only using 5.9 MB right now.
When I built this system last year I made the partition 2 GB in case I switched to systemd-boot so that I didn't have to adjust the size later to hold the kernels.
EFI partition size really shouldn't be a factor for a firmware update, though (that's one thing I was asking here to confirm). -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/21/24 20:58, Jim Henderson wrote:
On Thu, 21 Mar 2024 17:41:19 -0400, Joe Salmeri wrote:
Also forgot to mention
My EFI partition is 2 GB in size and only 1% used as I'm only using 5.9 MB right now.
When I built this system last year I made the partition 2 GB in case I switched to systemd-boot so that I didn't have to adjust the size later to hold the kernels. EFI partition size really shouldn't be a factor for a firmware update, though (that's one thing I was asking here to confirm).
Agree 100%, the efivars are stored in NVRAM and are not related to the EFI partition at all. I was just giving you additional confirmation of that since I have a 2 GB EFI partition -- Regards, Joe
On Thu, 21 Mar 2024 17:37:18 -0400, Joe Salmeri wrote:
With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
It's possible, but I also wonder if your update happened to be larger than mine because you haven't updated your firmware recently as well. I was able to update to one of the intermediate database versions (I was previously at v217, I believe, and was able to update to 220, but haven't been able to go from 220 to 371, which was the next version available.) It's also unclear to me what the block size is (I'm guessing because it isn't a real filesystem, it's effectively a single byte because the space isn't allocated like that, but I don't know for certain). -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/21/24 17:48, Jim Henderson wrote:
On Thu, 21 Mar 2024 17:37:18 -0400, Joe Salmeri wrote:
With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ? It's possible, but I also wonder if your update happened to be larger than mine because you haven't updated your firmware recently as well. I was able to update to one of the intermediate database versions (I was previously at v217, I believe, and was able to update to 220, but haven't been able to go from 220 to 371, which was the next version available.)
It's also unclear to me what the block size is (I'm guessing because it isn't a real filesystem, it's effectively a single byte because the space isn't allocated like that, but I don't know for certain).
I actually saw the msg about that firmware update back when I built the system ( Nov 2023 ) but I didn't bother updating. While I was following this thread, I wasn't concerned until I did the update then saw the huge jump. Also, seems like the total size for people that commented has varied..... 64k, 128, and my 192k. Just found this link on another forum https://forum.garudalinux.org/t/efivars-is-full-is-this-a-problem/34616/13 IS it a problem for the efivars directory to be full? It is not a problem. /sys/firmware is not a “real” filesystem, it is part of a RAM-based filesystem called sysfs. The sysfs filesystem provides information about devices and drivers in the kernel through virtual files, so certain device properties can be interacted with using “normal” file operations. Since sysfs resides in memory, it is kind of a temporary structure that gets recreated fresh at every boot. There is no need to make it larger than whatever virtual files it needs to contain, so usually it appears to be full or nearly full. NVRAM and EFI boot variables are not related to the sysfs virtual filesystem. Make me wonder if that explains the difference in total size people are seeing and also that a large size might be allocated if it was needed. Possibly it is allocating space based on real memory size ??? How much memory is in your system ? I have 64 GB in this machine which might be why my size is 192k vs others with 64k or 128k ? -- Regards, Joe
Am 21.03.24 um 22:54 schrieb Joe Salmeri:
Just found this link on another forum
https://forum.garudalinux.org/t/efivars-is-full-is-this-a-problem/34616/13
IS it a problem for the efivars directory to be full?
It is not a problem. /sys/firmware is not a “real” filesystem, it is part of a RAM-based filesystem called sysfs.
I'm afraid that is wrong, because it mixes up two different filesystems. Yes, /sys hosts a sysfs that lives in memory and should have no size limitation. But there is an efivarfs file system at /sys/firmware/efi/efivars, and that has a size limitation (what df shows). In fact there are quite a few more: $ awk '$2 ~ /^\/sys/ {print $1, $2}' /proc/mounts sysfs /sys securityfs /sys/kernel/security cgroup2 /sys/fs/cgroup pstore /sys/fs/pstore efivarfs /sys/firmware/efi/efivars bpf /sys/fs/bpf debugfs /sys/kernel/debug tracefs /sys/kernel/tracing fusectl /sys/fs/fuse/connections configfs /sys/kernel/config Apparently, only efivarfs has a size limitation. (pstore is also quite limited, but df doesn't show anything for me.)
Since sysfs resides in memory, it is kind of a temporary structure that gets recreated fresh at every boot. There is no need to make it larger than whatever virtual files it needs to contain, so usually it appears to be full or nearly full.
Does it? $ df /sys Filesystem 1K-blocks Used Available Use% Mounted on sysfs 0 0 0 - /sys
NVRAM and EFI boot variables are not related to the sysfs virtual filesystem.
Right, but they are related to efivarfs.
On 3/21/24 19:51, Aaron Puchert wrote:
Am 21.03.24 um 22:54 schrieb Joe Salmeri:
Just found this link on another forum
https://forum.garudalinux.org/t/efivars-is-full-is-this-a-problem/34616/13
IS it a problem for the efivars directory to be full?
It is not a problem. /sys/firmware is not a “real” filesystem, it is part of a RAM-based filesystem called sysfs.
I'm afraid that is wrong, because it mixes up two different filesystems. Yes, /sys hosts a sysfs that lives in memory and should have no size limitation. But there is an efivarfs file system at /sys/firmware/efi/efivars, and that has a size limitation (what df shows). In fact there are quite a few more:
$ awk '$2 ~ /^\/sys/ {print $1, $2}' /proc/mounts sysfs /sys securityfs /sys/kernel/security cgroup2 /sys/fs/cgroup pstore /sys/fs/pstore efivarfs /sys/firmware/efi/efivars bpf /sys/fs/bpf debugfs /sys/kernel/debug tracefs /sys/kernel/tracing fusectl /sys/fs/fuse/connections configfs /sys/kernel/config
Apparently, only efivarfs has a size limitation. (pstore is also quite limited, but df doesn't show anything for me.)
Since sysfs resides in memory, it is kind of a temporary structure that gets recreated fresh at every boot. There is no need to make it larger than whatever virtual files it needs to contain, so usually it appears to be full or nearly full.
Does it?
$ df /sys Filesystem 1K-blocks Used Available Use% Mounted on sysfs 0 0 0 - /sys
NVRAM and EFI boot variables are not related to the sysfs virtual filesystem.
Right, but they are related to efivarfs.
Thanks Aaron, I was a little skepticle of some of what I read there but this is not an area I have a lot of experience with. It is my understanding that those variables are stored in NVRAM which I would expect would have a fixed size which would vary between different motherboards. -- Regards, Joe
On Thu, 21 Mar 2024 17:54:46 -0400, Joe Salmeri wrote:
Also, seems like the total size for people that commented has varied..... 64k, 128, and my 192k.
It's NVRAM in the system firmware itself - different vendors (and even different systems from the same vendor) use different amounts, because it depends on what the system's requirements are.
Just found this link on another forum
https://forum.garudalinux.org/t/efivars-is-full-is-this-a-problem/ 34616/13
IS it a problem for the efivars directory to be full?
It is not a problem. /sys/firmware is not a “real” filesystem, it is part of a RAM-based filesystem called sysfs.
Well, it's a filesystem representation of stuff in NVRAM. If you delete files there, you're modifying NVRAM, and you can brick your system (which is why I'm concerned about just deleting stuff to make space - I don't need a multi-thousand-dollar doorstop.)
Make me wonder if that explains the difference in total size people are seeing and also that a large size might be allocated if it was needed.
Not really, as far as I understand it.
Possibly it is allocating space based on real memory size ???
How much memory is in your system ? I have 64 GB in this machine which might be why my size is 192k vs others with 64k or 128k ?
I have 128 GB of RAM in my system. It's not proportional to RAM or that. It's effectively your BIOS settings. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/21/24 21:02, Jim Henderson wrote:
On Thu, 21 Mar 2024 17:54:46 -0400, Joe Salmeri wrote:
Also, seems like the total size for people that commented has varied..... 64k, 128, and my 192k. It's NVRAM in the system firmware itself - different vendors (and even different systems from the same vendor) use different amounts, because it depends on what the system's requirements are.
Right. And I suspect my system has 192k because of the fact that it a top of the line motherboard ( ASUS only makes one that is more expensive ).
Just found this link on another forum
https://forum.garudalinux.org/t/efivars-is-full-is-this-a-problem/ 34616/13 IS it a problem for the efivars directory to be full?
It is not a problem. /sys/firmware is not a “real” filesystem, it is part of a RAM-based filesystem called sysfs. Well, it's a filesystem representation of stuff in NVRAM. If you delete files there, you're modifying NVRAM, and you can brick your system (which is why I'm concerned about just deleting stuff to make space - I don't need a multi-thousand-dollar doorstop.)
I have the same concern !!! One thing interesting, I've found since I started searching on this topic is that it sure seems like there are quite a few different links which are all recent ( last 6 months or so ) where people are running into the issue of efivars being full or almost full. If you look at the list from my other msg where I sorted efivars by size, it seems that on my system these 4 items are about 50% of the consumed space -rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 -- Regards, Joe
On Thu, 21 Mar 2024 21:53:05 -0400, Joe Salmeri wrote:
I have the same concern !!!
One thing interesting, I've found since I started searching on this topic is that it sure seems like there are quite a few different links which are all recent ( last 6 months or so ) where people are running into the issue of efivars being full or almost full.
If you look at the list from my other msg where I sorted efivars by size, it seems that on my system these 4 items are about 50% of the consumed space
-rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824
On your system, if you total the space used by all of the nodes in the filesystem, does it add up to the total space reported as used? Andrei noted that on my system, it apparently does not. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/22/24 13:25, Jim Henderson wrote:
On Thu, 21 Mar 2024 21:53:05 -0400, Joe Salmeri wrote:
I have the same concern !!!
One thing interesting, I've found since I started searching on this topic is that it sure seems like there are quite a few different links which are all recent ( last 6 months or so ) where people are running into the issue of efivars being full or almost full.
If you look at the list from my other msg where I sorted efivars by size, it seems that on my system these 4 items are about 50% of the consumed space
-rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 On your system, if you total the space used by all of the nodes in the filesystem, does it add up to the total space reported as used?
Andrei noted that on my system, it apparently does not.
Any idea how we "trick" the firmware into garbage collection ? I have rebooted and that didn't change anything the amount used. I also summed up the sizes and it is quite smaller than the used amount. df -h /sys/firmware/efi/efivars Filesystem Size Used Avail Use% Mounted on efivarfs 192K 167K 21K 89% /sys/firmware/efi/efivars FYI, if you have pwsh installed on Linux is is really easy to sum up the values: ( dir /sys/firmware/efi/efivars | Measure-Object -Sum Length ).Sum 125253 125253 is only 122.3K quite different from the 167K that df is reporting as used. I will be interested to hear what you find out after talking to support. Feel free to email me directly.... Thanks -- Regards, Joe
On Fri, 22 Mar 2024 15:51:15 -0400, Joe Salmeri wrote:
Any idea how we "trick" the firmware into garbage collection ?
No idea. I've asked.
I have rebooted and that didn't change anything the amount used.
I also summed up the sizes and it is quite smaller than the used amount.
Bear in mind that there's a block size involved (I noticed that when running df on my system, it reported it as '1K blocks'). So a 5-byte node is going to take 1K regardless. I found I had 5 1K blocks remaining when I looked at it that way.
FYI, if you have pwsh installed on Linux is is really easy to sum up the values:
Nice trick. Might have to look into that. I imported the directory listing into oocalc and then used a formula to count the number of 1K blocks in use for each entry and summed that, along with the raw byte sum. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 23.03.2024 19:54, Jim Henderson wrote:
Bear in mind that there's a block size involved (I noticed that when running df on my system, it reported it as '1K blocks'). So a 5-byte node is going to take 1K regardless.
This block size has nothing to do with NVRAM. It is hardcoded in df output. Run stat -f /sys/firmware/efi/efivars/
On Sun, 24 Mar 2024 08:20:30 +0300, Andrei Borzenkov wrote:
On 23.03.2024 19:54, Jim Henderson wrote:
Bear in mind that there's a block size involved (I noticed that when running df on my system, it reported it as '1K blocks'). So a 5-byte node is going to take 1K regardless.
This block size has nothing to do with NVRAM. It is hardcoded in df output. Run
stat -f /sys/firmware/efi/efivars/
Thanks - that is what I suspected, thought the output from df was strange because I didn't think that was a block device. The stat command confirms what I thought (but I didn't think to use that to check). -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 3/22/24 3:51 PM, Joe Salmeri wrote:
On 3/22/24 13:25, Jim Henderson wrote:
On Thu, 21 Mar 2024 21:53:05 -0400, Joe Salmeri wrote:
I have the same concern !!!
One thing interesting, I've found since I started searching on this topic is that it sure seems like there are quite a few different links which are all recent ( last 6 months or so ) where people are running into the issue of efivars being full or almost full.
If you look at the list from my other msg where I sorted efivars by size, it seems that on my system these 4 items are about 50% of the consumed space
-rw-r--r-- root root 3/21/2024 13:15 10,448 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c -rw-r--r-- root root 3/21/2024 13:15 17,964 dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f -rw-r--r-- root root 3/21/2024 13:15 23,216 VARSTORE_OCMR_SETTINGS_NAME-a3a3b874-7636-4182-ba1e-f52c584494e3 -rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 On your system, if you total the space used by all of the nodes in the filesystem, does it add up to the total space reported as used?
Andrei noted that on my system, it apparently does not.
Any idea how we "trick" the firmware into garbage collection ?
I have rebooted and that didn't change anything the amount used.
I also summed up the sizes and it is quite smaller than the used amount.
df -h /sys/firmware/efi/efivars
Filesystem Size Used Avail Use% Mounted on efivarfs 192K 167K 21K 89% /sys/firmware/efi/efivars
FYI, if you have pwsh installed on Linux is is really easy to sum up the values:
( dir /sys/firmware/efi/efivars | Measure-Object -Sum Length ).Sum
125253
125253 is only 122.3K quite different from the 167K that df is reporting as used.
I will be interested to hear what you find out after talking to support. Feel free to email me directly....
Hi Jim, Did you ever find a resolution to this ? I updated my motherboard bios yesterday and after the update df -h /sys/firmware/efi/efivars when from 167k / 89% used to 156k / 84% used. Joe -- Regards, Joe
On Thu, 2 May 2024 16:10:46 -0400, Joe Salmeri wrote:
Did you ever find a resolution to this ?
I updated my motherboard bios yesterday and after the update
df -h /sys/firmware/efi/efivars when from 167k / 89% used to 156k / 84% used.
Ultimately, I used a setting in the EFI BIOS utility to reset the certificates back to the default, and then upgraded to the latest. For me, it was a matter of deleting the certificates and then using the option in my system to reinstall the default certificates. Then I started the system and used fwupdmgr to upgrade to the latest version - the reset/reinstall freed up enough space. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 5/2/24 4:22 PM, Jim Henderson wrote:
On Thu, 2 May 2024 16:10:46 -0400, Joe Salmeri wrote:
Did you ever find a resolution to this ?
I updated my motherboard bios yesterday and after the update
df -h /sys/firmware/efi/efivars when from 167k / 89% used to 156k / 84% used.
Ultimately, I used a setting in the EFI BIOS utility to reset the certificates back to the default, and then upgraded to the latest. For me, it was a matter of deleting the certificates and then using the option in my system to reinstall the default certificates.
Then I started the system and used fwupdmgr to upgrade to the latest version - the reset/reinstall freed up enough space.
When I installed the new BIOS yesterday I did the Load Optimized BIOS defaults, I'll have to look further for a setting to reinstall the default certificates. Do you recall the verbiage it used ? There are hundreds of setting in the BIOS on this motherboard. -- Regards, Joe
On Thu, 2 May 2024 17:09:41 -0400, Joe Salmeri wrote:
When I installed the new BIOS yesterday I did the Load Optimized BIOS defaults, I'll have to look further for a setting to reinstall the default certificates.
Do you recall the verbiage it used ?
There are hundreds of setting in the BIOS on this motherboard.
On my system, it was under the Secure Boot settings - I ended up asking the vendor for a pointer (because, like yours, there are lots of settings in mine as well). There was a section for managing the certificates, as I recall, and there were two options - one to delete, and one to reinstall the defaults. I deleted them, rebooted, and tried running the update, but it didn't run, so I reinstalled the defaults, and then the update ran. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 5/2/24 5:24 PM, Jim Henderson wrote:
On Thu, 2 May 2024 17:09:41 -0400, Joe Salmeri wrote:
When I installed the new BIOS yesterday I did the Load Optimized BIOS defaults, I'll have to look further for a setting to reinstall the default certificates.
Do you recall the verbiage it used ?
There are hundreds of setting in the BIOS on this motherboard.
On my system, it was under the Secure Boot settings - I ended up asking the vendor for a pointer (because, like yours, there are lots of settings in mine as well). There was a section for managing the certificates, as I recall, and there were two options - one to delete, and one to reinstall the defaults. I deleted them, rebooted, and tried running the update, but it didn't run, so I reinstalled the defaults, and then the update ran.
Ok, I dug out the mb manual as I cannot reboot the system right now. In the Secure Boot settings there is a section "Key Management" with an option "Install Default Secure Boot keys", which says: Allows you to immediately load the default Secure Boot keys, Platform key (PK), Key exchange Key (KEK), Signature database (db) and revoked Signatures (dbx). When the default Secure boot keys are loaded the PK state will change from Unloaded mode to loaded mode. Not sure what they mean about the PK state changing..... I thought that during the TW install that they install a key that is signed by Microsoft so that TW can use Secure Boot ? Don't we need to do something so that gets reinstalled ? Cannot afford to brick this system ! -- Regards, Joe
On Thu, 2 May 2024 17:40:51 -0400, Joe Salmeri wrote:
Ok, I dug out the mb manual as I cannot reboot the system right now.
In the Secure Boot settings there is a section "Key Management" with an option "Install Default Secure Boot keys", which says:
That's similar wording to what I did on my system after removing all the keys (that was also in my key management configuration area)
I thought that during the TW install that they install a key that is signed by Microsoft so that TW can use Secure Boot ?
Don't we need to do something so that gets reinstalled ?
Cannot afford to brick this system !
I actually am not using secure boot, so I'm probably not the best person to advise. What I'd probably do while doing this work is disable it, make the changes, and then re-enable it. It appears that enabling/disabling the support is in the UEFI firmware itself. In my case, I was mostly just trying to get rid of the persistent "you have firmware updates" message. But if I were in your shoes, I would wait for someone with more experience with secure boot to weigh in on how best to avoid bricking the system. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits
On 22.03.2024 00:37, Joe Salmeri wrote:
-rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
It is quite possible that your firmware only performs garbage collection and/or compaction on demand or power on. P.S. you systematically paste large content without line breaks that makes it unreadable.
On Fri, Mar 22, 2024 at 07:10:07AM +0300, Andrei Borzenkov wrote:
On 22.03.2024 00:37, Joe Salmeri wrote:
-rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
It is quite possible that your firmware only performs garbage collection and/or compaction on demand or power on.
Different firmware has different garbage collection logic. Some firmware always runs garbage colleciton when booting. But some firmware wants to speed up booting, so it set a threshold of free NVRAM space. The garbage collection be triggered when the free NVRAM space lower than threshold. In early UEFI firmware implementation, someone found a case that the threshold is too low. The threshold is lower than the requirements of garbage collection mechanism. Firmware does not reserve enough free NVRAM space fore garbage collection. Then the machine is brick, which always hang in garbage collection. That's why Linux kernel reserves a space to prevent the situation. But actually nobody knows all UEFI firmware's requeirement of minimum space for garbage collection. Only OEM/ODM firmware team knows. Regards Joey Lee
On 3/22/24 00:10, Andrei Borzenkov wrote:
On 22.03.2024 00:37, Joe Salmeri wrote:
-rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
It is quite possible that your firmware only performs garbage collection and/or compaction on demand or power on.
Thanks, I have rebooted since I sent the message but the size used hasn't changed.
P.S. you systematically paste large content without line breaks that makes it unreadable.
My apologies, when I cut/paste that into the email, it did so as HTML. In Thunderbird it appeared fine with line breaks/etc because it was HTML and not plain text. -- Regards, Joe
On 3/22/24 15:12, Joe Salmeri wrote:
On 3/22/24 00:10, Andrei Borzenkov wrote:
On 22.03.2024 00:37, Joe Salmeri wrote:
-rw-r--r-- root root 3/21/2024 13:15 33,481 StdDefaults-4599d26f-1a11-49b8-b91f-858745cff824 With the large jump in percent used, could there be an issue with fwupdmgr that is causing your issue ?
It is quite possible that your firmware only performs garbage collection and/or compaction on demand or power on.
Thanks, I have rebooted since I sent the message but the size used hasn't changed.
P.S. you systematically paste large content without line breaks that makes it unreadable.
My apologies, when I cut/paste that into the email, it did so as HTML. In Thunderbird it appeared fine with line breaks/etc because it was HTML and not plain text.
Something strange going going on with Thunderbird because I just replied to something else and I specifically set it to send as plain text only and when I got the email it was reformatted without the line breaks. -- Regards, Joe
participants (6)
-
Aaron Puchert -
Andrei Borzenkov -
Fritz Hudnut -
Jim Henderson -
Joe Salmeri -
joeyli