Boot Loader Specification, securing initrd
Hi, There was some discussion about 18 months ago on the file system layout [1]. It also mentions the Boot Loader Specification. systemd 252 NEWS [2] mentions changes and clarifications in sd-boot, bootctl, and the Boot Loader Specification. And it's got me wondering if there's interest in openSUSE adopting Boot Loader Spec? I recently came across Boom[3] a project to manage BLS snippets for the boot-to-snapshot use case, Btrfs and LVM. One hurdle is BLS pretty much obviates the idea of an encrypted $BOOT, which openSUSE supports right now. While the kernel and initramfs are not secrets, thus don't need confidentiality, the initrd in particular needs to be protected from malicious insertions, which (somewhat indirectly) encryption achieves. Whether no initramfs or implementing BLS Type 2 (EFI Unified Kernel Images, which includes an initrd and bootloader config, and the whole thing is signed), suggests pretty significant changes to implement. Any recent thoughts on the general direction to go in? Thanks. [1] Thoughts on the file system layout https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/G... [2] https://github.com/systemd/systemd/blob/main/NEWS [3] https://github.com/snapshotmanager/boom -- Chris Murphy
On Di, Aug 30 2022 at 13:56:40 -0400, Chris Murphy <lists@colorremedies.com> wrote:
Hi,
There was some discussion about 18 months ago on the file system layout [1]. It also mentions the Boot Loader Specification.
systemd 252 NEWS [2] mentions changes and clarifications in sd-boot, bootctl, and the Boot Loader Specification. And it's got me wondering if there's interest in openSUSE adopting Boot Loader Spec?
I have lifted support from Fedora in some of my grub2 builds on obs in the past, and used my system with modified /boot mount from how it comes preinstalled, it works, but it is most certainly janky the way I did it.
I recently came across Boom[3] a project to manage BLS snippets for the boot-to-snapshot use case, Btrfs and LVM.
We could go two ways with snapper support, it's currently done with a shell script: https://build.opensuse.org/package/view_file/Base:System/grub2/grub2-snapper... we could either integrate this shell script with boom, or rewrite the shell script to still do the full handling of boot entries. Considering the status quo, boom never seemed necessary for our use case to me, but it could be a nice idea if it was integrated more directly with snapper. Since BLS is a pretty generic, it's probably a better idea than the current hook system anyhow, but I assume I would be lynched by anybody using snapper outside of the openSUSE context ;) https://github.com/openSUSE/snapper/blob/master/snapper/Hooks.cc
One hurdle is BLS pretty much obviates the idea of an encrypted $BOOT, which openSUSE supports right now. While the kernel and initramfs are not secrets, thus don't need confidentiality, the initrd in particular needs to be protected from malicious insertions, which (somewhat indirectly) encryption achieves. Whether no initramfs or implementing BLS Type 2 (EFI Unified Kernel Images, which includes an initrd and bootloader config, and the whole thing is signed), suggests pretty significant changes to implement.
Any recent thoughts on the general direction to go in? Thanks.
I would say this starts with implementing bls in perl-bootloader or trying to lift support for bls config generation from other distributions (though that is simple enough, it seems Fedora for example deprecated grubby as a whole and instead maintains a bunch of "short" shell scripts that generate bls config). From having used both grub2 with bls and systemd-boot on openSUSE, the set up is a janky mess, but the experience is fine. BLS's strengths are in the much lower complexity of maintenance of boot entries, which is most certainly a plus for a person that lives off of a hacked up distro at all times ;) So I would say yes, but I assume not everyone will be this enthused about changing the snapshots to not include /boot, since we already had a discussion about this a few years back, and I got a big fat nope last time around https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/4... (you will have to search for my mention of /boot/efi, bls or systemd-boot to find it though, it's a long thread about a lot of stuff) LCP [Jake] https://lcp.world/
On Tue, 2022-08-30 at 21:01 +0200, Jacob Michalskie wrote:
So I would say yes, but I assume not everyone will be this enthused about changing the snapshots to not include /boot, since we already had a discussion about this a few years back, and I got a big fat nope last time around
If /boot is not included in snapshots, then users will not be able to use snapshots/rollback whenever there is an issue with the kernel, grub, initrd, and such that prevents the user booting I would strongly argue that would be a huge, detrimental, regression compared to the status quo -- Richard Brown Linux Distribution Engineer - Future Technology Team SUSE Software Solutions Germany GmbH, Frankenstraße 146, D-90461 Nuremberg, Germany (HRB 36809, AG Nürnberg) Managing Directors/Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
On Wed, Aug 31, Richard Brown wrote:
On Tue, 2022-08-30 at 21:01 +0200, Jacob Michalskie wrote:
So I would say yes, but I assume not everyone will be this enthused about changing the snapshots to not include /boot, since we already had a discussion about this a few years back, and I got a big fat nope last time around
If /boot is not included in snapshots, then users will not be able to use snapshots/rollback whenever there is an issue with the kernel, grub, initrd, and such that prevents the user booting
Even worse: you can never do a rollback to an older snapshot if the last update did contain a kernel update. Because if you do a rollback because your Firefox is broken, you would end with a boot kernel which cannot find it's modules anymore... Thorsten
I would strongly argue that would be a huge, detrimental, regression compared to the status quo -- Richard Brown Linux Distribution Engineer - Future Technology Team SUSE Software Solutions Germany GmbH, Frankenstraße 146, D-90461 Nuremberg, Germany (HRB 36809, AG Nürnberg) Managing Directors/Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman (HRB 36809, AG Nürnberg)
On Wednesday, August 31, 2022 10:40:32 AM CEST Thorsten Kukuk wrote:
Even worse: you can never do a rollback to an older snapshot if the last update did contain a kernel update. Because if you do a rollback because your Firefox is broken, you would end with a boot kernel which cannot find it's modules anymore...
As today this is true, but there is a tool that can do the bookkeeping tasks to track the combinations of kernel + initrd + sysroot that worked, so grub or bootctl can choose later the correct combination to do the rollback. It is challenging! But there are some invariants that should make the problem doable. -- SUSE Software Solutions Germany GmbH Frankenstrasse 146 90461 Nuremberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman (HRB 36809, AG Nürnberg)
On Wed, Aug 31, 2022, at 5:03 AM, Alberto Planas wrote:
On Wednesday, August 31, 2022 10:40:32 AM CEST Thorsten Kukuk wrote:
Even worse: you can never do a rollback to an older snapshot if the last update did contain a kernel update. Because if you do a rollback because your Firefox is broken, you would end with a boot kernel which cannot find it's modules anymore...
As today this is true, but there is a tool that can do the bookkeeping tasks to track the combinations of kernel + initrd + sysroot that worked, so grub or bootctl can choose later the correct combination to do the rollback.
It is challenging! But there are some invariants that should make the problem doable.
I agree with this. I strongly believe the design should not expect special functionality or knowledge on the part of the bootloader. Depending on a specialized bootloader means basic functionality (snapshots and rollbacks) hinges on a particular bootloader. -- Chris Murphy
On Wed, Aug 31, 2022, at 4:33 AM, Richard Brown wrote:
On Tue, 2022-08-30 at 21:01 +0200, Jacob Michalskie wrote:
So I would say yes, but I assume not everyone will be this enthused about changing the snapshots to not include /boot, since we already had a discussion about this a few years back, and I got a big fat nope last time around
If /boot is not included in snapshots, then users will not be able to use snapshots/rollback whenever there is an issue with the kernel, grub, initrd, and such that prevents the user booting
It is one method to pin vmlinuz+initramfs versions based on the installed kernels in any (system) snapshot. Btrfs snapshots aren't the only way to achieve this. The bigger issue is how to go about making $BOOT the correct size, because it's no longer part of the Btrfs pool. The size depends on the amount of rentention.
I would strongly argue that would be a huge, detrimental, regression compared to the status quo
Were it to be dropped entirely, I agree. But the idea anyone needs weeks, let alone months, of system snapshots isn't very compelling. And also the farther back the rollback, the more security and bug fixes are rolled back too. -- Chris Murphy
On Wed, 2022-08-31 at 12:25 -0400, Chris Murphy wrote:
On Wed, Aug 31, 2022, at 4:33 AM, Richard Brown wrote:
On Tue, 2022-08-30 at 21:01 +0200, Jacob Michalskie wrote:
So I would say yes, but I assume not everyone will be this enthused about changing the snapshots to not include /boot, since we already had a discussion about this a few years back, and I got a big fat nope last time around
If /boot is not included in snapshots, then users will not be able to use snapshots/rollback whenever there is an issue with the kernel, grub, initrd, and such that prevents the user booting
It is one method to pin vmlinuz+initramfs versions based on the installed kernels in any (system) snapshot. Btrfs snapshots aren't the only way to achieve this. The bigger issue is how to go about making $BOOT the correct size, because it's no longer part of the Btrfs pool. The size depends on the amount of rentention.
Pinning versions only works if its only the versions being changed that caused the problem. One of the reasons btrfs snapshots beats every other automatic healing mechanism is that they also capture unexpected behaviour, such as issues caused by rpm scripts. And our rpm packaging scripts are just as likely to cause problems as actual version bumps of binaries..if not more so, as those scripts are often, by design, doing intelligent stuff based on what they detect on the installing host..we sometimes NEED packages to behave differently depending on what else they find on the system, but in doing so we NEED a way to capture those differences and roll them back if they prove to be invalid. We have that now with snapshots/transactional-updates, and nothing else. Any new augmentation/replacement to the status quo needs to at least keep parity with what we already have.
I would strongly argue that would be a huge, detrimental, regression compared to the status quo
Were it to be dropped entirely, I agree. But the idea anyone needs weeks, let alone months, of system snapshots isn't very compelling. And also the farther back the rollback, the more security and bug fixes are rolled back too.
Someone who doesn't reboot their system for months, may need months of system snapshots. As a MicroOS fan, I might say that such long uptimes are stupid and best avoided, but I'm also well aware that such long uptimes are VERY common among our userbase, especially on the enterprise side of things. So, yeah, we need months of snapshots as long as their are months of potential boot-breaking changes which have not been validated by a boot. -- Richard Brown Linux Distribution Engineer - Future Technology Team SUSE Software Solutions Germany GmbH, Frankenstraße 146, D-90461 Nuremberg, Germany (HRB 36809, AG Nürnberg) Managing Directors/Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
On Thu, Sep 1, 2022, at 4:40 AM, Richard Brown wrote:
Pinning versions only works if its only the versions being changed that caused the problem.
One of the reasons btrfs snapshots beats every other automatic healing mechanism is that they also capture unexpected behaviour, such as issues caused by rpm scripts.
Sure but I'm not suggesting the end to snapshots in the system root. I assume rpm scripts that could cause issues only target the system root, not $BOOT. And if scripts can act on $BOOT, I think it's limited to that of adding or removing vmlinuz, initrd, and bootloader config. If that assumption is wrong, then I need to know more about how $BOOT is being modified, and I'm likely to be pretty skeptical of any sort of complexity. The whole idea of Boot Loader Spec is keeping things simple, almost to the point of silliness (which happens to be one of the more common arguments against BLS, funny enough).
And our rpm packaging scripts are just as likely to cause problems as actual version bumps of binaries..if not more so, as those scripts are often, by design, doing intelligent stuff based on what they detect on the installing host..we sometimes NEED packages to behave differently depending on what else they find on the system, but in doing so we NEED a way to capture those differences and roll them back if they prove to be invalid.
I agree but I'm not imagining what unique differences could happen on $BOOT specifically? If the constraints on $BOOT are sufficient, it seems pretty unlikely, i.e. only certain tools can create or delete files; no tool can modify files; and we can use mount options to enforce DAC and MAC.
We have that now with snapshots/transactional-updates, and nothing else. Any new augmentation/replacement to the status quo needs to at least keep parity with what we already have.
I tentatively agree but also would counter that it suggests more constraints on $BOOT are needed, rather than snapshotting complexity in order to be able to rollback prior complexity. Perhaps the strongest argument I can make in favor of file system snapshots is having a fallback if some distro breaks the boot of another distro. But if we were to establish Btrfs as the preferred volume format for $BOOT, we don't really get around this problem because any mistake in another distribution could just do something like a super aggressive garbage collection and delete a bunch of snapshots owned by another distro. No, I really think the solution is simplicity via constraints, and holding distros accountable to the spec. If the spec is lacking in clarity about proper or improper behaviors, we need to get that added into the spec itself.
I would strongly argue that would be a huge, detrimental, regression compared to the status quo
Were it to be dropped entirely, I agree. But the idea anyone needs weeks, let alone months, of system snapshots isn't very compelling. And also the farther back the rollback, the more security and bug fixes are rolled back too.
Someone who doesn't reboot their system for months, may need months of system snapshots.
Sure. The system root would be part of that, as I'm thinking you're still doing limited software updates and taking a snapshot before and after them - but reboot isn't required. However, in these cases, $BOOT is changing very little if at all. The only time a snapshot would capture some change is if there's an addition or removal of a kernel. But we don't need Btrfs snapshots to merely perform retention of three files. I could get on board some optional fallback boot implementation. Maybe it's a distro specific copy of everything that would go on $BOOT, but found in /var. Either the distro specific grub.cfg, or individual BLS snippets, could point to this alternative/fallback location. It's not ridiculous to want $BOOT to be repairable. At some point it could be a spec enhancement.
As a MicroOS fan, I might say that such long uptimes are stupid and best avoided, but I'm also well aware that such long uptimes are VERY common among our userbase, especially on the enterprise side of things.
So, yeah, we need months of snapshots as long as their are months of potential boot-breaking changes which have not been validated by a boot.
OK but would you expect to need to retain 3 kernels for these use case? 30? 300? I still think that question goes back to the loss of pooling with Btrfs, and how to properly estimate the size $BOOT needs to be. -- Chris Murphy
On Thu, 2022-09-01 at 12:21 -0400, Chris Murphy wrote:
On Thu, Sep 1, 2022, at 4:40 AM, Richard Brown wrote:
Pinning versions only works if its only the versions being changed that caused the problem.
One of the reasons btrfs snapshots beats every other automatic healing mechanism is that they also capture unexpected behaviour, such as issues caused by rpm scripts.
Sure but I'm not suggesting the end to snapshots in the system root. I assume rpm scripts that could cause issues only target the system root, not $BOOT. And if scripts can act on $BOOT, I think it's limited to that of adding or removing vmlinuz, initrd, and bootloader config.
If that assumption is wrong, then I need to know more about how $BOOT is being modified, and I'm likely to be pretty skeptical of any sort of complexity. The whole idea of Boot Loader Spec is keeping things simple, almost to the point of silliness (which happens to be one of the more common arguments against BLS, funny enough).
Your assumption is wrong. Any rpm is installed as root. Any rpm can introduce new modules which result in modifications to /boot. Any rpm script can modify whathever it wants. No restriction we could impose in our own packaging would translate to 3rd parties. 3rd party kernel modules are, despite the wishes of the Kernel development community, used by a large proportion of Linux users. Therefore any rollback mechanism needs to suspect unexpected alterations to /boot, hence why our typical btrfs snapshotting approach integrates it as part of the whole system.
No, I really think the solution is simplicity via constraints, and holding distros accountable to the spec. If the spec is lacking in clarity about proper or improper behaviors, we need to get that added into the spec itself.
Constraints cannot be imposed on 3rd parties, but 3rd party kernel modules are a thing, as already mentioned. Are you really going to tell everyone they cant have NVIDIA drivers any more?
As a MicroOS fan, I might say that such long uptimes are stupid and best avoided, but I'm also well aware that such long uptimes are VERY common among our userbase, especially on the enterprise side of things.
So, yeah, we need months of snapshots as long as their are months of potential boot-breaking changes which have not been validated by a boot.
OK but would you expect to need to retain 3 kernels for these use case? 30? 300? I still think that question goes back to the loss of pooling with Btrfs, and how to properly estimate the size $BOOT needs to be.
But as Thorsten pointed out, just keeping excess kernels around is not enough. /boot is not some magical directory that exists in isolation from the rest of the operating system. It's entirely possible that a kernel once loaded may depend on other modules, libraries, and files elsewhere in the filesystem. Which is why we treat /boot as part of the whole operating system, and ensure that when someone rolls back they get the OS, including /boot, exactly as they used to have it. No one wants to go from a known-working-bootable system to a broken system and then onward to a half-rolled-back-hybrid of /root and /boot being different from how they had it before. People want their system restored back to the state they knew worked..which means rolling back /boot in sync with the rest of their system. -- Richard Brown Linux Distribution Engineer - Future Technology Team SUSE Software Solutions Germany GmbH, Frankenstraße 146, D-90461 Nuremberg, Germany (HRB 36809, AG Nürnberg) Managing Directors/Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
On Fri, Sep 2, 2022, at 4:47 AM, Richard Brown wrote:
Your assumption is wrong.
Any rpm is installed as root.
Any rpm can introduce new modules which result in modifications to /boot.
Are there examples of this actually happening? Or is it entirely hypothetical?
Any rpm script can modify whathever it wants.
Same for /boot/efi but it's excluded from the snapshot regime. Why is this risk acceptable? But two additional files per retained kernel version is an unacceptable risk?
No restriction we could impose in our own packaging would translate to 3rd parties. 3rd party kernel modules are, despite the wishes of the Kernel development community, used by a large proportion of Linux users.
Therefore any rollback mechanism needs to suspect unexpected alterations to /boot, hence why our typical btrfs snapshotting approach integrates it as part of the whole system.
There must be some limit to this. Because already $ESP is excluded. And if RPM scripts can do arbitrary things to $BOOT then it can do arbitrary things to snapshots on the system root, it's also a much bigger and more valuable target. A strong case can be made that MAC should disallow RPM from touching $BOOT $ESP $XBOOTLDR, only allowing privileged programs to do that. Constraints.
Constraints cannot be imposed on 3rd parties, but 3rd party kernel modules are a thing, as already mentioned.
Are you really going to tell everyone they cant have NVIDIA drivers any more?
100% 3rd party RPMs do not need to touch $BOOT directly. They install a module on the system root, add a dracut module, add or suggest to add a boot parameter, and its other tools that actually do that, rolling in the changes into the initramfs. But it's still dracut writing to $BOOT not an RPM inserting arbitrary kernel modules.
But as Thorsten pointed out, just keeping excess kernels around is not enough.
/boot is not some magical directory that exists in isolation from the rest of the operating system.
It's entirely possible that a kernel once loaded may depend on other modules, libraries, and files elsewhere in the filesystem.
I understand. There could be a kernel version with multiple variant initramfs's. A way to match the "generation" of the initramfs with system root would be needed.
Which is why we treat /boot as part of the whole operating system, and ensure that when someone rolls back they get the OS, including /boot, exactly as they used to have it.
No one wants to go from a known-working-bootable system to a broken system and then onward to a half-rolled-back-hybrid of /root and /boot being different from how they had it before.
Yeah I understand the logic. And BLS does not proscribe using Btrfs for $XBOOTLDR. The question is about tradeoffs, having Btrfs track /boot changes versus another mechanism. And the drawback of providing signed efifs drivers in order for $XBOOTLDR being anything other than FAT.
People want their system restored back to the state they knew worked..which means rolling back /boot in sync with the rest of their system.
I doubt any users care about rolling back /boot specifically. But rather the mechanism ensuring the pairing of kernel+initramfs+/usr is trustworthy. -- Chris Murphy
On 02.09.22 16:52, Chris Murphy wrote:
I doubt any users care about rolling back /boot specifically. But rather the mechanism ensuring the pairing of kernel+initramfs+/usr is trustworthy.
There could, for example, be a copy of /boot in the snapshotted btrfs root. And our initrd then just needs to contain enough magic sauce so that it can be told to "restore the kernel+initrd from snapshot to /boot" There even could be a special mini-rescue-system-kernel+initrd combo that can do just that. Then the "boot old snapshot" option in the boot loader becomes two step: first boot the "restore-kernel-magic-rescue-system" and then reboot into the old snapshot. This does not sound like it is not doable at all to me. But yes, understood. At least one of the participants in this discussion does not want that to happen, so we are looking for problems instead of possible solutions ;-) Note that I have absolutely no interest in that at all. I plan to skip all that BTRFS / snapshot /stuff at least until my kids start administering my computers for me (hopefully) some decades from now. I'm just writing down one obvious way to solve the "problem". -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On 2022-09-02 16:52, Chris Murphy wrote:
On Fri, Sep 2, 2022, at 4:47 AM, Richard Brown wrote:
Your assumption is wrong.
Any rpm is installed as root.
Any rpm can introduce new modules which result in modifications to /boot.
Are there examples of this actually happening? Or is it entirely hypothetical?
nvidia/virtualbox/vmware, for instance. Rolling back a snapshot without rolling back the entire /boot, is absurd, IMHO. -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
On Tuesday, August 30, 2022 7:56:40 PM CEST Chris Murphy wrote:
Any recent thoughts on the general direction to go in? Thanks.
There are some developers (Ludwig and Michael Chang) making progress in this direction. One of them is that there is an experimental patch to GRUB2 that make it to understand the BLS layout. This patch needs work and stabilization, but can be a future candidate to express in the ESP a layout that can be both be understood by GRUB and sd-boot (the devil is in the details, so this will be a lot of work) There is also some work on expressing the snapshots subvolumes in terms of this BLS, and some draft tool will create new boot entries (for now only for sd-boot) per combination of tested kernel + initrd + snapshot. In this way a rollback is a matter of a bootctl call (reeeeally neat) Maybe more work needs to be done in the standardization of some of the subvolume naming, to provide more information to the tools. All this is very exploratory, but IMHO is a very promising path as it present a more straightforward mechanism, that can work the same for Tumbleweed and for MicroOS. -- SUSE Software Solutions Germany GmbH Frankenstrasse 146 90461 Nuremberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman (HRB 36809, AG Nürnberg)
On Wed, Aug 31, 2022 at 10:57:30AM +0200, Alberto Planas wrote:
On Tuesday, August 30, 2022 7:56:40 PM CEST Chris Murphy wrote:
Any recent thoughts on the general direction to go in? Thanks.
There are some developers (Ludwig and Michael Chang) making progress in this direction.
One of them is that there is an experimental patch to GRUB2 that make it to understand the BLS layout. This patch needs work and stabilization, but can be a future candidate to express in the ESP a layout that can be both be understood by GRUB and sd-boot (the devil is in the details, so this will be a lot of work)
Yes I think it is good to have those BLS support in grub. It is not in factory yet but will carry on that.
There is also some work on expressing the snapshots subvolumes in terms of this BLS, and some draft tool will create new boot entries (for now only for sd-boot) per combination of tested kernel + initrd + snapshot. In this way a rollback is a matter of a bootctl call (reeeeally neat)
It should be suffice to just run `btrfs subvolume set-default ...` and reboot to rollback to `any` root snapshot. No bootctl or any bootloader related reinstall/reconfig should take place in the middle of the process, otherwise the operation is not going to be an atomic rollback immute to sudden failure like power outage in between. Thanks, Michael
Maybe more work needs to be done in the standardization of some of the subvolume naming, to provide more information to the tools.
All this is very exploratory, but IMHO is a very promising path as it present a more straightforward mechanism, that can work the same for Tumbleweed and for MicroOS.
-- SUSE Software Solutions Germany GmbH Frankenstrasse 146 90461 Nuremberg Germany
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman (HRB 36809, AG Nürnberg)
On Wed, Aug 31, 2022, at 10:49 PM, Michael Chang wrote:
On Wed, Aug 31, 2022 at 10:57:30AM +0200, Alberto Planas wrote:
On Tuesday, August 30, 2022 7:56:40 PM CEST Chris Murphy wrote:
Any recent thoughts on the general direction to go in? Thanks.
There are some developers (Ludwig and Michael Chang) making progress in this direction.
One of them is that there is an experimental patch to GRUB2 that make it to understand the BLS layout. This patch needs work and stabilization, but can be a future candidate to express in the ESP a layout that can be both be understood by GRUB and sd-boot (the devil is in the details, so this will be a lot of work)
Yes I think it is good to have those BLS support in grub. It is not in factory yet but will carry on that.
There is also some work on expressing the snapshots subvolumes in terms of this BLS, and some draft tool will create new boot entries (for now only for sd-boot) per combination of tested kernel + initrd + snapshot. In this way a rollback is a matter of a bootctl call (reeeeally neat)
It should be suffice to just run `btrfs subvolume set-default ...` and reboot to rollback to `any` root snapshot. No bootctl or any bootloader related reinstall/reconfig should take place in the middle of the process, otherwise the operation is not going to be an atomic rollback immute to sudden failure like power outage in between.
I'm not fussy about what tool invokes rollbacks. If $BOOT is an independent file system from $ROOT, then two modifications are needed and thus we can't assume atomicity. e.g. you'd have to do both a btrfs subvolume set-default and change grubenv saved_entry to point to a proper kernel+initramfs combination; and yeah since that's not atomic there is the potential, no matter which order those are done in, that a crash happen after the 1st, before the 2nd, and now you can't boot. Either the kernel you need isn't being pointed to; or the kernel modules you need aren't available. But I think that's not insurmountable, the question is what are the follow on consequences? e.g. you can write a new BLS snippet that points to a specific kernel+initramfs+$ROOT combination, and then when you're ready to make that the current boot, just change the proper variable in grubenv or with bootctl -- Chris Murphy
Chris Murphy wrote:
There was some discussion about 18 months ago on the file system layout [1]. It also mentions the Boot Loader Specification.
systemd 252 NEWS [2] mentions changes and clarifications in sd-boot, bootctl, and the Boot Loader Specification. And it's got me wondering if there's interest in openSUSE adopting Boot Loader Spec?
I can't speak for others but I am intested :-)
I recently came across Boom[3] a project to manage BLS snippets for the boot-to-snapshot use case, Btrfs and LVM.
One hurdle is BLS pretty much obviates the idea of an encrypted $BOOT, which openSUSE supports right now. While the kernel and initramfs are not secrets, thus don't need confidentiality, the initrd in particular needs to be protected from malicious insertions, which (somewhat indirectly) encryption achieves. Whether no initramfs or implementing BLS Type 2 (EFI Unified Kernel Images, which includes an initrd and bootloader config, and the whole thing is signed), suggests pretty significant changes to implement.
As long as we don't secure the whole boot chain using eg tpm, it doesn't really matter whether the initrd is within the encrypted volume or not. I think people have a false sense of security with encrypted /boot as it is now. Anyway, yes initrd authentication needs to be solved at some point to counter evil maid attacks but is not a blocker for BLS IMO.
Any recent thoughts on the general direction to go in? Thanks.
In general I think we should prepare for BLS/sd-boot. To solve the snapshot issue I made a PoC that hooks into snapper while ago: https://build.opensuse.org/package/show/home:lnussel:legacyfree/kernel-insta... Works on both traditional systems as well as MicroOS. Here's an image that boots with sd-boot and has snapshots support: https://build.opensuse.org/package/binaries/home:lnussel:legacyfree/openSUSE... Also some upstream discussion: https://github.com/systemd/systemd/pull/23841 In the long run we need some more radical changes though. I don't think /.snaphots, transactional-update and overlayfs hacks to fool packages are things to keep going forward. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Ivo Totev HRB 36809 (AG Nürnberg)
On Mon, Sep 5, 2022, at 5:46 AM, Ludwig Nussel wrote:
Chris Murphy wrote:
One hurdle is BLS pretty much obviates the idea of an encrypted $BOOT...[snip]
As long as we don't secure the whole boot chain using eg tpm, it doesn't really matter whether the initrd is within the encrypted volume or not. I think people have a false sense of security with encrypted /boot as it is now. Anyway, yes initrd authentication needs to be solved at some point to counter evil maid attacks but is not a blocker for BLS IMO.
Agreed.
In general I think we should prepare for BLS/sd-boot. To solve the snapshot issue I made a PoC that hooks into snapper while ago: https://build.opensuse.org/package/show/home:lnussel:legacyfree/kernel-insta...
At the very least, BLS permits greater flexibility in bootloader selection, while enhancing consistency from the user perspective. Of course not everything can be abstracted at such a low-level, but I think it helps users when the user facing portions of bootloader configuration and logic are the same.
Works on both traditional systems as well as MicroOS. Here's an image that boots with sd-boot and has snapshots support: https://build.opensuse.org/package/binaries/home:lnussel:legacyfree/openSUSE...
Also some upstream discussion: https://github.com/systemd/systemd/pull/23841
In the long run we need some more radical changes though. I don't think /.snaphots, transactional-update and overlayfs hacks to fool packages are things to keep going forward.
Agreed on all these points, except "transactional-update" because I'm not sure exactly what it refers to. Also I'm seeing more BLS changes in the last few days that aren't yet merged. https://github.com/systemd/systemd/pull/24521/files -- Chris Murphy
participants (9)
-
Alberto Planas -
Carlos E. R. -
Chris Murphy -
Jacob Michalskie -
Ludwig Nussel -
Michael Chang -
Richard Brown -
Stefan Seyfried -
Thorsten Kukuk