Hi, Am Freitag, 28. April 2023, 09:36:57 CEST schrieb Andrei Borzenkov:

On Fri, Apr 28, 2023 at 12:24 AM Joe Salmeri <jmscdba@gmail.com> wrote:

...

I am trying to replicate that subvolume layout with several other linux distros which I need to use but after installation and on first boot it errors because it cannot find 'normal.mod' because for some reason the /boot/grub2/x86-64 subvolume is not mounted.

It's not a configuration issue because if I boot a live DVD and mount snapshot 1 and chroot into it and do mount -a then /boot/grub2/x86-64 is mounted properly.

If I switch /boot/grub2/x86_64-efi from a subvolume to a folder in snapshot 1, then the distros boot fine.

SUSE grub has patches to "mount" this subvolume at boot time and the grub configuration file as generated by SUSE will include commands to do it.

Yes: https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-bt... On top of that, with the secure boot compatible installation (as done by shim-install), the directory is not actually used or needed anymore. The prebuilt and signed grub.efi contains all modules. Cheers, Fabian

SUSE grub has patches to "mount" this subvolume at boot time and the grub configuration file as generated by SUSE will include commands to do it.

Thanks Andrei, I really appreciate that info as it explains why other distros cannot boot when it is a subvolume. I looked in my TW /boot/grub2/grub.cfg but don't see references to mount the /boot/grub2/x86_64-efi subvolume. I also grep'd in /etc/grub.d/* but still didn't find where it does that mount. Can you give me a pointer was to where you are referring in the grub configuration? Thanks!

Hi Andrei,

...

Can you give me a pointer was to where you are referring in the grub configuration?

user@uefi:~> cat /boot/grub2/x86_64-efi/load.cfg set btrfs_relative_path='y' btrfs-mount-subvol ($root) /boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi user@uefi:~>

Thanks for the pointer, but your answer raises the question...... If the file that does the mount ( /boot/grub2/x86_64-efi/load.cfg ) is located on the subvolume being mounted ( /boot/grub2/x86_64-efi/ ) then how does it read that file when the subvolume isn't mounted yet ?   There must be another piece I am missing here ??? Also, do you know if SUSE has modified grub to mount the default btrfs subvolume? If yes, are there any plans to submit that patch upstream so that other distros will pick it up? If no, then how is the default btrfs subvolume being mounted because TW grub.cfg files do NOT use the rootflags option for the kernel and the linux and initrd commands only specify the path to vmlinz and initrd relative to the default subvolume mounted as root instead of the btrfs root volume's full path. I have been doing a bunch of testing with snapper and several other linux distros and they all have the same issue in that since they don't have the default subvolume mounted they have to use the rootflags kernel option to specify it and the linux and initrd commands have to specify the full path from the btrfs root volume down to the subvolume. This breaks snapper rollback functionality on all other distros I've tried because after making the r/w snapshot snapper sets it as the new default snapshot, which on TW, works great and the next boot works from that new default snapshot, whereas on other distros since they don't mount the default snapshot, the boot is still from the original snapshot because of all the hard coded references to the snapshot # in their grub configuration. Does that make sense? In summary, what I am trying to understand is how TW is using whatever btrfs snapshot is set to the default during the boot process without hard coding the snapshot into all the grub configuration files. -- Regards, Joe

Hi Andrei, On 5/4/23 02:57, Andrei Borzenkov wrote:

Also, do you know if SUSE has modified grub to mount the default btrfs

...

subvolume?

Yes and no (yes, I know, and no, it did not). It modified grub to resolve paths on btrfs relative to default subvolume if snapshots are enabled.

Thank you so much for your responses! After writing that message, I realized that fstab specifies the fully qualified paths to the subvolumes and therefore my assumption was wrong about grub mounting the default subvolume. Your answer seems to be the "magic" that the other distros don't have which is why their grubs all have the full path to subvolumes coded in grub and which results in snapper rollback not working. Looking through grub I found the following items which I believe are what you are referring to     /etc/default/grub         SUSE_BTRFS_SNAPSHOT_BOOTING="true"     /usr/share/grub2/grub-mkconfig_lib         make_system_path_relative_to_its_root ()         {           if [ "x${SUSE_BTRFS_SNAPSHOT_BOOTING}" = "xtrue" ] &&              [ "x${GRUB_FS}" = "xbtrfs" ] ; then           "${grub_mkrelpath}" -r "$1"           else           "${grub_mkrelpath}" "$1"           fi         }     /usr/bin/grub2-mkrelpath         Which supports -r option         Other distros are using grub-mkrelpath which does NOT have the -r option Are there additional pieces which are also related to the grub changes that I have missed? I am planning to try and patch those into one of the other distros I have been testing with to see if I can get it working.

...

If yes, are there any plans to submit that patch upstream so that other distros will pick it up?

Upstream grub used default subvolume in the past. It was changed to always access full btrfs exactly because it was impossible to access anything outside of the default subvolume.

Right, that all makes sense now, but are there any plans to submit the grub modifications to resolve paths on btrfs relative to the default subvolume if snapshots are enabled ? That would allow snapper to work exactly the same on all distros and then snapper rollback would work too. Since it is a config option, they would not have to enable it or change their subvolume layout, but if they did or the installer chose to enable and use the TW subvolume layout then it would all just work.

Snapshot (rollback) is simply incompatible with how grub is using /boot by default. The way SUSE does it remains compatible with "grub at boot time is located below /boot" and "kernel is installed in /boot" and allows to still use grub-install. This breaks os-prober, but os-prober as implemented is the wrong way to multiboot Linux anyway.

Snapshot rollback works on TW ( or it did at least last time I tested it ) so I assume you mean incompatible with other distros because they don't have the TW grub patch. I was not aware that os-prober was broken, I tend to use VMs for other distros so no need for it, but I know it was working a year or so ago, because I had a VM which had TW as well as several other distros installed into the same vm.

It is quite possible to

- install grub in separate directory (subvolume) - make grub to query for current default subvolume and load its configuration from there

grub.cfg in each default subvolume would use full paths that would point to this subvolume, so each grub.cfg would always remain valid and would also work from within other distributions with os-prober. Default subvolume here would be used just as an indication of "current root". Solaris (which implemented this long ago) is using a separate zfs property to indicate currently active root (or boot environment), but this is implementation detail.

This is (almost) exactly what happens if Secure Boot is enabled, except for paths relative to the default subvolume.

This would require just implementing a command to query the default subvolume which is certainly upstreamable. For completeness one could also allow setting location of /boot/grub/grub.cfg explicitly which would go into load.cfg making grub-install "just work". The main problem of this is the fact that /boot as a grub location is really imprinted deep in the subconscious. Ironically, in the past grub-install *did* support --grub-directory which was then deprecated in favour of always using /boot.

Is there a reason the TW grub patches could not just be submitted upstream?   Since it is a configurable option, it should not break anything and other distros "could" just leave it disabled.

...

In summary, what I am trying to understand is how TW is using whatever btrfs snapshot is set to the default during the boot process without hard coding the snapshot into all the grub configuration files.

grub is using default subvolume so changing default subvolume automatically changes grub configuration.

On TW because of the grub patches, but I'm trying to get full snapper functionality working across distros.   Unless I've missed something, based on  your responses ( which have been MOST HELPEFUL) It seems that the missing piece is the grub patch which I am hoping could be submitted upstream ? -- Regards, Joe

On 5/8/23 11:48, Andrei Borzenkov wrote:

Is there a reason the TW grub patches could not just be submitted

...

upstream?

No idea. I hope nobody is going to do it and if someone does upstream will reject them. It is absolutely wrong way to implement this functionality.

That surprises me because having it resolve the paths to the default subvolume seems like a natural and easily understandable method to me. Otherwise, it would "appear" to me that the only time the default subvolume would matter is when you do a mount without using the -o subvol= or subvolid= option.

...

  Since it is a configurable option, it should not break anything and other distros "could" just leave it disabled.

Yes. And it means we have two different grub binaries which behave differently and use incompatible config file. Which is the last thing we want.

I don't follow.   Seems that there could still be one grub binary which would generate the config files based on whether the config was set or not. That's what TW does so I am confused as why you said that. Do you expect TW to get rid of its grub patch ?     If they did then snapper rollback would have the same issues in TW as it does on other distros. MicroOS also sets the default snapshot to the new snapshot after transactional-updates completes successfully.

grub-mkconfig tries everything possible to generate configuration file that is compatible with current and past grub versions. The only way we can achieve it with these patches is to generate two set of paths depending on whether this option is enabled. Which makes absolutely no sense.

I do not doubt your reasoning ( clearly you are very familiar with all of this - one of the few I've encountered ) and I am trying to understand your reasoning, but I keep coming back to this works GREAT in TW (with the patch ) and because all the other distros don't have that patch they struggle with manual methods to do a rollback ( which are  not that hard, but certainly not as simple as snapper rollback ).

I already told you how to implement this functionality with zero changes to upstream grub. Even command to get default subvolume is not needed, because the currently active root could be simply stored in grub "jump config".

Yes, and I appreciate that but at the end of that you said "The main problem of this is the fact that /boot as a grub location is really imprinted deep in the subconscious. Ironically, in the past grub-install*did* support --grub-directory which was then deprecated in favour of always using /boot." Which I took to mean that that since --grub-directory was deprecated that it also meant that the solution was not a long term answer to the issue.'

...

make grub to query for current default subvolume and load its configuration from there

If you are talking about modifying the grub source and recompiling, I'm not up to the task of modifying grub source code as I haven't touched C code in 25+ years. I also did not completely follow your how to implement method. In rereading it though, I realized that you might think that all these questions are for a multi-boot environment where I have different linux distros all installed on the same machine. That is not the case. Each linux distro is standalone and generally all are in VMs. The other distros provide methods for booting to snapshots ( with other packages installed, but their methods are different than TW uses ) but the other distros cannot handle snapper rollback because they don't recognize what the default subvolume is set to like TW does. Other distros, generally mount the active/live system as snapshot /@  whereas TW starts with /@/.snapshot/1/snapshot as the default subvolume . On TW, when you do a snapper rollback the end result is that the default snapshot was changed ( it first makes a r/o snapshot of the currently booted snapshot, then it makes a r/w snapshot of the one that you want to rollback too and then it sets that r/w snapshot as the default snapshot ) so that on the next boot you are booting from the new desired snapshot. On other distros, all the configs have the active/live system as snapshot /@ and if you want to rollback they rename/move that snapshot to a new name ( like timeshift does ) and then rename/copy the snapshot you want to rollback to as /@.    All the configs are coded to use /@ snapshot they just switch out which snapshot that really refers too. IMHO, all the other distros jump through a lot of hoops simply because they are missing the grub patch which TW has. The solution you provided seems like it is geared towards the multi boot setup which is not the setup I am talking about and wouldn't have an impact on the way grub works in the other distro vms. Sorry if I've misunderstood or misinterpreted what you are saying. Regards, Joe