[opensuse-factory] grub2: booting from btrfs snapshots
Hi all, Apologize if I missed something. What are the results of the following: https://hackweek.suse.com/projects/71 Will 13.2 do this kind of stuff. It sounds quite promising. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Sat, 09 Aug 2014 21:43:40 +0400
"Matwey V. Kornilov"
Hi all,
Apologize if I missed something.
What are the results of the following: https://hackweek.suse.com/projects/71
Will 13.2 do this kind of stuff. It sounds quite promising.
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Sat, 09 Aug 2014 22:47:41 +0400
"Matwey V. Kornilov"
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
snapshots are read-only by definition. I do not know how it was supposed to work ... I Cc to Michael who added snapshot support to grub. Assuming strict requirements that arbitrary snapshot must be bootable, one possible solution would be to clone snapshot in initrd on the fly. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
10.08.2014 09:19, Andrey Borzenkov пишет:
В Sat, 09 Aug 2014 22:47:41 +0400 "Matwey V. Kornilov"
пишет: 09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
snapshots are read-only by definition. I do not know how it was supposed to work ... I Cc to Michael who added snapshot support to grub.
Assuming strict requirements that arbitrary snapshot must be bootable, one possible solution would be to clone snapshot in initrd on the fly.
AFAIK not the snapshots themselves. The snapshots are just kind of subvolumes. I've looked through snapper sources and found that it creates readonly snapshots intentionally passing appropriate flag to the btrfs ioctl. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Matwey and all, On 2014-08-09 T 22:47 +0400 Matwey V. Kornilov wrote:
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
Yes. -- Booting from a snapshot can be used in two ways: 1. You chose a RO Snapshot in the Grub2 menu. In this case you will boot into a RO Snapshot, and you should be able to work in this system. If you want to stay in this "state", you will have to copy the RO Snapshot into a RW, reboot and work from it. Sounds complex!? Maybe, but there is a reason for this: The assumption is that the majority of people will use RO Snapshots as "well known states" into which they want to jump back in case of emergency. If we would boot into such a snapshot "RW", the state would be changed, would not be "well known" anymore. In other words, it would be a "one time use well known state". Not very helpful, ... For that reason, booting into "RO" and changing to RW afterwards is a right way, as it preserves the "well known state" in the RO snapshot. The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2. 2. If you are in a running system, you can chose: "snapper rollback <number>", reboot, and you are in a RW copy of <number>. Hope this explains (a bit). so long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Aug 11, 2014 at 10:06:45AM +0200, Matthias Eckermann wrote:
Hello Matwey and all,
On 2014-08-09 T 22:47 +0400 Matwey V. Kornilov wrote:
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
Yes. -- Booting from a snapshot can be used in two ways:
1. You chose a RO Snapshot in the Grub2 menu. In this case you will boot into a RO Snapshot, and you should be able to work in this system. If you want to stay in this "state", you will have to copy the RO Snapshot into a RW, reboot and work from it.
Sounds complex!? Maybe, but there is a reason for this: The assumption is that the majority of people will use RO Snapshots as "well known states" into which they want to jump back in case of emergency. If we would boot into such a snapshot "RW", the state would be changed, would not be "well known" anymore. In other words, it would be a "one time use well known state". Not very helpful, ... For that reason, booting into "RO" and changing to RW afterwards is a right way, as it preserves the "well known state" in the RO snapshot.
The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2.
2. If you are in a running system, you can chose: "snapper rollback <number>", reboot, and you are in a RW copy of <number>.
This part does not work currently because some necessary changes in grub2 is not yet submitted to factory. regards, Michael
Hope this explains (a bit).
so long - MgE
-- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11 August 2014 10:30, Michael Chang
On Mon, Aug 11, 2014 at 10:06:45AM +0200, Matthias Eckermann wrote:
Hello Matwey and all,
On 2014-08-09 T 22:47 +0400 Matwey V. Kornilov wrote:
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
Yes. -- Booting from a snapshot can be used in two ways:
1. You chose a RO Snapshot in the Grub2 menu. In this case you will boot into a RO Snapshot, and you should be able to work in this system. If you want to stay in this "state", you will have to copy the RO Snapshot into a RW, reboot and work from it.
Sounds complex!? Maybe, but there is a reason for this: The assumption is that the majority of people will use RO Snapshots as "well known states" into which they want to jump back in case of emergency. If we would boot into such a snapshot "RW", the state would be changed, would not be "well known" anymore. In other words, it would be a "one time use well known state". Not very helpful, ... For that reason, booting into "RO" and changing to RW afterwards is a right way, as it preserves the "well known state" in the RO snapshot.
The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2.
2. If you are in a running system, you can chose: "snapper rollback <number>", reboot, and you are in a RW copy of <number>.
This part does not work currently because some necessary changes in grub2 is not yet submitted to factory.
regards, Michael
Why not? sounds like something we'd really like to have in openSUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Aug 11, 2014 at 11:48:09AM +0200, Richard Brown wrote:
On 11 August 2014 10:30, Michael Chang
wrote: On Mon, Aug 11, 2014 at 10:06:45AM +0200, Matthias Eckermann wrote:
Hello Matwey and all,
On 2014-08-09 T 22:47 +0400 Matwey V. Kornilov wrote:
09.08.2014 22:34, Matwey V. Kornilov пишет:
09.08.2014 21:51, Andrey Borzenkov пишет:
There were quite a number of commits to support booting from btrfs snapshots. So I assume this should work, at least under some restrictions.
Just installed 13.2, there are grub2-snapper-plugin package, but it doesn't work out of the box. Trying to understand what is to do with it.
Ok. Get it working. When I boot into the snapshot, the / seems to be mounted as ro, and the OS is almost not working. Is it supposed to be so?
Yes. -- Booting from a snapshot can be used in two ways:
1. You chose a RO Snapshot in the Grub2 menu. In this case you will boot into a RO Snapshot, and you should be able to work in this system. If you want to stay in this "state", you will have to copy the RO Snapshot into a RW, reboot and work from it.
Sounds complex!? Maybe, but there is a reason for this: The assumption is that the majority of people will use RO Snapshots as "well known states" into which they want to jump back in case of emergency. If we would boot into such a snapshot "RW", the state would be changed, would not be "well known" anymore. In other words, it would be a "one time use well known state". Not very helpful, ... For that reason, booting into "RO" and changing to RW afterwards is a right way, as it preserves the "well known state" in the RO snapshot.
The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2.
2. If you are in a running system, you can chose: "snapper rollback <number>", reboot, and you are in a RW copy of <number>.
This part does not work currently because some necessary changes in grub2 is not yet submitted to factory.
regards, Michael
Why not? sounds like something we'd really like to have in openSUSE
It will be. The problem is that it has to be fully tested and functional to a certain degree and work as the PM expectations before publishing. Otherwise factory user would be affected or suffered from it's immature status. Regards, Michael
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
11.08.2014 12:06, Matthias G. Eckermann пишет:
The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2.
Why this can not be done inside initrd? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Matwey and all, On 2014-08-11 T 21:25 +0400 Matwey V. Kornilov wrote:
11.08.2014 12:06, Matthias G. Eckermann пишет:
The perfect solution would be, if Grub2 would be able to clone the (existing) RO into a (new) RW btrfs snapshot. But I don't know if it is even possible to implement this into Grub2.
Why this can not be done inside initrd?
For "full system rollback" (or at least "close to full system rollback") you need initrd and kernel/vmlinuz to be _inside_ the snapshot. In other words: once you are in the initrd, it's too late already. Does that explain (a bit)? So long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
2014-08-11 21:32 GMT+04:00 Matthias G. Eckermann
For "full system rollback" (or at least "close to full system rollback") you need initrd and kernel/vmlinuz to be _inside_ the snapshot.
In other words: once you are in the initrd, it's too late already.
Does that explain (a bit)?
Wait... I see that the grub2 snapshot entries looks for vmlinuz and initrd into /.snapshots/*. So, I think we really take the kernel and initrd from the snapshot. Until initrd mounts / we don't perform disc access, so here we can decide whether we need rw-clone the snapshot before the mounting. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Aug 11, 2014 at 09:46:57PM +0400, Matwey V. Kornilov wrote:
2014-08-11 21:32 GMT+04:00 Matthias G. Eckermann
: For "full system rollback" (or at least "close to full system rollback") you need initrd and kernel/vmlinuz to be _inside_ the snapshot.
In other words: once you are in the initrd, it's too late already.
Does that explain (a bit)?
Wait... I see that the grub2 snapshot entries looks for vmlinuz and initrd into /.snapshots/*. So, I think we really take the kernel and initrd from the snapshot. Until initrd mounts / we don't perform disc access, so here we can decide whether we need rw-clone the snapshot before the mounting.
Booting to btrfs snapshot is target to fulfill full system rollback requirement, you can consider it's implementation a necessary piece in full system rollback's bootloader integration. While booting to an arbitary ro snasphot as rw is possible, it's not implemented in initird and no effort was spent on it because that's currently not the goal of snapshoting booting. That's my understanding to this topic and hope it helpful. Thanks, Michael
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 12.08.2014 08:20, Michael Chang wrote:
Booting to btrfs snapshot is target to fulfill full system rollback requirement, you can consider it's implementation a necessary piece in full system rollback's bootloader integration.
What do you mean by 'full system rollback requirement' here? What I would finally expect is the behavior like ChromeOS/CoreOS: after failing to boot state N it tries to boot state N-1 and so on. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-08-12 T 11:30 +0400 Matwey V. Kornilov wrote:
On 12.08.2014 08:20, Michael Chang wrote:
Booting to btrfs snapshot is target to fulfill full system rollback requirement, you can consider it's implementation a necessary piece in full system rollback's bootloader integration.
What do you mean by 'full system rollback requirement' here?
The development of the feature "make Grub2 boot from a btrfs snapshot" was driven by the requirement for SUSE Linux Enterprise 12 to be able to rollback to a well known state. In that context several options have been considered how to implement this. As described, we have two cases: 1. Boot fails -> reboot into an RO snapshot 2. From a running system reboot into an older state We consider #2 the case which more often will be used (Actively rollback), while #1 (Boot totally fails) is considered a "worst case" situation, which hopefully nobody ever runs into, yet where having a RO snapshot to boot into is more than any, let's say, "mainstream" Linux distribution offered in the past. The openSUSE 13.2 codebase currently inherits what has been implemented for SUSE Linux Enterprise 12. Obviously optimization shall happen, and your input and code contributions are welcome.
What I would finally expect is the behavior like ChromeOS/CoreOS: after failing to boot state N it tries to boot state N-1 and so on.
This is a long term goal, yes. There are a few challenges on the way though, e.g. how to realize that boot failed (without special hardware), how to implement this on all hardware architectures (there is a world beyond x86-64!). Ideas welcome, ... so long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
12.08.2014 11:49, Matthias G. Eckermann пишет:
In that context several options have been considered how to implement this. As described, we have two cases: 1. Boot fails -> reboot into an RO snapshot 2. From a running system reboot into an older state
In the case #2, you can (by accident) reboot the system into an older state remotely (via ssh), but network does not start with RO root. That could be the problem.
We consider #2 the case which more often will be used (Actively rollback), while #1 (Boot totally fails) is considered a "worst case" situation, which hopefully nobody ever runs into, yet where having a RO snapshot to boot into is more than any, let's say, "mainstream" Linux distribution offered in the past.
No doubts.
There are a few challenges on the way though, e.g. how to realize that boot failed (without special hardware), how to implement this on all hardware architectures (there is a world beyond x86-64!). Ideas welcome, ...
For ARM, u-boot probably can boot from btrfs also, but not sure. The single one possibility I see is to mark somehow the snapshot. Then bootloader clear the flag before the booting and systemd set the flag after the boot (or doesn't mark if the boot was failed). Then the question is how to return to bootloader after failed boot. I see two possibilities here either reboot_on_panic or use watchdog timer. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-08-14 T 19:30 +0400 Matwey V. Kornilov wrote:
12.08.2014 11:49, Matthias G. Eckermann пишет:
In that context several options have been considered how to implement this. As described, we have two cases: 1. Boot fails -> reboot into an RO snapshot 2. From a running system reboot into an older state
In the case #2, you can (by accident) reboot the system into an older state remotely (via ssh), but network does not start with RO root. That could be the problem.
If you are using the suggested method to achieve #2 (this is: "snapper rollback <number>"), then snapper will do everything for you, including creating a new RW snapshot and set this as the default for the next reboot => no risk to boot into an RO snapshot. This should work on the most recent 13.2 snapshot with btrfs. If not, it's worth a bugzilla in my view. so long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX Products GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (5)
-
Andrey Borzenkov
-
Matthias G. Eckermann
-
Matwey V. Kornilov
-
Michael Chang
-
Richard Brown