On 17 January 2018 at 20:04, Michael Hamilton wrote:

On Thu, 18 Jan 2018, jdd@dodin.org wrote:

...

Hello,

I want a copy of my /

I don't really plan to restore the hole thing with the backup, but I need to find again the various config files I may have spread in / to use them in the future, in case of disaster.

rsync seems *not* to be able to do this, at least with the usual -av options.

as far as I can tel, rsync recurses int he .snapshot folder and fills the target.

I don't really understand what is said there:

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand....

"Initial Bootstrapping"

I don't really need incremental backups, I only care of the last config

the source is / (excluding other file system used by /home or data), the target any other disk, for example usb HDD

what may I do?

thanks -- http://dodin.org

Why not just use the below? btrfs send / | btrfs receive /mnt/backup That will backup the root subvolume (but none of the other subvolumes) to a different btrfs filesystem or subvolume located at /mnt/backup -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 18 January 2018 at 10:44, jdd@dodin.org wrote:

Le 18/01/2018 à 10:39, Richard Brown a écrit :

...

Why not just use the below?

btrfs send / | btrfs receive /mnt/backup

That will backup the root subvolume (but none of the other subvolumes) to a different btrfs filesystem or subvolume located at /mnt/backup

right now my backup hdd is ext4, is btrfs on target necessary? (I can manage this)

thanks

jdd

For btrfs send/receive to work (which would be easiest), you do, yes. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 01/18/2018 08:14 PM, Andrei Borzenkov wrote:

18.01.2018 12:39, Richard Brown пишет:

...

On 17 January 2018 at 20:04, Michael Hamilton wrote:

...

On Thu, 18 Jan 2018, jdd@dodin.org wrote:

...

Hello,

I want a copy of my /

...

...

Why not just use the below?

btrfs send / | btrfs receive /mnt/backup

That will backup the root subvolume (but none of the other subvolumes) to a different btrfs filesystem or subvolume located at /mnt/backup

I'm missing instructions how to recreate bootable openSUSE installation from /mnt/backup ...

Me too. I know two possibilities. One is install fresh from YaST, then overwrite it from backup. I have never tried and I have my doubts. Maybe overwrite using that send-receive trick. The assumption is that YaST will create the proper volumes. Oh, wait, send-receive only copies the root subvolume. The other is to recreate from dd image. Maybe clonezilla, but it is not a true imager, as it doesn't image the boot sectors (it reinstalls grub instead). -- Cheers / Saludos, Carlos E. R. (from openSUSE Malachite, Leap 42.3 x86_64 (ssd)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Le 18/01/2018 à 21:23, Richard Brown a écrit :

jdd's original post clearly stated that he was NOT interested in fully restoring a system, just wanting to backup the rootfs to have a reference for things like config files

exactly. Two things: * when I have a disaster, I like better rebuild a system from scratch with the most up to date openSUSE version. During my usual work, I very often try applications and don't like them or don't like them anymore, so starting from scratch is a good way of cleaning :-) * recovering from disaster as documented need very cautious preliminary work that is aimed to system that have to recover fast. Not my case (I have more than one computer I can use :-)

It was a fair request, simple, and yet unique enough that there is no clear documentation for that scenario

with ext4 it's pretty simple, just rsync the / with -x but rsync do not cope well with .snapshots (I just discovered this folder :-(), but I could get a pretty good result copying / with dolphin, but excluding a lot of things, liken /sys, /proc/ /dev/ /.snapshot, /run, /windows... thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 19 January 2018 at 01:55, Carlos E. R. wrote:

...

Andrei your contribution to the thread is unwelcome, unhelpful to the original poster, and pointless to the wider world in the light of the facts that full system recovery is not the context of jdd's support request and that full system recovery options are extensively documented.

Andrei comments are very welcome by others, though. Threads have a life of their own, people add more questions to them, and in the end we all learn, including of course the OP.

Sorry, but we have asked many times for that information.

Fine, I wrote the following just for you https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/ Enjoy, -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 20 January 2018 at 04:45, Michael Hamilton wrote:

On Sat, 20 Jan 2018, Richard Brown wrote:

...

On 19 January 2018 at 01:55, Carlos E. R. wrote:

...

Fine, I wrote the following just for you

https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/

Enjoy,

Thanks for that, it confirms some past research I'd done. But I'm still uncertain about your approach. When I last looked at this (see previous email in thread from me) I also saw many other subvolumes:

@/var/lib/machines @/var/lib/mailman @/var/lib/mariadb @/var/lib/mysql @/var/lib/named @/var/lib/pgsql @/var/log @/var/spool

Is creating the above unnecessary?

Like the article says, my guide is suitable only for Tumbleweed 20180117 (ie the latest snapshot), Leap 15, and SLE 15 or later These versions contain the new structure which has a single /var subvolume This is the culmination of months of work changing many packages. Older SUSE versions had important root filesystem data in the /var filetree, specifically the rpm database in /var/lib/rpmdb (put there by rpm) and fillup-templates in /var/adm/fillup-templates (put there by 207 packages). These have been progressively relocated in Tumbleweed over the last months (rpmdb is now in /usr/lib/sysimage/rpm for example) enabling us to introduce the new /var-as-a-subvolume structure this week The old structure will still be supported for those people upgrading, we won't be automatically migrating users data from the old structure to the new structure, but I did not want to document my guide for the old structure given it was a pain in the arse to work with compared to the shiny new future we have. That's also why I put the relevant versions in bold in the article - if anyone follows my guide on earlier versions of our distributions, they will end up with a broken system when rolling back, because things like the rpm database will not be snapshotted due to the rpm package in those versions still using it's old location

Are some subvolumes created by software installs?

Yes, /var/lib/machines is created by systemd/machined - at time of writing it is the only example I know of software creating it's own subvolume, and I just recently installed every-single-package in Tumbleweed in the hunt for others.

I had thought this might be the case and that I would need to use some commands to discover the list of actual subvolumes?

`btrfs subvolume list /` will show all of the subvolumes in the `/` partition the `btrfs` command is really useful, I'd recommend reading up on it's capabilities if you have any similar questions regarding subvolumes or other btrfs functionality.

I thought of creating a script that could replicate the structure of an existing root, but number of steps and amount of error checking put me off.

Well that's what my guide in essence is doing, for the new default subvolume layout (and as /var/lib/machines is a subvolume within a subvolume managed by systemd we don't need to care about it in this case) The challenge with any analysis of an existing system would be identifying whether or not a subvolume was created by the system (ie. YaST) and therefore certainly needs to be replicated, or whether the subvolume was created dynamically by the needs of the software (eg. machined) in which case whether or not it needs to be replicated is more ambiguous. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Sat, 20 Jan 2018, Richard Brown wrote:

On 20 January 2018 at 04:45, Michael Hamilton wrote:

...

On Sat, 20 Jan 2018, Richard Brown wrote:

...

On 19 January 2018 at 01:55, Carlos E. R. wrote:

...

...

Fine, I wrote the following just for you

https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/

Enjoy,

Thanks for that, it confirms some past research I'd done. But I'm still uncertain about your approach. When I last looked at this (see previous email in thread from me) I also saw many other subvolumes:

@/var/lib/machines @/var/lib/mailman @/var/lib/mariadb @/var/lib/mysql @/var/lib/named @/var/lib/pgsql @/var/log @/var/spool

Is creating the above unnecessary?

Like the article says, my guide is suitable only for Tumbleweed 20180117 (ie the latest snapshot), Leap 15, and SLE 15 or later

These versions contain the new structure which has a single /var subvolume

This is the culmination of months of work changing many packages. Older SUSE versions had important root filesystem data in the /var filetree, specifically the rpm database in /var/lib/rpmdb (put there by rpm) and fillup-templates in /var/adm/fillup-templates (put there by 207 packages).

That's great news. The messy var structure was one of the reasons I backed away from btrfs, it's nice that this been given some attention.

These have been progressively relocated in Tumbleweed over the last months (rpmdb is now in /usr/lib/sysimage/rpm for example) enabling us to introduce the new /var-as-a-subvolume structure this week

The old structure will still be supported for those people upgrading, we won't be automatically migrating users data from the old structure to the new structure, but I did not want to document my guide for the old structure given it was a pain in the arse to work with compared to the shiny new future we have.

That's also why I put the relevant versions in bold in the article - if anyone follows my guide on earlier versions of our distributions, they will end up with a broken system when rolling back, because things like the rpm database will not be snapshotted due to the rpm package in those versions still using it's old location

...

Are some subvolumes created by software installs?

Yes, /var/lib/machines is created by systemd/machined - at time of writing it is the only example I know of software creating it's own subvolume, and I just recently installed every-single-package in Tumbleweed in the hunt for others.

...

I had thought this might be the case and that I would need to use some commands to discover the list of actual subvolumes?

`btrfs subvolume list /` will show all of the subvolumes in the `/` partition

the `btrfs` command is really useful, I'd recommend reading up on it's capabilities if you have any similar questions regarding subvolumes or other btrfs functionality.

Yes I had read up extensively and carried out many experiments in a VM before creating my own notes which parallel your own. But mine were written over a year ago for 42.2 (see link in first message I posted in this thread). I still have concerns that some important details required for recreating the root structure are not reported by the btrfs command, such as the need to find all the directories where copy-on-write has been disabled, something like: (find `btrfs subvolume list / | awk '$NF !~ "[@]/[.]snapshots" { sub("@/","/", $NF); print $NF}'` -xdev -type d | xargs lsattr -d | awk '$1 ~ /C/ {print $NF}') 2> /dev/null It would be great if the the btrfs command could report this directly.

...

I thought of creating a script that could replicate the structure of an existing root, but number of steps and amount of error checking put me off.

Well that's what my guide in essence is doing, for the new default subvolume layout (and as /var/lib/machines is a subvolume within a subvolume managed by systemd we don't need to care about it in this case)

I was looking to create a script that could be trusted to do the job automatically like the installer is doing. But I wasn't sure whether I knew enough to cover all the different options for root and all the ways these commands might error. I often manage to screw up if there are more than 10 steps to read and follow, so I prefer to automate.

The challenge with any analysis of an existing system would be identifying whether or not a subvolume was created by the system (ie. YaST) and therefore certainly needs to be replicated, or whether the subvolume was created dynamically by the needs of the software (eg. machined) in which case whether or not it needs to be replicated is more ambiguous.

It would be good if a subvolume could somehow be tagged or labelled with info such as what created it and any other special requirements. Anyway, it sounds like there are many improvements coming through that might yet win me over to a btrfs root. Thanks for making the effort to provide your notes and for providing some insight into the recent tumbleweed changes to var. Cheers, Michael -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 20 January 2018 at 12:00, Anton Aylward wrote:

So, yes, for me /var is a separate mountable file system. You need all those "parts". What goes in them is, well 'data', but it is 'variable data' such as the databases -- and of course the spool and the logs :-)

Whether you let BtrFS implement them as subdivisions -- subvolumes -- of the single volume or implement them as a separate file system is a decision on your part. As they say "YMMV". You really need to make this decision at install time; trying to 'fix it up' afterwards is a !PAIN!. BTDT.

On any openSUSE version before Tumbleweed 20180117 or Leap 15 then your above advice is not sensible. Treating /var as a separate mountable filesystem, be it as a partition, subvolume, or separate disk, will result in a broken system in the event of using snapper to rollback your system This is primarily because the rpm database used to record which packages are installed on your system is located in /var/lib/rpmdb Therefore rolling back to a point where you had a different set of packages installed will lead to the rpm database still thinking it had your pre-rollback packages installed That's a broken system. There are also other examples, such as the fillup-templates used to popular /etc/sysconfig, which are also located in /var/adm/fillup-templates until recently. So, sure, NOW, in a freshly installed Tumbleweed and SOON in Leap users will be able to treat /var as a separate mountable filesystem without system-breaking consequences, but for the majority of openSUSE's current users it's essential for the integrity of their system that they do NOT treat /var as a separate mountable filesystem when using btrfs, snapshots, and rollback. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2018-01-18 22:12, jdd@dodin.org wrote:

Le 18/01/2018 à 20:24, Carlos E. R. a écrit :

...

...

I'm missing instructions how to recreate bootable openSUSE installation from /mnt/backup ...

Me too.

I know two possibilities.

always star a backup system from the recovery possible. dd (piped by gzip) is one of the only way I know, provided I have an exact or very similar hardware, which is never the case...

unuseful for a desktop. Do backup/archive the DATA, not the system. If config files where really centralized, there would be no problem.

but web sites, databases, config files can be nearly anywhere...

Yes, which is why we need a method to create an empty btrfs filesystem, create the volumes and subvolumes, and rsync the files back. Lastly, install the booter. I can recover from scratch a system only if it does not use btrfs. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)