-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/03/15 20:54, Malcolm wrote:

On Fri 06 Mar 2015 01:47:41 PM CST, don fisher wrote:

...

Hello,

In the pas I have user rsync to update portable disks that I can keep off site. It is not clear to me how to do the same thing with the subvolumes. I assume I could duplicate the structure and use rsync on each subvolume, but if the structure is changed how will rsync know?

There must be a tool, but I could not locate it on google or openSuse. With standard file systems all one has to do is rsync / to the other disk. Looking for a way to duplicate my system.

Don

Hi Don This should give you some ideas.... https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

In the event of something amiss you should be able to boot into a previous snapshot?

I have a similar setup, using btrfs on my desktop's internal drives and also on an external removable enclosure. I tried using btrfs send|receive and can confirm that it is much faster than rsync. However, every 'folder' has to be a btrfs subvolume, which requires a bit of advanced planning, and lacks flexibility. I ended up writing a python script that uses rsync to do the daily backup of each folder to an equivalent folder on the destination drive. This is followed by a btrfs snapshot of the backed up folder, and deletion of snapshots older than a predefined age (90 days here, but that can be changed in the python script). Combined with .rsync-filter files in each source directory and the rsync -F option, this gives me the granularity I need. I'm prepared to tolerate the increased time for each backup - currently 4.5 minutes to backup / (8.5GB), ~/Documents (5.5GB), ~/Pictures (136GB), ~/music (590GB), and various miscellaneous folders. Obviously the initial run takes much longer. Bob - -- Bob Williams System: Linux 3.16.7-7-desktop Distro: openSUSE 13.2 (x86_64) with KDE Development Platform: 4.14.3 Uptime: 06:00am up 7:55, 3 users, load average: 0.16, 0.05, 0.06 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlT6GycACgkQ0Sr7eZJrmU6y4wCdH9IIYiA0tyCyf7Ump/OuuxQu mFMAmwd3wjmUOuHl7Ip/aGgxn8FIHhhv =8JbF -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri 06 Mar 2015 04:42:21 PM CST, don fisher wrote:

On 03/06/15 14:24, Bob Williams wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/03/15 20:54, Malcolm wrote:

...

On Fri 06 Mar 2015 01:47:41 PM CST, don fisher wrote:

...

Hello,

In the pas I have user rsync to update portable disks that I can keep off site. It is not clear to me how to do the same thing with the subvolumes. I assume I could duplicate the structure and use rsync on each subvolume, but if the structure is changed how will rsync know?

There must be a tool, but I could not locate it on google or openSuse. With standard file systems all one has to do is rsync / to the other disk. Looking for a way to duplicate my system.

Don

Hi Don This should give you some ideas.... https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

In the event of something amiss you should be able to boot into a previous snapshot?

I have a similar setup, using btrfs on my desktop's internal drives and also on an external removable enclosure. I tried using btrfs send|receive and can confirm that it is much faster than rsync. However, every 'folder' has to be a btrfs subvolume, which requires a bit of advanced planning, and lacks flexibility.

I ended up writing a python script that uses rsync to do the daily backup of each folder to an equivalent folder on the destination drive. This is followed by a btrfs snapshot of the backed up folder, and deletion of snapshots older than a predefined age (90 days here, but that can be changed in the python script). Combined with .rsync-filter files in each source directory and the rsync -F option, this gives me the granularity I need. I'm prepared to tolerate the increased time for each backup - currently 4.5 minutes to backup / (8.5GB), ~/Documents (5.5GB), ~/Pictures (136GB), ~/music (590GB), and various miscellaneous folders. Obviously the initial run takes much longer.

Bob - -- Bob Williams System: Linux 3.16.7-7-desktop Distro: openSUSE 13.2 (x86_64) with KDE Development Platform: 4.14.3 Uptime: 06:00am up 7:55, 3 users, load average: 0.16, 0.05, 0.06 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2

iEYEARECAAYFAlT6GycACgkQ0Sr7eZJrmU6y4wCdH9IIYiA0tyCyf7Ump/OuuxQu mFMAmwd3wjmUOuHl7Ip/aGgxn8FIHhhv =8JbF -----END PGP SIGNATURE-----

I actually have an additional problem. I built my system on a hard drive, then purchased a SSD, both 1TB. The first thing I would like to do is move my system to the SSD. But the SSDs have limited lifespan so I was hoping to keep an image of the SSD on the existing hard drive. btrfs is supposed to optimize blocking for SSD drives, so I would like it to make the optimized partitions, which I assume will be different than on the rotating media. I would have thought somebody would have made an app like rsync to do this before putting btrfs is full use. In the medical community, for instance, one is required by law to keep off site backups in case of fire etc.

Don Hi You will find your worries about SSD's are that just worries that can be IMHO ignored these days.

I don't get large SSD's instead I use 120GB devices and bcache to feed rotating rust (hybrid drive). I have three SSD's with thousands (5000~19000) of hours with no wear or errors indicated. I don't even worry about them, all write about 6GB per day, the OCZ is rated at 20GB per day. -- Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890) SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.36-38-default up 1 day 8:17, 3 users, load average: 0.23, 0.28, 0.31 CPU AMD A4-5150M APU @ 3.3GHz | GPU Richland Radeon HD 8350G -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Mar 6, 2015 at 5:37 PM, Chris Murphy wrote:

Exactly reproducing a Btrfs volume onto another volume while retaining the deduplication of snapshots isn't possible yet, and poses some challenges.

A possible approximation is Btrfs seed device. There are some bugs in this area though so you should have backups and be prepared to depend on them. But the gist is that the volume is unmounted, made a seed, remount is forced read-only, add a new device, unmount, mount again which causes a read-write mount to be possible. You now have two devices: a read-only seed, and a read-write 2nd device. You can use 'btrfs device delete' on the seed and everything on the source is migrated to the read-write volume. Also, at the time the additional device is added and mounted, the volume gets a new volume UUID. So the seed device and its derivatives have separate UUIDs. However, subvolumes do not get new UUIDs so it's possible for there to be subvolume confusion if a workflow depends on subvolumes actually having unique UUIDs with no duplicates. *shrug* Like I said, challenges.

You can still rsync the current tree(s) excluding snapshots. There couldn't possibly be a law that requires you to use the same filesystem for the backup, as for the source. Certainly you could rsync the current tree to e.g. XFS using rsync and meet backup requirements.

In fact I advise a backup on something other than Btrfs, meaning exclusively depending on only Btrfs for backups is probably not a good idea. The development is just too active and there are still edge cases being found. So if the data is valuable, chances are only one backup isn't good enough, so it's find for one or more backups to be on Btrfs, even primary backups, but one of the backups should be on something else. -- Chris Murphy -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Mar 6, 2015 at 7:53 PM, don fisher wrote:

You are correct about the legals. It was just my lazy way to get everything saved in one operation.

I think that's a distinct weak area on all Linux distros, honestly. I'm hopeful of a path toward more stateless systems, so that it's possible to tolerate system resets without user data or app install loss, and full on hardware failures with simple reliable OS reinstall that includes an option at firstboot to do a restore of stateful system data and user data. That's probably the domain of the DE's to do that work although there's certainly room for agreement on the underlying infrastructure. All the technological pieces are there, but it still needs a logical code design, code actually written (huge barrier) and then code tested and polished (another huge barrier). But it's possible, and I think valuable. There are just too many ways to put together a Linux system, too many locations for variable data, antiquated and unhelpful hierarchy for separating these things, and we need to get to some kind of standardization/agreements to constrain the layouts being used. That would help considerably with enabling sane single button reliable backup and restore. Btrfs can make this easier, it doesn't make it harder although like I said there are some complications with its deduplication -- Chris Murphy -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Tom, On 07/03/15 05:44, Thomas Taylor wrote:

Bob, would you be willing to provide the source code for your backup routine software? I'd be interested in how it specifies drives/partitions/folders. It wouldn't need to be well documented in MY case as I am fairly familiar with python but others might find that useful. (Hopefully you did that for your own future use - I forget things TOOOOO quickly).

I take your point about comments, but try to keep them to a minimum by choosing meaningful (to me) names for variables. Also, comments should explain *what* a piece of code does, not *how* it does it. The backup mountpoint and destination folders should be regular linux directories, but the snapshot folders need to be btrfs subvolumes. I'm a relative beginner with python, and owe thanks to the helpful guys on tutor@python.org for help in polishing my code and making it more 'pythonic'. ;-) I have also supplied a typical .rsync.filter file, below the python code. Any follow up should perhaps be by PM, as this is now getting OT for this list. - ---cut here---- #!/usr/bin/python3 ############################# # Module: btrfs-backup.py # Author: Robert Williams # Date: 2014/07/08 # Version: 0.4 """ This module uses rsync and btrfs-tools to create backups of selected files and folders on an externally mounted btrfs raid1 array. Backup modules currently include /, ~/Documents, ~/Pictures, ~/music, /etc, /srv/www, miscellaneous stuff, and a local repository containing downloaded RPMs and Joomla extensions. After rsync has completed each incremental backup, btrfs creates a new snapshot and deletes any existing snapshots that are older than a specified interval (see 'retain' variable in function 'initialise'). Note: Rsync uses the '--safe-links' option and does not follow symlinks in this script. """ ############################# # Log: # 2014/07/07 RW File created # 2014/07/08 RW Initial debugging # 2014/07/13 RW Converted creation & deletion of snapshots to functions # 2014/07/20 RW Converted python2 code to python 3 # 2014/10/10 RW Fine tuning of code in expire_snaps() # 2015/01/07 RW Expiry intervals can be set independently # 2015/02/09 RW All sources & destinations put into a class. OOP? ############################# ############################# # When adding a newbackup stanza to 'initialise(worklist=[])',do # mount LABEL=backup /home/bob/A3 # btrfs subvolume create /home/bob/A3/'newbackup' # btrfs subvolume create /home/bob/A3/'newbackup'snaps # chown -R bob:bob /home/bob/A3/'newbackup' # before running this script ############################# import datetime import glob import os, os.path import subprocess import sys src_path = "/home/bob" mnt_path = "/home/bob/A3" # should be a regular directory *not* a btrfs subvolume today = datetime.datetime.now() fname = today.strftime("%y-%m-%d_%H-%M") if not os.getuid() == 0: print("\n*** This script must be run as root. ***\n") sys.exit() subprocess.call(["mount", "LABEL=backup", mnt_path]) if not os.path.ismount(mnt_path): print("\nBackup drive is not mounted\nCheck if it is attached.\n") sys.exit() else: print("\nBackup drive mounted at", mnt_path, "\n") class Job: def __init__(self, retain, srcPath, srcFolder, syncFolder, snapVolume, snapList): self.retain = retain = datetime.timedelta(days=retain) self.srcPath = os.path.join(src_path, srcFolder) self.syncPath = os.path.join(mnt_path, syncFolder) self.snapPath = os.path.join(mnt_path, snapVolume, fname) self.snapList = glob.glob(mnt_path + snapVolume + "*" ) def initialise(worklist=[]): worklist.append(Job(90, "", "/", "system", "systemsnaps", "systemsnaps")) worklist.append(Job(90, src_path, "Documents", "documents", "docsnaps", "docsnaps")) worklist.append(Job(90, src_path, "Pictures", "pictures", "picsnaps", "picsnaps")) worklist.append(Job(90, src_path, "", "miscellaneous", "miscsnaps", "miscsnaps")) worklist.append(Job(30, "", "/etc", "etc", "etcsnaps", "etcsnaps")) worklist.append(Job(30, src_path, "music", "music", "musicsnaps", "musicsnaps")) worklist.append(Job(30, "", "/srv/www", "www", "wwwsnaps", "wwwsnaps")) worklist.append(Job(30, src_path, "download", "repo", "reposnaps", "reposnaps")) return worklist def do_sync(source, dest): subprocess.call(['rsync', '-av', '--safe-links', '--delete-excluded', '-F', source, dest]) print("\n") def create_snaps(newSnap, snapDest): subprocess.call(['btrfs', 'subvolume', 'snapshot', newSnap, snapDest]) print("\n") def expire_snaps(snapList, retainInterval): x = 0 print("Deleting snapshots older than", str(retainInterval)[:7], "...") for snapShot in snapList: snapDate = datetime.datetime.strptime(snapList[snapShot][-14:], "%y-%m-%d_%H-%M") if today - snapDate >= retainInterval: subprocess.call(['btrfs', 'subvolume', 'delete', snapList[snapShot]]) x += 1 if x == 0: print("... No snapshots older than", str(retainInterval)[:7], "found - nothing to do.\n\n") else: print("...", x, "snapshot(s) deleted.\n\n") def main(): jobs = initialise() for job in jobs: print("Backing up", job.srcPath, "to", job.syncPath, "\n") do_sync(job.srcPath, job.syncPath) create_snaps(job.syncPath, job.snapPath) expire_snaps(job.snapList, job.retain) print("\nAll backups completed.") print("\nUnmounting backup drive.\n") subprocess.call(['umount', mnt_path]) print("\n>>> Please power down the Quad external drive enclosure <<<\n") if __name__ == "__main__": main() - ---cut here--- ~/bob/.rsync-filter - ---cut here--- # rsync filters for 'miscellaneous' backup # root of backup is /home/bob # include these: + /bin/ + /ebooks/ + /guitar/ + /ScanImages/ + /security/ + /speech/ + /tuxguitar/ + /Videos_homemade/ - - /Videos_homemade/tmp + /Videos-HD/ # hidden files & folders + /.AfterShotPro/ + /.config/ + /.gnupg/ + /.gramps/ + /.hilsmpd/ + /.kde4/share/ + /.keepassx/ + /.keychain/ + /.leo/ + /.mozilla/ + /.mpd/ + /.spamassassin/ + /.ssh/ + /.thunderbird/ + /.bashrc + /.mpdconf + /.popt + /.profile + /.rsync-filter + /.tarsnaprc + /.viminfo + /.vimrc # and exclude the rest - - /* - ---cut here--- HTH Bob - -- Bob Williams System: Linux 3.16.7-7-desktop Distro: openSUSE 13.2 (x86_64) with KDE Development Platform: 4.14.3 Uptime: 06:00am up 7:55, 3 users, load average: 0.16, 0.05, 0.06 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlT7GS0ACgkQ0Sr7eZJrmU5NtACeImPDHzEHuDOTEvGMPvEtNOUN S+sAn1UmpT8liSfUJnTtnZtReZzYPFsL =zY4k -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org