-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Tom, On 07/03/15 05:44, Thomas Taylor wrote:
Bob, would you be willing to provide the source code for your backup routine software? I'd be interested in how it specifies drives/partitions/folders. It wouldn't need to be well documented in MY case as I am fairly familiar with python but others might find that useful. (Hopefully you did that for your own future use - I forget things TOOOOO quickly).
I take your point about comments, but try to keep them to a minimum by choosing meaningful (to me) names for variables. Also, comments should explain *what* a piece of code does, not *how* it does it. The backup mountpoint and destination folders should be regular linux directories, but the snapshot folders need to be btrfs subvolumes. I'm a relative beginner with python, and owe thanks to the helpful guys on tutor@python.org for help in polishing my code and making it more 'pythonic'. ;-) I have also supplied a typical .rsync.filter file, below the python code. Any follow up should perhaps be by PM, as this is now getting OT for this list. - ---cut here---- #!/usr/bin/python3 ############################# # Module: btrfs-backup.py # Author: Robert Williams # Date: 2014/07/08 # Version: 0.4 """ This module uses rsync and btrfs-tools to create backups of selected files and folders on an externally mounted btrfs raid1 array. Backup modules currently include /, ~/Documents, ~/Pictures, ~/music, /etc, /srv/www, miscellaneous stuff, and a local repository containing downloaded RPMs and Joomla extensions. After rsync has completed each incremental backup, btrfs creates a new snapshot and deletes any existing snapshots that are older than a specified interval (see 'retain' variable in function 'initialise'). Note: Rsync uses the '--safe-links' option and does not follow symlinks in this script. """ ############################# # Log: # 2014/07/07 RW File created # 2014/07/08 RW Initial debugging # 2014/07/13 RW Converted creation & deletion of snapshots to functions # 2014/07/20 RW Converted python2 code to python 3 # 2014/10/10 RW Fine tuning of code in expire_snaps() # 2015/01/07 RW Expiry intervals can be set independently # 2015/02/09 RW All sources & destinations put into a class. OOP? ############################# ############################# # When adding a newbackup stanza to 'initialise(worklist=[])',do # mount LABEL=backup /home/bob/A3 # btrfs subvolume create /home/bob/A3/'newbackup' # btrfs subvolume create /home/bob/A3/'newbackup'snaps # chown -R bob:bob /home/bob/A3/'newbackup' # before running this script ############################# import datetime import glob import os, os.path import subprocess import sys src_path = "/home/bob" mnt_path = "/home/bob/A3" # should be a regular directory *not* a btrfs subvolume today = datetime.datetime.now() fname = today.strftime("%y-%m-%d_%H-%M") if not os.getuid() == 0: print("\n*** This script must be run as root. ***\n") sys.exit() subprocess.call(["mount", "LABEL=backup", mnt_path]) if not os.path.ismount(mnt_path): print("\nBackup drive is not mounted\nCheck if it is attached.\n") sys.exit() else: print("\nBackup drive mounted at", mnt_path, "\n") class Job: def __init__(self, retain, srcPath, srcFolder, syncFolder, snapVolume, snapList): self.retain = retain = datetime.timedelta(days=retain) self.srcPath = os.path.join(src_path, srcFolder) self.syncPath = os.path.join(mnt_path, syncFolder) self.snapPath = os.path.join(mnt_path, snapVolume, fname) self.snapList = glob.glob(mnt_path + snapVolume + "*" ) def initialise(worklist=[]): worklist.append(Job(90, "", "/", "system", "systemsnaps", "systemsnaps")) worklist.append(Job(90, src_path, "Documents", "documents", "docsnaps", "docsnaps")) worklist.append(Job(90, src_path, "Pictures", "pictures", "picsnaps", "picsnaps")) worklist.append(Job(90, src_path, "", "miscellaneous", "miscsnaps", "miscsnaps")) worklist.append(Job(30, "", "/etc", "etc", "etcsnaps", "etcsnaps")) worklist.append(Job(30, src_path, "music", "music", "musicsnaps", "musicsnaps")) worklist.append(Job(30, "", "/srv/www", "www", "wwwsnaps", "wwwsnaps")) worklist.append(Job(30, src_path, "download", "repo", "reposnaps", "reposnaps")) return worklist def do_sync(source, dest): subprocess.call(['rsync', '-av', '--safe-links', '--delete-excluded', '-F', source, dest]) print("\n") def create_snaps(newSnap, snapDest): subprocess.call(['btrfs', 'subvolume', 'snapshot', newSnap, snapDest]) print("\n") def expire_snaps(snapList, retainInterval): x = 0 print("Deleting snapshots older than", str(retainInterval)[:7], "...") for snapShot in snapList: snapDate = datetime.datetime.strptime(snapList[snapShot][-14:], "%y-%m-%d_%H-%M") if today - snapDate >= retainInterval: subprocess.call(['btrfs', 'subvolume', 'delete', snapList[snapShot]]) x += 1 if x == 0: print("... No snapshots older than", str(retainInterval)[:7], "found - nothing to do.\n\n") else: print("...", x, "snapshot(s) deleted.\n\n") def main(): jobs = initialise() for job in jobs: print("Backing up", job.srcPath, "to", job.syncPath, "\n") do_sync(job.srcPath, job.syncPath) create_snaps(job.syncPath, job.snapPath) expire_snaps(job.snapList, job.retain) print("\nAll backups completed.") print("\nUnmounting backup drive.\n") subprocess.call(['umount', mnt_path]) print("\n>>> Please power down the Quad external drive enclosure <<<\n") if __name__ == "__main__": main() - ---cut here--- ~/bob/.rsync-filter - ---cut here--- # rsync filters for 'miscellaneous' backup # root of backup is /home/bob # include these: + /bin/ + /ebooks/ + /guitar/ + /ScanImages/ + /security/ + /speech/ + /tuxguitar/ + /Videos_homemade/ - - /Videos_homemade/tmp + /Videos-HD/ # hidden files & folders + /.AfterShotPro/ + /.config/ + /.gnupg/ + /.gramps/ + /.hilsmpd/ + /.kde4/share/ + /.keepassx/ + /.keychain/ + /.leo/ + /.mozilla/ + /.mpd/ + /.spamassassin/ + /.ssh/ + /.thunderbird/ + /.bashrc + /.mpdconf + /.popt + /.profile + /.rsync-filter + /.tarsnaprc + /.viminfo + /.vimrc # and exclude the rest - - /* - ---cut here--- HTH Bob - -- Bob Williams System: Linux 3.16.7-7-desktop Distro: openSUSE 13.2 (x86_64) with KDE Development Platform: 4.14.3 Uptime: 06:00am up 7:55, 3 users, load average: 0.16, 0.05, 0.06 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlT7GS0ACgkQ0Sr7eZJrmU5NtACeImPDHzEHuDOTEvGMPvEtNOUN S+sAn1UmpT8liSfUJnTtnZtReZzYPFsL =zY4k -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org