[opensuse] Q: Burning backup CD/DVD incrementally
As regular readers might recall, I have much of my system arranged as 5G partitions of that I can back up each partition onto a DVD. I use K3B. I'm looking at doing selected, perhaps "incremental', backups. It is easy enough to select , for example, changed files using FIND. FIND produces a list, but K3B and the tools for making ISO images want files in a directory. The best I can think of is to have a dummy directory that has symlinks to the files generated by FIND. It seems a bit of a kludge. I'm certainly not happy about the 'tree'. Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dne čtvrtek 11. ledna 2018 16:05:51 CET, Anton Aylward napsal(a):
As regular readers might recall, I have much of my system arranged as 5G partitions of that I can back up each partition onto a DVD. I use K3B.
I'm looking at doing selected, perhaps "incremental', backups. It is easy enough to select , for example, changed files using FIND. FIND produces a list, but K3B and the tools for making ISO images want files in a directory.
The best I can think of is to have a dummy directory that has symlinks to the files generated by FIND. It seems a bit of a kludge. I'm certainly not happy about the 'tree'.
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
Two quick solutions came to my mind: 1) If You are running Btrfs, You can burn the snapshots (perhaps packed by tar to that ~4 GB blocks). 2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) - it makes incremental backups (tar, zip, gpg) - You can set size of volumes, You can compress the volumes, if require also encrypt. In both cases You'll have one directory of files of given size and You can burn them. -- Vojtěch Zeisek https://trapa.cz/
On 11/01/18 10:12 AM, Vojtěch Zeisek wrote:
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
Two quick solutions came to my mind: 1) If You are running Btrfs, You can burn the snapshots (perhaps packed by tar to that ~4 GB blocks).
I don't use BtrFS. Let's not have a discussion about that.
2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) - it makes incremental backups (tar, zip, gpg) - You can set size of volumes, You can compress the volumes, if require also encrypt. In both cases You'll have one directory of files of given size and You can burn them.
Well, I installed duplicity and deja-dup. I started deja-dup and it stated it was doing an initial scan for the first backup. Then it told me it was out of space. There's not much in the way of command line options and I never got the point where the GUI offered options. I tried it on a less populated FS, one of my ones under /var that has only 15% occupancy. After a short while the machine froze. I can't think of anything else that was going on to account for that, I was just watching the scan list; it stopped as the machine froze. Neither ctl-c nor any three fingered salute worked, I had to power cycle. I also looked at the man page for duplicity as was very put off. Much to much to worry about. For the record: My full backups of the file systems on the 5G partitions using k3B has the advantage that they are not TAR'd. I can mount the DVD and it is just another file system. I can read or copy any file without having to any special program. The idea of doing incremental to CDs, or incremental of a number of the 5G's to a DVD with the same kind of access is too attractive. Running FIND to get a list by whatever criteria, date of change is a good enough one, but FIND allows for other criteria as well, remember, then generating symlinks in a dummy directory and pointing K3B at that directory is ..... well it works, but it takes a bit of hacking to get sensible pathnames under the dummy directory, and k3b isn't the most reliable tool for following symlinks. Sometimes that has an unexpected consequence. For example: in my ~/Document there is a symlink to ~/Downloads I would rather that was not followed. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Anton Aylward <opensuse@antonaylward.com> [01-11-18 14:55]: [...]
For the record: My full backups of the file systems on the 5G partitions using k3B has the advantage that they are not TAR'd. I can mount the DVD and it is just another file system. I can read or copy any file without having to any special program. The idea of doing incremental to CDs, or incremental of a number of the 5G's to a DVD with the same kind of access is too attractive.
Running FIND to get a list by whatever criteria, date of change is a good enough one, but FIND allows for other criteria as well, remember, then generating symlinks in a dummy directory and pointing K3B at that directory is ..... well it works, but it takes a bit of hacking to get sensible pathnames under the dummy directory, and k3b isn't the most reliable tool for following symlinks. Sometimes that has an unexpected consequence.
For example: in my ~/Document there is a symlink to ~/Downloads I would rather that was not followed.
you *should* consider backintime -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/01/18 03:04 PM, Patrick Shanahan wrote:
you *should* consider backintime
Reading the docco pages it seems that it requires hardlinks .... See my notes on using find to generate the list across a number of file systems and why I consider the use of symlinks. example: I have a tree of mounted FS under ~/Photography/ByYear Actually there are side threes for special projects and experimentation. So I want to backup *all* the sidecar files that have changed in the last month under that tree. All in one swoop. I use FIND. There are currently 9 such sub-branches on 5 file systems, each in a LVM LE. (read: 'Partition') -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/01/18 03:04 PM, Patrick Shanahan wrote:
you *should* consider backintime
Well I spent some time studying it and it is definitely not what I wantfor this role. I can see other backup use cases, in particular for a situation that i'm discussing and advising a salesman of my acquaintance on. But not me, here, now. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dne čtvrtek 11. ledna 2018 20:53:52 CET, Anton Aylward napsal(a):
On 11/01/18 10:12 AM, Vojtěch Zeisek wrote:
2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) - it makes incremental backups (tar, zip, gpg) - You can set size of volumes, You can compress the volumes, if require also encrypt. In both cases You'll have one directory of files of given size and You can burn them.
Well, I installed duplicity and deja-dup. I started deja-dup and it stated it was doing an initial scan for the first backup. Then it told me it was out of space.
I never used any GUI over duplicity...
There's not much in the way of command line options and I never got the point where the GUI offered options. I tried it on a less populated FS, one of my ones under /var that has only 15% occupancy. After a short while the machine froze. I can't think of anything else that was going on to account for that, I was just watching the scan list; it stopped as the machine froze. Neither ctl-c nor any three fingered salute worked, I had to power cycle.
It uses plenty of CPU when preparing the archive volume, but I never experienced such a freeze. Might be watching htop in separate terminal or so could show more. It is not expected behavior, although its true, that initial backup can take long time (later incremental backups are much faster).
I also looked at the man page for duplicity as was very put off. Much to much to worry about. -- Vojtěch Zeisek https://trapa.cz/
* Anton Aylward <opensuse@antonaylward.com> [01-11-18 10:06]:
As regular readers might recall, I have much of my system arranged as 5G partitions of that I can back up each partition onto a DVD. I use K3B.
I'm looking at doing selected, perhaps "incremental', backups. It is easy enough to select , for example, changed files using FIND. FIND produces a list, but K3B and the tools for making ISO images want files in a directory.
The best I can think of is to have a dummy directory that has symlinks to the files generated by FIND. It seems a bit of a kludge. I'm certainly not happy about the 'tree'.
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
not an iso builder but a space saving backup that provides the directory structure where one could easily create an iso. Look at backintime, is in openSUSE OSS builds. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Thu, 11 Jan 2018, Anton Aylward wrote:
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
The one that is used by basically all apps anyway behind the scenes: mkisofs. SYNOPSIS mkisofs [ options ] [ -o filename ] pathspec [pathspec ...] mkisofs [ options ] [ -o filename ] -find [find expression] See the manpage for more (esp. 'mkisofs -find -help'). Once you have the iso, burning that can be as simple as cdrecord dev=/dev/sr0 foo.iso HTH, -dnh -- printk("IOP: oh my god, they killed the ISM IOP!\n"); linux-2.6.19/arch/m68k/mac/iop.c -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/01/18 12:09 PM, David Haller wrote:
Hello,
On Thu, 11 Jan 2018, Anton Aylward wrote:
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
The one that is used by basically all apps anyway behind the scenes: mkisofs.
Yes, I'm aware of this. I've drilled down on that and other aspects of making ISOs. Frankly, the option list is downright scary1 -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 11:18 -0500, Anton Aylward wrote:
On 11/01/18 12:09 PM, David Haller wrote:
Hello,
On Thu, 11 Jan 2018, Anton Aylward wrote:
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
The one that is used by basically all apps anyway behind the scenes: mkisofs.
Yes, I'm aware of this. I've drilled down on that and other aspects of making ISOs. Frankly, the option list is downright scary1
I have created ISOs on the CLI years ago, it is not that difficult. I can help with that part. The dificulty is adjusting the size. I have no idea how to split and span to several DVDs. What I envison is going through this list one by one: <https://en.wikipedia.org/wiki/List_of_backup_software> Some are proprietary but work on Linux. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpZH4EACgkQtTMYHG2NR9VbUgCfVzVI8TzDKGV6vV8lmDRg13lB ZC4AnRl5tES3msrVXj9MxjJ39gIMAGto =xjSj -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/01/18 03:50 PM, Carlos E. R. wrote:
I have created ISOs on the CLI years ago, it is not that difficult. I can help with that part. The dificulty is adjusting the size. I have no idea how to split and span to several DVDs.
As far as I can make out there seems to be a few different approaches to all this. On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable Then there are the ones that don't keep the database, or at best rely one the label on the DVD, written or electronic, and are probably event drive, such as a completion of a particular project and hence archiving all its working papers and reports. There *may* be some way to logically organise this archiving, for example 'monthly' as the project progresses. That takes the pressure off the 'spanning multiple DVDs' issue. They don't need the database of the individual files in the same way. Once you do get into spanning multiple DVDs then you get into two issues. The decision of where to split goes hand-in-hand with the decision of what files to put on each DVD most efficiently (or optimally) -- the 'packing problem' which is at the very least NP-hard and quite probably NP-complete https://en.wikipedia.org/wiki/Bin_packing_problem https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem https://en.wikipedia.org/wiki/Packing_problems The second issue is that you need a global index of what is on each DVD. Either you have to pre-compute the "bin packing" and ahead of time and record the index on each DVD as a 'header' that says 'it's here or its on the other one' or the index has to be external. I can't say I like any of this. What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
What I envison is going through this list one by one:
Well, yes, there is that. Let me know .... -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-13 02:27, Anton Aylward wrote:
On 12/01/18 03:50 PM, Carlos E. R. wrote:
I have created ISOs on the CLI years ago, it is not that difficult. I can help with that part. The dificulty is adjusting the size. I have no idea how to split and span to several DVDs.
As far as I can make out there seems to be a few different approaches to all this.
On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.
Then there are the ones that don't keep the database, or at best rely one the label on the DVD, written or electronic, and are probably event drive, such as a completion of a particular project and hence archiving all its working papers and reports.
There *may* be some way to logically organise this archiving, for example 'monthly' as the project progresses. That takes the pressure off the 'spanning multiple DVDs' issue.
They don't need the database of the individual files in the same way.
Once you do get into spanning multiple DVDs then you get into two issues. The decision of where to split goes hand-in-hand with the decision of what files to put on each DVD most efficiently (or optimally) -- the 'packing problem' which is at the very least NP-hard and quite probably NP-complete https://en.wikipedia.org/wiki/Bin_packing_problem https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem https://en.wikipedia.org/wiki/Packing_problems
The second issue is that you need a global index of what is on each DVD. Either you have to pre-compute the "bin packing" and ahead of time and record the index on each DVD as a 'header' that says 'it's here or its on the other one' or the index has to be external.
I can't say I like any of this.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
I'm sleepy. Let me say that the best backup program I have ever used was PCtools Backup, before the 90's. For MsDOS. I used it with floppies. It was so fast that I barely had time to write the label on the floppies. It formatted them on the go. It could verify them. It wrote the index of files on the last floppy, and on a file on the hard disk. A file could span more than one floppy. It did compression on the fly, and used forward error recovery (it could recover bad sectors). It did full backup or incremental writing changed files automatically, taking note of deleted files too. I could define a list of directories and files to include or exclude. I miss it. I don't understand how thirty years later there is nothing similar for DVDs on Linux. Or any other media.
What I envision is going through this list one by one:
Well, yes, there is that. Let me know ....
In time. It is large. I have done nothing tonight. I can say that the list includes dead projects. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 12/01/18 10:43 PM, Carlos E. R. wrote:
On 2018-01-13 02:27, Anton Aylward wrote:
[snip]
On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.
DVDS are under Cdn$15/100 on the street herefor the generics that sem to have a failure rate of less than 0.01%. I don't know what their lifetime is, but it seems to be at least 5 years :-) SATA rotating Rust is about Cdn$50/Terabyte. The real advantage to me of DVD is that I have lots of shelf space but not many drive slots or SATA ports. Yes I know about external drives, but that gets into housing and power supplies. Ultimately its just as "off line" as the DVDs.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
I'm sleepy.
Let me say that the best backup program I have ever used was PCtools Backup, before the 90's. For MsDOS.
Hmmm.
I used it with floppies. It was so fast that I barely had time to write the label on the floppies. It formatted them on the go. It could verify them. It wrote the index of files on the last floppy, and on a file on the hard disk. A file could span more than one floppy. It did compression on the fly, and used forward error recovery (it could recover bad sectors).
It did full backup or incremental writing changed files automatically, taking note of deleted files too. I could define a list of directories and files to include or exclude.
I miss it. I don't understand how thirty years later there is nothing similar for DVDs on Linux. Or any other media.
Hmmmm. Sounds like a project. Who wrote the original, do you know?
What I envision is going through this list one by one:
Well, yes, there is that. Let me know ....
In time. It is large. I have done nothing tonight.
I can say that the list includes dead projects.
-- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Anton Aylward <opensuse@antonaylward.com> [01-13-18 00:41]:
On 12/01/18 10:43 PM, Carlos E. R. wrote:
On 2018-01-13 02:27, Anton Aylward wrote:
[snip]
On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.
DVDS are under Cdn$15/100 on the street herefor the generics that sem to have a failure rate of less than 0.01%. I don't know what their lifetime is, but it seems to be at least 5 years :-)
SATA rotating Rust is about Cdn$50/Terabyte.
The real advantage to me of DVD is that I have lots of shelf space but not many drive slots or SATA ports.
Yes I know about external drives, but that gets into housing and power supplies. Ultimately its just as "off line" as the DVDs.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
I'm sleepy.
Let me say that the best backup program I have ever used was PCtools Backup, before the 90's. For MsDOS.
Hmmm.
I used it with floppies. It was so fast that I barely had time to write the label on the floppies. It formatted them on the go. It could verify them. It wrote the index of files on the last floppy, and on a file on the hard disk. A file could span more than one floppy. It did compression on the fly, and used forward error recovery (it could recover bad sectors).
It did full backup or incremental writing changed files automatically, taking note of deleted files too. I could define a list of directories and files to include or exclude.
I miss it. I don't understand how thirty years later there is nothing similar for DVDs on Linux. Or any other media.
Hmmmm. Sounds like a project. Who wrote the original, do you know?
there is dar, disk archive, which can make slices and span disks/cds/dvd/floppies and write indexes to disk. Dar (Disk Archive) is a hardware-independent backup solution. Dar uses catalogs (unlike tar),which it makes it possible to extract a single file without having to read the entire archive. It is also possible to create incremental backups. Dar archives can also be created or used with the libdar library (for example, with KDar, a KDE application). This package contains the command line tools and documentation. https://software.opensuse.org/package/dar there is also a gui available but not build for openSUSE, dargui. but it must be easy to build as I have it installed/working. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/01/18 08:40 AM, Patrick Shanahan wrote:
there is dar, disk archive, which can make slices and span disks/cds/dvd/floppies and write indexes to disk.
Dar (Disk Archive) is a hardware-independent backup solution. Dar uses catalogs (unlike tar),which it makes it possible to extract a single file without having to read the entire archive. It is also possible to create incremental backups. Dar archives can also be created or used with the libdar library (for example, with KDar, a KDE application). This package contains the command line tools and documentation.
https://software.opensuse.org/package/dar
there is also a gui available but not build for openSUSE, dargui. but it must be easy to build as I have it installed/working.
I'll look into this. It seems there is a 64-tarball for DarGui. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2018-01-13 at 08:40 -0500, Patrick Shanahan wrote:
there is dar, disk archive, which can make slices and span disks/cds/dvd/floppies and write indexes to disk.
Dar (Disk Archive) is a hardware-independent backup solution. Dar uses catalogs (unlike tar),which it makes it possible to extract a single file without having to read the entire archive. It is also possible to create incremental backups. Dar archives can also be created or used with the libdar library (for example, with KDar, a KDE application). This package contains the command line tools and documentation.
https://software.opensuse.org/package/dar
there is also a gui available but not build for openSUSE, dargui. but it must be easy to build as I have it installed/working.
<https://en.wikipedia.org/wiki/Dar_(disk_archiver)> No mention of DVD there. http://dar.linux.free.fr/doc/presentation.html Ah. there: «From a filesystem, dar creates an archive, which may be split in a set of files (called slices) which size is user defined. Dar archives are suitable to be stored on floppy, CD, DVD, usb key, hard disks, and since release 2.4.0 to tapes too. But no, dar itself cannot burn a DVD. Instead the user can give dar a command to execute each time a slice is completed. Dar can perform full backup1, incremental backup2, differential backup3 and decremental backup4. It also records files that have been removed since the last backup was made, leading the restoration of a system to get the exact same state it was at the time of the differential/incremental/decremental backup (removing files that ought to be removed, adding files that ought to be added and modifing files as expected).» I don't have clear if the archive has to be created first on hard disk. I think this is the case. Maybe "Lazy Backup". <http://lazybackup.sourceforge.net/> Ha! Look, missing feature: "parchive - to recover from lost or damaged disks " The PCtools backup did that on 1980. No one I know does it now. Look at this one, interesting: <<new in 2017>> Darbrrd by Jared Jennings, to back up a few hundred gigabytes of data onto dozens of optical discs in a way that it can be restored ten years later. <https://github.com/jaredjennings/darbrrb> dar-based Blu-Ray redundant backup - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpaGqIACgkQtTMYHG2NR9XGpQCePYxn5t/H9GlGpQDbqLJo8nWb kV0Aniz+9n4lEYGHeGmN/Ms3iiotMyaJ =wxwZ -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2018-01-13 at 00:40 -0500, Anton Aylward wrote:
On 12/01/18 10:43 PM, Carlos E. R. wrote:
On 2018-01-13 02:27, Anton Aylward wrote:
[snip]
On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.
DVDS are under Cdn$15/100 on the street herefor the generics that sem to have a failure rate of less than 0.01%. I don't know what their lifetime is, but it seems to be at least 5 years :-)
SATA rotating Rust is about Cdn$50/Terabyte.
The real advantage to me of DVD is that I have lots of shelf space but not many drive slots or SATA ports.
Yes I know about external drives, but that gets into housing and power supplies. Ultimately its just as "off line" as the DVDs.
I bought a Blue ray writer days ago, I still have to try it. With this media: <http://www.verbatim.com/prod/m-disc/m-disc-bd-r/branded-surface.-sku-98913/> Guaranteed (!) to last 1000 Yrs. Sizes 25, 50, 100 GB. I intend to use them to backup the photo and document directories at least.I bought just a box of 25GB for the initial testing, they are a bit expensive. I don't know how to handle encryption yet.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
I'm sleepy.
Let me say that the best backup program I have ever used was PCtools Backup, before the 90's. For MsDOS.
Hmmm.
It was truly good. When it finished a floppy, it prompted for the next, and would detect it automatically without needing a key. I had a two floppy drive machine, and it would alternate the drives, so that I would have the next floppy ready for it in time - usually, it was so fast that sometimes I was not ready. It detected errors and solved them easily. Some people I think had automatic floppy feeders. The machine would take them from a pile without human intervention. I never saw one. I just googled for photos and it finds me cat feeders instead. Sigh.
I used it with floppies. It was so fast that I barely had time to write the label on the floppies. It formatted them on the go. It could verify them. It wrote the index of files on the last floppy, and on a file on the hard disk. A file could span more than one floppy. It did compression on the fly, and used forward error recovery (it could recover bad sectors).
It did full backup or incremental writing changed files automatically, taking note of deleted files too. I could define a list of directories and files to include or exclude.
I miss it. I don't understand how thirty years later there is nothing similar for DVDs on Linux. Or any other media.
Hmmmm. Sounds like a project. Who wrote the original, do you know?
No idea, it was commercial. Some commercial products had the names of the programmers hidden somewhere in the help or about windows with a secret combination of keys. For example, on Borland IDE it was Alt-I, I think. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpaFsQACgkQtTMYHG2NR9Vy6ACcDM+WzpJOOuXLoQIPmw0RbEqd sPQAnAhCzrX24IMK9kVg/I0NBgETqCKO =5Xnr -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/01/2018 16:25, Carlos E. R. wrote:
Sounds like a project. Who wrote the original, do you know?
No idea, it was commercial.
Central Point Software bought out by Symantec and dropped. https://en.wikipedia.org/wiki/PC_Tools_(software) I used to use it for backup as well. Dave P -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2018-01-13 at 16:44 +0200, Dave Plater wrote:
On 13/01/2018 16:25, Carlos E. R. wrote:
Sounds like a project. Who wrote the original, do you know?
No idea, it was commercial.
Central Point Software bought out by Symantec and dropped. https://en.wikipedia.org/wiki/PC_Tools_(software) I used to use it for backup as well.
Microsoft also bought it. A reduced or limited version. I think it was with MsDOS 5 or 6. I don't know what kind of agreement they had. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpaHQoACgkQtTMYHG2NR9VkgACfR3AMaf1ZwZCOFUzPlYmlUQNs CYQAn0+eVIf5KSv8R+hL0/6VXU+ByHOg =+C2p -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2018-01-13 at 00:40 -0500, Anton Aylward wrote:
On 12/01/18 10:43 PM, Carlos E. R. wrote:
On 2018-01-13 02:27, Anton Aylward wrote:
[snip]
On the one hand there are the 'snapshot management' types such as dev-dup and backintime that need to maintain a database and really are a dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.
I meant that the software you mention is designed for doing the backup to a single huge destination hard disk. Thus, can not be used with DVDs.
DVDS are under Cdn$15/100 on the street herefor the generics that sem to have a failure rate of less than 0.01%. I don't know what their lifetime is, but it seems to be at least 5 years :-)
SATA rotating Rust is about Cdn$50/Terabyte.
The real advantage to me of DVD is that I have lots of shelf space but not many drive slots or SATA ports.
Verbatim M-DISC 50GB seems to be 0,259€/GB. Verbatim Bd-R-M 25 GB seems to be 0,20432/GB. Verbatim M-Disco Bd-R Xl 100 GB seems to be 23,26€/disk, so 0,23/GB. Ah, found a box of 5, 91,82€, thus 0,18364/GB. Archival quality, they say they last a millennia.
Yes I know about external drives, but that gets into housing and power supplies. Ultimately its just as "off line" as the DVDs.
Well, you can use a caddy box. Insert a disk, do the backup, remove and store. And it is far easier to backup to a HD than to a number of DVDs. Plus, we know how to encrypt hard disks. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpbJ2cACgkQtTMYHG2NR9WRyQCfYW40T/dLuUFm+2ZcJxqL8qZ5 OFQAniWGMIEOrahVT+keu+/0HqR/V1Es =fnr0 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 20:27 -0500, Anton Aylward wrote: ... Forgot to comment on this.
Once you do get into spanning multiple DVDs then you get into two issues. The decision of where to split goes hand-in-hand with the decision of what files to put on each DVD most efficiently (or optimally) -- the 'packing problem' which is at the very least NP-hard and quite probably NP-complete https://en.wikipedia.org/wiki/Bin_packing_problem https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem https://en.wikipedia.org/wiki/Packing_problems
With real backup software this is not an issue at all. Files are stored one after the other on the medium, and the last one is split at the byte that doesn't fit, without mercy, and continues the following byte on the next DVD. Plain as that. Finally, there is an index that says in which DVD is each single file, and that index is stored in the last DVD, and in the hard disk. Or the other way round, first in the HD, then in the last DVD. This might be a problem if it takes a bit to calculate while the disk is burned on the fly - or not.
The second issue is that you need a global index of what is on each DVD. Either you have to pre-compute the "bin packing" and ahead of time and record the index on each DVD as a 'header' that says 'it's here or its on the other one' or the index has to be external.
See above.
I can't say I like any of this.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
Things can be improved if the slices contain compressed data, plus extra recovery data to protect from read errors in the future. All I described is what PCtools backup did back in 1980, but with floppies. Nothing new. No, I have not found software that does this in Linux. Yet, I hope. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpbIqEACgkQtTMYHG2NR9WqFwCfaARUEfaS0JeMhnCBGkJGn+AQ oz4An3yhs5r3oyGESaJfV7csGkyE8rNG =e0CL -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Carlos E. R. <robin.listas@telefonica.net> [01-14-18 04:29]:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Friday, 2018-01-12 at 20:27 -0500, Anton Aylward wrote:
...
Forgot to comment on this.
Once you do get into spanning multiple DVDs then you get into two issues. The decision of where to split goes hand-in-hand with the decision of what files to put on each DVD most efficiently (or optimally) -- the 'packing problem' which is at the very least NP-hard and quite probably NP-complete https://en.wikipedia.org/wiki/Bin_packing_problem https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem https://en.wikipedia.org/wiki/Packing_problems
With real backup software this is not an issue at all. Files are stored one after the other on the medium, and the last one is split at the byte that doesn't fit, without mercy, and continues the following byte on the next DVD. Plain as that. Finally, there is an index that says in which DVD is each single file, and that index is stored in the last DVD, and in the hard disk. Or the other way round, first in the HD, then in the last DVD. This might be a problem if it takes a bit to calculate while the disk is burned on the fly - or not.
The second issue is that you need a global index of what is on each DVD. Either you have to pre-compute the "bin packing" and ahead of time and record the index on each DVD as a 'header' that says 'it's here or its on the other one' or the index has to be external.
See above.
I can't say I like any of this.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
Things can be improved if the slices contain compressed data, plus extra recovery data to protect from read errors in the future.
All I described is what PCtools backup did back in 1980, but with floppies. Nothing new.
No, I have not found software that does this in Linux. Yet, I hope.
pretty certain dar, disk archiver, will do that, except it does not do the write to cd/dvd/bluray -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2018-01-14 at 08:52 -0500, Patrick Shanahan wrote:
* Carlos E. R. <> [01-14-18 04:29]:
On Friday, 2018-01-12 at 20:27 -0500, Anton Aylward wrote:
...
Forgot to comment on this.
Once you do get into spanning multiple DVDs then you get into two issues. The decision of where to split goes hand-in-hand with the decision of what files to put on each DVD most efficiently (or optimally) -- the 'packing problem' which is at the very least NP-hard and quite probably NP-complete https://en.wikipedia.org/wiki/Bin_packing_problem https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem https://en.wikipedia.org/wiki/Packing_problems
With real backup software this is not an issue at all. Files are stored one after the other on the medium, and the last one is split at the byte that doesn't fit, without mercy, and continues the following byte on the next DVD. Plain as that. Finally, there is an index that says in which DVD is each single file, and that index is stored in the last DVD, and in the hard disk. Or the other way round, first in the HD, then in the last DVD. This might be a problem if it takes a bit to calculate while the disk is burned on the fly - or not.
The second issue is that you need a global index of what is on each DVD. Either you have to pre-compute the "bin packing" and ahead of time and record the index on each DVD as a 'header' that says 'it's here or its on the other one' or the index has to be external.
See above.
I can't say I like any of this.
What we need is a K3B that is super-super smart, since I really don't fancy trying to all this with shell script.
Things can be improved if the slices contain compressed data, plus extra recovery data to protect from read errors in the future.
All I described is what PCtools backup did back in 1980, but with floppies. Nothing new.
No, I have not found software that does this in Linux. Yet, I hope.
pretty certain dar, disk archiver, will do that, except it does not do the write to cd/dvd/bluray
As far as I have seen, no, it doesn't. For instance, for the redundant recovery data you have to use parchive on it yourself. Tools that build on top of dar may do. For instance, this one looks promissing: «<<new in 2017>> Darbrrd by Jared Jennings, to back up a few hundred gigabytes of data onto dozens of optical discs in a way that it can be restored ten years later. <https://github.com/jaredjennings/darbrrb>» - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpbnKIACgkQtTMYHG2NR9UZYACePuvhOacIpeyM4HQL4EJTC4A/ ILMAmQFmrkMOkMW+rC4uTw4hUGVSC7FY =4HlU -----END PGP SIGNATURE-----
On 2018-01-11 16:05, Anton Aylward wrote:
As regular readers might recall, I have much of my system arranged as 5G partitions of that I can back up each partition onto a DVD. I use K3B.
I'm looking at doing selected, perhaps "incremental', backups. It is easy enough to select , for example, changed files using FIND. FIND produces a list, but K3B and the tools for making ISO images want files in a directory.
The best I can think of is to have a dummy directory that has symlinks to the files generated by FIND. It seems a bit of a kludge. I'm certainly not happy about the 'tree'.
Rather hardlinks, or the software would have to "follow" the links.
Can the readership think of an alternative? Is there a .iso builder that takes a file list or input stream of file names?
I asked about that the other day. I want to do backups to Bluray DVDs (can be 100 GB each). Same as you, not a tar (not reliable, and no gain), but files directly saved on the ISO. Needs a database that tells which DVD contains which file and version of it. And of course, should handle splitting of source across as many output DVDs as needed. And should be encrypted. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 11/01/18 06:30 PM, Carlos E. R. wrote:
The best I can think of is to have a dummy directory that has symlinks to the files generated by FIND. It seems a bit of a kludge. I'm certainly not happy about the 'tree'.
Rather hardlinks, or the software would have to "follow" the links.
Since the dummy directory is on new (temporary, 'throw-away') partition and the sources are from a variety of partitions, no, it has to be symlinks. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton -- ...and then Anton Aylward said... % % On 11/01/18 06:30 PM, Carlos E. R. wrote: % >> % >> The best I can think of is to have a dummy directory that has symlinks to the % >> files generated by FIND. It seems a bit of a kludge. I'm certainly not happy % >> about the 'tree'. % % > Rather hardlinks, or the software would have to "follow" the links. % % Since the dummy directory is on new (temporary, 'throw-away') partition and the % sources are from a variety of partitions, no, it has to be symlinks. Wait ... Why would the scratch area have to be on a separate partition? Hard links take [almost] no space, so unless the volume is crazy full there should be room to create a working dir and throw in a zillion links. HTH & HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/01/18 11:25 AM, David T-G wrote:
Anton --
...and then Anton Aylward said... % % On 11/01/18 06:30 PM, Carlos E. R. wrote: % >> % >> The best I can think of is to have a dummy directory that has symlinks to the % >> files generated by FIND. It seems a bit of a kludge. I'm certainly not happy % >> about the 'tree'. % % > Rather hardlinks, or the software would have to "follow" the links. % % Since the dummy directory is on new (temporary, 'throw-away') partition and the % sources are from a variety of partitions, no, it has to be symlinks.
Wait ... Why would the scratch area have to be on a separate partition? Hard links take [almost] no space, so unless the volume is crazy full there should be room to create a working dir and throw in a zillion links.
Go back and read what I said as 'specification' In absolute terms, maybe you're right in that it doesn't have to be on a throwaway partition. Maybe it could be a directory under /tmp for example, that was deleted afterwards. I'm well aware that hard links are just an entry in the directory with an inode number of an existing file. But really! A symlink is a small file with the of the linked file. it's really not that much more. Unless you have FS with bigblocks. And, even so, many late model FS use tricks to deal with files of only a few bytes, perhaps storing the data in the inode space. Check you MAN pages. But if you go back you'll also see mention of scatter-gather. Hard links have to be on the same file system' symlinks can cross file systems. Yes, I could have a scratch hardlink directory for backup on each FS and then point K3B at *ALL* of them, but that gets to be a nightmare from the admin & list generation POV. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton, et al -- ...and then Anton Aylward said... % % On 12/01/18 11:25 AM, David T-G wrote: % > % > ...and then Anton Aylward said... % > % % > % Since the dummy directory is on new (temporary, 'throw-away') partition and the % > % sources are from a variety of partitions, no, it has to be symlinks. % > % > Wait ... Why would the scratch area have to be on a separate partition? ... % % Go back and read what I said as 'specification' I see only four other messages from you in this thread, and 'specification' is not in any of them, so I don't think I've missed a direct quote. In your original email you proposed a dummy dir (tree?) of symlinks but also said that you weren't too happy with that idea. I don't mean to challenge you; it's your question and you can ask anything you wish :-) But I'm honestly confused, because I guess I've overlooked something. % ... % But if you go back you'll also see mention of scatter-gather. % Hard links have to be on the same file system' symlinks can cross file systems. Yeah, but aren't you the guy who has all 5G volumes so that each fits on a disc, and so doesn't it make sense to write your script tackle a volume at a time with the local "scratch backup" dir right there? That makes the script more general anyway. % % Yes, I could have a scratch hardlink directory for backup on each FS and then % point K3B at *ALL* of them, but that gets to be a nightmare from the admin & % list generation POV. OK, and that's fair enough. And, yes, traversing symlinks will get you the actual files, so that works. One nice thing about taking every volume individually is that when one contains so many changes that it would overflow the incremental backup past a single DVD you could then fire off a full backup of just that volume and thus exclude it from the incrementals. Eventually you'd probably settle down to a fluid mix of full backups and a catch-all incremental and just not have to worry about the specifics of what's going on. [I'd be sure to set some timeout per volume, too, so that you are guaranteed a fresh full every so often.] % % % % -- % A: Yes. % > Q: Are you sure? % >> A: Because it reverses the logical flow of conversation. % >>> Q: Why is top posting frowned upon? % % % -- % To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org % To contact the owner, e-mail: opensuse+owner@opensuse.org % :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/01/18 12:15 PM, David T-G wrote:
Anton, et al --
...and then Anton Aylward said... % % On 12/01/18 11:25 AM, David T-G wrote: % > % > ...and then Anton Aylward said... % > % % > % Since the dummy directory is on new (temporary, 'throw-away') partition and the % > % sources are from a variety of partitions, no, it has to be symlinks. % > % > Wait ... Why would the scratch area have to be on a separate partition? ... % % Go back and read what I said as 'specification'
I see only four other messages from you in this thread, and 'specification'
Take that as 'description'
is not in any of them, so I don't think I've missed a direct quote. In your original email you proposed a dummy dir (tree?) of symlinks but also said that you weren't too happy with that idea.
You missed out a bit. You missed out -- generating the list according to 'whatever' criteria using FIND -- using K3B pointed to the dummy dir with the symlinks and telling it to follow the symlinks which might have odd side effects if the symlinks cascade
% ... % But if you go back you'll also see mention of scatter-gather. % Hard links have to be on the same file system' symlinks can cross file systems.
Yeah, but aren't you the guy who has all 5G volumes so that each fits on a disc, and so doesn't it make sense to write your script tackle a volume at a time with the local "scratch backup" dir right there? That makes the script more general anyway.
Are you proposing a system whereby each of the "5G" partitions has its own dummy dir with symlinks that pertain ONLY to the FS on that partition, generate them all and point k3B at *all* of those dummy dirs: for each mounted file system in $LIST create $FSROOT/dummy_dir find $FSROOT --criteria | \ build sylinks list in $FSROOT/dummy_dir done k3b $( for each mounted file system in $LIST echo $FSROOT/dummy_dir done ) That assumes the whole will fit on a single DVD. But it still leaves me with the issue of .... Oh, wait, they don't have to by symlinks any more! They are all links on the same FS :-)
One nice thing about taking every volume individually is that when one contains so many changes that it would overflow the incremental backup past a single DVD you could then fire off a full backup of just that volume and thus exclude it from the incrementals. Eventually you'd probably settle down to a fluid mix of full backups and a catch-all incremental and just not have to worry about the specifics of what's going on. [I'd be sure to set some timeout per volume, too, so that you are guaranteed a fresh full every so often.]
The reason for the 5G partitions was so that I could do a full backup of the whole of the FS on that partition to a single DVD. If that was all I wanted to do then yes, I could back up the whole thing. But, as an example case, I have the "ByYear" photograph for more than a decade. in one month I may alter a scattered set across that decade. I don't want to have to make 10 full DVD backups if there is just one file changed. If this was a BtrFS rather than a ReiserFS or JFS then I could have snapper see the single change and there would be the single entry in the .snapper tree. The whole point of this exercise if to backup ONLY the changes. That's where FIND came into it. Well, OK, with FIND I can use other criteria. - I could back up only certain sidebar files - I could archive only the changed or newly produced JPGs With a bit of sophistication, I suppose, I could backup based on an EXIF field of the photographs. Tat will take some experimentation :-P -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 11:57 -0500, Anton Aylward wrote: ...
But if you go back you'll also see mention of scatter-gather. Hard links have to be on the same file system' symlinks can cross file systems.
Yes, I could have a scratch hardlink directory for backup on each FS and then point K3B at *ALL* of them, but that gets to be a nightmare from the admin & list generation POV.
But most tools by default copy the symlink, not the file pointed at. If you tell the tool to copy the file pointed at, ie, to follow the symlinks, it will do so with all symlinks, destroying the symlink structures you may have on the source. Years ago I wrote a backup script that would first create a new directory tree without files, replicating the directory tree that I was going to backup. Then the files were copied to that tree, but using "mkzftree", which compressed the files (I'm reading from the script, I tried several variants). Finally I would create an ISO image using "mkisofs -z ...", ie, resulting in a compressed DVD, which is transparently readable on Linux. AFAIK k3b does not support this. No way I know of to create such an ISO in one go, nor to estimate the size before starting. - From the command line it is possible to create an ISO taking a list of source paths, thus scriptable. Nothing of the above handles incremental backups. It could be done by doing rsyncs to another hard disk (single huge partition), using a new directory for each "increment". Each of those directories might be then copied to separate DVDs - but you need to keep the hard disk. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpZG3wACgkQtTMYHG2NR9WdrwCfb/ZBSvA9+hdU5cXGrkDgQzmc NqoAn38K9pv4Agit3fWRBRhUglgs708F =oJI4 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 21:32 +0100, Carlos E. R. wrote:
Years ago I wrote a backup script that would first create a new directory tree without files, replicating the directory tree that I was going to backup. Then the files were copied to that tree, but using "mkzftree", which compressed the files (I'm reading from the script, I tried several variants). Finally I would create an ISO image using "mkisofs -z ...", ie, resulting in a compressed DVD, which is transparently readable on Linux.
AFAIK k3b does not support this. No way I know of to create such an ISO in one go, nor to estimate the size before starting.
I have been asked offlist about this. I will translate a small howto I wrote in Spanish back on 2004 about the procedure I used to create compressed DVDs (well, I used CDs at the time, but it is the same thing). Notice that these DVDs are readable directly and normally in Linux: the kernel expands the files transparently for us. (translation made by <https://www.deepl.com/translator>, then revised by me) +++----------------- The fact that we can create compressed CDs does not seem to be well known. They are normal CDs, ISO9660 + Rock Ridge format, but with a small extension, Zisofs. The result is a CD with a capacity of approximately one gigabyte (depending on the compressibility of the files), and that the kernel allows you to mount and read as is, with the same tools we use to mount and read any CD, but on Linux - no, it doesn't work on other's systems. - ISO9660 CDs: readable on any operating system. - CDs ...Joliet: with extensions for Windows. - CDs ...RockRidge: with extensions for Linux or Unix (permissions), owners, etc.) - HFS: Same for Macs. (More: in "man mkisofs") Requirements: - Program zisofs and a patched mkisofs (SuSE does it from version 8.0 or 8.1). - 2.4.x kernel with zisofs support (SuSE has had it since version 7.3 or so). Compressed CDs use a non-standard RockRidge extension: they can only be used well on Linux, starting with kernel 2.4.14. In windows they are legible, but the files must be unpacked manually. In linux, if the kernel was compiled with zisofs support (all SuSE kernels bring it from version 8 or above, but even in 7.3 you can use it) these CDs can be used transparently, without us noticing. The advantage is obvious: we write more files on a CD. Compared to tar.gz, it is more robust (a write/read error in a tgz can render the entire archive useless), and it's faster to use, since you don't have to decompress the files to use them: simply mount the CD and read, copy, explore, etc., the files with any tool we want: console, midnight comander, konkeror, mozilla... And as it is RockRidge format, files and directories keep the original permission information. Inconveniences? There are. The main one is that it is tedious to create: while reading is transparent, generation is not. Plus, requires plenty of free disk space, such as gigabyte and half per CD (double the size of each CD). It is done in three steps - the first two need to be done in console: 1) Create a compressed copy of the tree of directories and files that if we want to backup; if we want to keep the permission information, owners, and dates, it is necessary to do it as root: mkzftree /source_tree /destination_compressed_tree Where "/destination_compressed_tree" must not exist previously, otherwise the command will give error and will not start. The "mkzftree" command has some possible options, but I use it like this. To mention a few, with "--parallelism 3" runs three simultaneous processes, for those lucky enough with several processors. We can generate several compressed directories to prepare our backup: /backup/cmp/etc, /backup/cmp/home, /backup/cmp/usr/local... etc, whichever you want. It can be seen that the files created are identical to the originals, but smaller, and illegible: they are compressed. 1.5) Group our directories into blocks of approximately 700 megabytes (slightly less) in size, which will fit each group on a CD. I usually play with mc (midnight commander) to get it. In the end, I end up with several directories: /backup/cmp-1, /backup/cmp-2, /backup/cmp-3... each one the right size. 2) Generate the ISO image to burn the CD. The basic command is: mkisofs -z -R -o image. iso /compressed_tree You can also make things better: mkisofs -z -R -quiet -graft-points \ -P "Unpublished, private backup" \ -p "Made by someone" \ -V "Disc label" \ -o image.iso /compressed_tree \ download/=/home/cer/download/ \ patches/=/var/lib/YaST/patches/i386/update/7.3/ The word "-graft-points" is used to change the directory name on the fly, like "patches/=" up there. In addition, I take advantage of this opportunity to fill in some labelling fields. The" -R" is for RockRidge, and the "-z" for zisofs - if you forget that one, the CD won't be readable. 2.5) Once the image is generated, you can test it: mount -t iso9660 -o ro, loop=/dev/loop1 image.iso /mnt and then check that "/mnt" contains exactly what you wanted to have. End with "umount /mnt" (mutandis mutandi). 3) Finally, burn the image - with the program you want, we can leave the console. I use xcdroast for that. You simply tell it to burn the image.iso image to a CD. If the image is oversized, it woll protest: listen to it. And that's it! You can read the resulting CD by mounting it as usual: mount /cdrom, for example. Don't forget to delete the intermediate files: the compressed tree, and the iso image. What about the DVDs? I don't know, I don't have a DVD burner. I think they can be created, but I can't say how. I suspect growsofs admits "-z", but I don't know. What, you find it tedious to generate? True, it is, and inflexible. Convince xcdroast or k3b developers to support zisofs; simply by admitting the "-z" option you already earn enough. Ideally we would need a system to create them directly, without going through "mkzftree", but directly into "mkisofs": the problem would then be to calculate the size. P.S: The DVDs. I have tried to create DVDs by this procedure in SuSE 9.1, and it works perfectly (record and then read it, in the same machine and in another one with 8.2); it is done with the same commands as if it were a CD, that is, with mkisofs. I haven't tried growsofs. - -----------------++- Please remember that I wrote the above on 2004, but I was using the procedure since at least 2002. Things have changed. I have not updated the writeup because I haven't used it recently. cer@Telcontar:~> zgrep -i zisofs /proc/config.gz CONFIG_ZISOFS=y So the kernel still supports the feature. DVDs should work. I have not tried Bluerays. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpaLBAACgkQtTMYHG2NR9WocACfT/nWFKpBd8+rfEK27U3KjDpb vNUAmgItMNlz33HxblKRiVoijBASxrn5 =UMiW -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (7)
-
Anton Aylward -
Carlos E. R. -
Dave Plater -
David Haller -
David T-G -
Patrick Shanahan -
Vojtěch Zeisek