[opensuse] Compression tools with data recovery ability
Hi. I want to make backup a partition and I want to compress the data. I'm worried about the possibility of the compressed files being damaged when I try to restore then. I've read that in compressed tarball you cannot recover any data after a single damaged bit. I'm looking for tools like rar that creates extra recovery files. What options are available? What I've found at https://en.wikipedia.org/wiki/Comparison_of_archive_formats#Comparison and available in the repositories are DAR and rar. Lzip also has a recovery utility called Lziprecover. Parchive is another related utility: https://en.wikipedia.org/wiki/Parchive I haven't used any of these except rar and I don't know if the have some kind of problems. And I don't know how other compression tools like gzip and bzip behave in case of file damage. I have used dd + tar.gz previously to make backups but I'm not sure if it is a good option anymore. Any hints? Thanks in advance. Greetings. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2014-08-06 a las 13:09 +0200, jcsl escribió:
Hi.
I want to make backup a partition and I want to compress the data. I'm worried about the possibility of the compressed files being damaged when I try to restore then. I've read that in compressed tarball you cannot recover any data after a single damaged bit.
Correct.
I'm looking for tools like rar that creates extra recovery files. What options are available?
Me too. Rar is the only one I know, but I'm unsure it supports all Linux file attributes and permissions, symlinks, hardlinks, etc. And of course, it is payware.
What I've found at
https://en.wikipedia.org/wiki/Comparison_of_archive_formats#Comparison
and available in the repositories are DAR and rar. Lzip also has a recovery utility called Lziprecover. Parchive is another related utility:
Parchive, if it is what I'm thinking of, is far from easy to use. It is not transparent. I have seen some compression tools with promise, but I have not tested them, or are listed as beta :-(
behave in case of file damage. I have used dd + tar.gz previously to make backups but I'm not sure if it is a good option anymore. Any hints?
In the event of media damage, or simple data corruption, badly, or not at all. You need to make two or three copies, and/or use external utilities (like par) to create external recovery files. With unsure results if it is the media itself what is damaged. I have not ever seen a Linux backup utility that implements both compression and forward data recovery methods - such as used, for example, by pctools backup, back in the 80's, with good results. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlPiNnwACgkQja8UbcUWM1y/IQD/UR9qQJgWRjBZ4kWnrAQUV2Mr iR3OlY6C4JlI9+9Rf34A/jGiJkD+0wzE44zXE6v5oCz1DTBnMq8H2j56FKNZqqDh =xz/F -----END PGP SIGNATURE-----
On 08/06/2014 10:06 AM, Carlos E. R. wrote:
El 2014-08-06 a las 13:09 +0200, jcsl escribió:
I want to make backup a partition and I want to compress the data. I'm worried about the possibility of the compressed files being damaged when I try to restore then. I've read that in compressed tarball you cannot recover any data after a single damaged bit.
Correct.
Not entirely. It depends on the 'damage'. You need to differentiate between the 'data' and the 'metadata'. You can flip alternate bits in a 'payload' and all you have is a corrupt file. That problem with .taz is that they tar up the files and then compress that resulting .tar file. That is different from compressing each file on the fly and tar'ing the result up into one package. This latter way the metadata is not compressed (or encrypted) and even if one file has flipped-bits they others are let alone. And what if the metadata gets damaged? Well there are tools that scan ahead to see the 'signature' of the next metadata header in a tar file. So its important to think about (a) what you are compressing and (b) the separation of data and metadata. Given that, compressing up the whole .tar file into a .taz does have the risk described by the OP, yes. But it doesn't have to be like that. Think, for a moment, of 'rsync'. It can transmit compressed data but use uncompressed metadata between the source and destination programs. I also recall 'shell archives' (q.v., go google) and it occurs to me that would be a simple way to package compressed data plus clear, tagged metadata in a single file/stream.
I have not ever seen a Linux backup utility that implements both compression and forward data recovery methods - such as used, for example, by pctools backup, back in the 80's, with good results.
Good point. But I wonder if many of those DOS/Windows tools used the separate metadata vs compressed data I spoke of? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2014-08-06 a las 11:04 -0400, Anton Aylward escribió:
On 08/06/2014 10:06 AM, Carlos E. R. wrote:
That problem with .taz is that they tar up the files and then compress that resulting .tar file.
Correct. There is an alternate procedure, with cpio, to compress files first, then archive the result. If there is damage it typically only affects one file. The other way round, the problem is that you simply can not expand the gzip compression if one byte is damaged - I have been bitten by that. It is possible, though, that the location of the damage is critical, as you say, metadata vs payload. I dunno.
I have not ever seen a Linux backup utility that implements both compression and forward data recovery methods - such as used, for example, by pctools backup, back in the 80's, with good results.
Good point. But I wonder if many of those DOS/Windows tools used the separate metadata vs compressed data I spoke of?
I have recovered sucesfully backup sets done 15 years before using about 80 flopies, with damaged, bad sectors. The software was able to recover everything, after some thinking. Of course, this particular software also used code of their own to manage the flopy drive unit. They did not use msdos for it. It would be curious to learn what they did... I have the very strong feeling they changed the floppy format metadata as well. They also did tape backups, I think, but I did not have a unit at that time. Too expensive for me. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlPiWUQACgkQja8UbcUWM1w9QwD/RLsZ3nZNF656nUCabo6ycagS jiCrQGpU/1cocGfxraQA/AphajTmpAlfAr0Wi6wR+69LNBmZlC9BTKSdRonuGf6z =6C4x -----END PGP SIGNATURE-----
Hello, On Wed, 06 Aug 2014, Carlos E. R. wrote:
Correct. There is an alternate procedure, with cpio, to compress files first, then archive the result. If there is damage it typically only affects one file.
Are you sure? I can't find it in the info page at a quick glance. I know 'afio' does work that way. You can find it in the Archiving repo. -dnh -- The goal of Computer Science is to build something that will last at least until we've finished building it. -- BSD fortune file -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2014-08-06 a las 20:07 +0200, David Haller escribió:
On Wed, 06 Aug 2014, Carlos E. R. wrote:
Correct. There is an alternate procedure, with cpio, to compress files first, then archive the result. If there is damage it typically only affects one file.
Are you sure? I can't find it in the info page at a quick glance.
I had a backup script, that was packaged by openSUSE several releases back, at SuSE times, that worked that way, as far as I recall.
I know 'afio' does work that way. You can find it in the Archiving repo.
Maybe I'm confused about what exactly they used... The backup done by YaST does something like that with tar and gz. It doesn't gzip the entire tar, it uses smaller "tars", I think. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlPi1kMACgkQja8UbcUWM1zQswD/V9c5GBmVS3TLfCuBk19tLKSM 40G7oKzIRGtLu2Sa1WUA+wcaHgxseOoNukzKZG0p+K/QlwgMzs/EmrP+1UIsobPO =bp4/ -----END PGP SIGNATURE-----
Am Mittwoch, 6. August 2014, 13:09:39 schrieb jcsl:
[...] I want to make backup a partition and I want to compress the data.
What do you mean by "partition" exactly? An exact image of the partition? Or all files of the file system on that partition?
I'm worried about the possibility of the compressed files being damaged when I try to restore then. I've read that in compressed tarball you cannot recover any data after a single damaged bit.
That is the worst case scenario. Compression tools often use a limited block size (bzip2 uses a default block size of 900,000 bytes) and check sums. Thus, the next undamaged block could be found and the rest of the archive might be recovered. E.g., bzip2 has a tool for recovery: | 2.6. RECOVERING DATA FROM DAMAGED FILES | [...] | bzip2recover should be of most use dealing with large .bz2 files, as these | will contain many blocks. It is clearly futile to use it on damaged | single-block files, since a damaged block cannot be recovered. If you wish | to minimise any potential data loss through media or transmission errors, | you might consider compressing with a smaller block size. http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#recovering
I'm looking for tools like rar that creates extra recovery files. What options are available?
First, if you want to backup all files of a partition, you could compress each file individually. Thus, a single damaged bit may affect one file at most: http://dar.linux.free.fr/doc/FAQ.html#compression dar can use Parchive to add parity files: http://dar.linux.free.fr/doc/usage_notes.html#Parchive
What I've found at
https://en.wikipedia.org/wiki/Comparison_of_archive_formats#Comparison
and available in the repositories are DAR and rar.
I used dar to compress and encrypt my backups.
[...] I haven't used any of these except rar and I don't know if the have some kind of problems.
I bet they will. :) However, the question is whether you will ever run into them.
And I don't know how other compression tools like gzip and bzip behave in case of file damage.
It looks like corrupted gzip files are the worst case scenario you described above.
I have used dd + tar.gz previously to make backups but I'm not sure if it is a good option anymore. Any hints? [...]
If it is sufficient to backup the individual files of a native linux partition (e.g., ext2/3/4) dar & par should do the trick. Gruß Jan -- Acceptance without proof is the fundamental characteristic of all religions. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Anton Aylward -
Carlos E. R. -
David Haller -
Jan Ritzerfeld -
jcsl