Hi all: For backup purposes, I need to save big dirs of data to tape. The easy way: a gzipped tarball (eg: tar zcvf ..). Another method: tar .. ; bzip2 ...-> you get a tar.bz2 which has better compression ratio. Now let's suppose I need to recover data from backup and the tape has some problems at any point. If I had saved separate files (not tarball) it's easy to recover most of the backup (only files on the broken point would be damaged and possibly couldn't be restored). But what does it happen when you have only a one big file (tarball, eg)????? Would you lose more data than the strictly affected by the tape failure? Would you lose all the data? Have you got any comparative showing which methods are the more robust ones, etc? I read that .bz2 is more "secure" (=robust) than .gz (maybe in tape howto). Any ideas / advices are welcome. TIA. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Thursday 12 April 2001 13:48, RoMaN SoFt / LLFB!! wrote:
Hi all:
For backup purposes, I need to save big dirs of data to tape. The easy way: a gzipped tarball (eg: tar zcvf ..). Another method: tar .. ; bzip2 ...-> you get a tar.bz2 which has better compression ratio.
Now let's suppose I need to recover data from backup and the tape has some problems at any point. If I had saved separate files (not tarball) it's easy to recover most of the backup (only files on the broken point would be damaged and possibly couldn't be restored). But what does it happen when you have only a one big file (tarball, eg)????? Would you lose more data than the strictly affected by the tape failure? Would you lose all the data?
You'd lose all the data, that is why using a compressed tarball on tape is a bad thing. Either compress files individually or maybe rely on the builtin hardwarecompression of the tapedrive, if you like. Personally I dropped using any compression, since the major part of what I backup are mostly compressed files anyway (gif / jpg / mpeg etc.) For text / html-files, which are quite small in comparison, you could make an archive. You could even put such an archive on tape twice, of course thereby sacrificing the space you wanted to save in the first place ;-) The only thing I definitely gzip are logfiles (because typically size decreases by 10-20 times!) YMMV
Have you got any comparative showing which methods are the more robust ones, etc?
Don't use compression unless absolutely imperative. IIRC single files from a big plain tarball CAN be recovered in the event of a tape block-error, but not when the tarball is compressed.
I read that .bz2 is more "secure" (=robust) than .gz (maybe in tape howto).
No info on that... sorry Maarten
Any ideas / advices are welcome. TIA.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Thu, 12 Apr 2001, Maarten van den Berg wrote:
For backup purposes, I need to save big dirs of data to tape. The easy way: a gzipped tarball (eg: tar zcvf ..). Another method: tar .. ; bzip2 ...-> you get a tar.bz2 which has better compression ratio.
BTW with any recent GNU tar you can use the "I" switch (instead of "z") to transparently use bzip2.
what does it happen when you have only a one big file (tarball, eg)????? Would you lose more data than the strictly affected by the tape failure? Would you lose all the data?
You'd lose all the data, that is why using a compressed tarball on tape is a bad thing. Either compress files individually or maybe rely on the builtin hardwarecompression of the tapedrive, if you like.
You wouldn't loose all the data. Both tar and bzip are just streaming so you'd be fine up to the damaged part. I'm not shure if you can recover behind the damage. There is a tar-repair tool somewhere... For a commercial tool I can recommend BRU (www.estinc.com) which is a nice enhanced tar replacement. They claim improved resistance to damaged data but luckily I haven't had to test this feature.
Personally I dropped using any compression, since the major part of what I backup are mostly compressed files anyway (gif / jpg / mpeg etc.)
That's true. We have a server full of JPG and TIFF files (160G) and there's no compression whatsoever. You only get half (or quarter) of the advertised "volume" on a tape :-(
I read that .bz2 is more "secure" (=robust) than .gz (maybe in tape howto).
I would be surprised if by robust you mean it's possible to reconstruct all data from a damaged file. Compression is all about removing redundancy in your data and reconstruction is about adding and distributing redundancy. If by robust you mean it's possible to recover partial data from a damaged file then it sounds possible but I don't know anything about the internals. Robert -- Robert Casties --------------------- http://philoscience.unibe.ch/~casties History & Philosophy of Science Tel: +41/31/631-8505 Room: 216 Institute for Exact Sciences Sidlerstrasse 5, CH-3012 Bern Uni Bern (PGP key on homepage: D7 2B DE 64 2D 65 16 A0)
On Thu, 12 Apr 2001 16:12:09 +0200 (CEST), you wrote:
BTW with any recent GNU tar you can use the "I" switch (instead of "z") to transparently use bzip2.
Which version? My gnu tar (SuSE 7.0) has the -l flag but it's not bzip2 related.
For a commercial tool I can recommend BRU (www.estinc.com) which is a nice enhanced tar replacement. They claim improved resistance to damaged data but luckily I haven't had to test this feature.
I'll have to re-look at it. I asked about this issue (backup apps) in this ml since time ago. Bru was one of the adviced programs. SuSE also ships some free backup program, but I remember I started to review one of them and it seemed a bit fuzzy. So finally I think I'm going to create some basic shell scripts and put it / them at crontab. At least I have full control and I can customize it to fit my needs.
If by robust you mean it's possible to recover partial data from a damaged file then it sounds possible but I don't know anything about the internals.
Yes, this is what I meant. It "sounds" possible to many people (/me included) but nobody seems to really confirm the issue. That's one of the reasons I asked here. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Fri, 13 Apr 2001, RoMaN SoFt / LLFB !! wrote:
On Thu, 12 Apr 2001 16:12:09 +0200 (CEST), you wrote:
BTW with any recent GNU tar you can use the "I" switch (instead of "z") to transparently use bzip2.
Which version? My gnu tar (SuSE 7.0) has the -l flag but it's not bzip2 related.
It's not a small "l" ("ell") but a capital "I" ("ieh") :-) And it doesn't figure in the manpage but I can remember using it some time ago already (tar 1.13.17 presently). [hera] ~ $ tar tIf /usr/local/tmp/linux/linux-2.2.19.tar.bz2 linux/ linux/Makefile linux/fs/ linux/fs/stat.c linux/fs/Makefile [...] Robert -- Robert Casties --------------------- http://philoscience.unibe.ch/~casties History & Philosophy of Science Tel: +41/31/631-8505 Room: 216 Institute for Exact Sciences Sidlerstrasse 5, CH-3012 Bern Uni Bern (PGP key on homepage: D7 2B DE 64 2D 65 16 A0)
On Fri, 13 Apr 2001, RoMaN SoFt / LLFB!! wrote:
On Thu, 12 Apr 2001 16:12:09 +0200 (CEST), you wrote:
BTW with any recent GNU tar you can use the "I" switch (instead of "z") to transparently use bzip2.
Which version? My gnu tar (SuSE 7.0) has the -l flag but it's not bzip2 related.
It's the letter "I" (capital 'i') and not "l" (small 'L'). # tar --help | grep bzip -I, --bzip2, --bunzip2 filter the archive through bzip2
On Fri, 13 Apr 2001, RoMaN SoFt / LLFB!! wrote:
If by robust you mean it's possible to recover partial data from a damaged file then it sounds possible but I don't know anything about the internals.
Yes, this is what I meant. It "sounds" possible to many people (/me included) but nobody seems to really confirm the issue. That's one of the reasons I asked here.
* afio is included in suse which claims to store filewise so only one file would be ruined on a read failure. * and i searched for an hour now but can not find the redundant storage thing (maybe i find it at home). * arkeia is somewhat big in our experience. * and i would prefer amanda (presumable not really smaller) as i can tweak it. -- --- Engelbert Gruber ----=~ SSG Fintl,Gruber,Lassnig A6140 Telfs Untermarkt 9 Tel. ++43-5262-64727 ----=~
Generally speaking the minute you encrypt files any error(s) will probably mangle the majority or all of the archive (so have multiple copies and make sure your tape program somehow verifies the data!). It does of course have it's place for sensitive files. Also rmemeber that the encryption is only as secure as the storage of the encryption key. No single person should have the full key; what if they quit and take a few tapes with them as an employment incentive to your competitor? As well mulitiple people should have access to the encryption keys in case your sysadmin gets hit by a truck. You should also pay attention to storing your tapes securely (i.e. to prevent theft, damage, destruction, etc.) as well as the copy offsite (i..e. a big safety deposit box and set it up wit the bank so two or more people are needed). Kurt Seifried, seifried@securityportal.com Securityportal - your focal point for security on the 'net
On Thu, 12 Apr 2001 15:07:22 +0200, you wrote:
You'd lose all the data, that is why using a compressed tarball on tape is a bad thing. Either compress files individually or maybe rely on the builtin hardwarecompression of the tapedrive, if you like.
How can you enable the hardware compression? I only have a logical device which I use with the -f flag of tar program.
The only thing I definitely gzip are logfiles (because typically size decreases by 10-20 times!) YMMV
In my case compression is useful since there are many plain text documents, .doc files, etc. Perhaps the better way is to create a .tar.bz2 archive and save it twice. BTW I'll make daily backup so if a tape results damaged I'd only loose one day of work (that being pesimist because it's difficult both tar's [original and duplicate] get damaged). =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Friday 13 April 2001 16:25, RoMaN SoFt / LLFB !! wrote:
On Thu, 12 Apr 2001 15:07:22 +0200, you wrote:
You'd lose all the data, that is why using a compressed tarball on tape is a bad thing. Either compress files individually or maybe rely on the builtin hardwarecompression of the tapedrive, if you like.
How can you enable the hardware compression? I only have a logical device which I use with the -f flag of tar program.
Dependant on the tapedrive in question. It might be a dipswitch on the drive itself and / or it might be a command you can send the drive. I use SCSI DAT drives at all times and with these you can en- or disable hardware-compression with 'mt', like this: mt -f /dev/nst0 datcompression 2 (man mt, options can differ) For other types of drives you may need other means. Maarten
* Maarten van den Berg wrote on Tue, Apr 17, 2001 at 18:31 +0200:
How can you enable the hardware compression? I only have a logical device which I use with the -f flag of tar program.
mt -f /dev/nst0 datcompression 2
I don't think that this is a nice idea always. AFAIK it may happen that you cannot read back the tape data on a different drive (maybe if it's newer or from a different vendor). Once we had a old DAT drive that were out of order. Other drives were not able to restore the backup tape. Thank god we had aother such old model... oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.
On Tue, 17 Apr 2001, Steffen Dettmer wrote:
* Maarten van den Berg wrote on Tue, Apr 17, 2001 at 18:31 +0200:
How can you enable the hardware compression? I only have a logical device which I use with the -f flag of tar program.
mt -f /dev/nst0 datcompression 2
I don't think that this is a nice idea always. AFAIK it may happen that you cannot read back the tape data on a different drive (maybe if it's newer or from a different vendor). Once we had a old DAT drive that were out of order. Other drives were not able to restore the backup tape. Thank god we had aother such old model...
I agree completely. Hardwarecompression is probably vendor specific, so potentially hasardous. Sorry I forgot to mention that, thanks. Then again, chances are you will not be able to read an old tape on a new device, period. With or without compression. Tapes tend to deteriorate quite fast anyhow, especially DAT. But I fear we're quite OT now... Maarten
Roman, If your backup are that critical (as they should be) have a look at arkeia (http://www.arkeia.com). It's included in the latest SuSE V7.1 distro for sure (dunno about the previous distro's). I think that for personal use licensing is free. Do full and incrementals (on many tapes) that way you can build a history and minimize the loss of your data. I think previous post's covered tar/gzip.bzip issues sufficiently... Regards James On Thu, 12 Apr 2001, RoMaN SoFt / LLFB!! wrote:
Hi all:
For backup purposes, I need to save big dirs of data to tape. The easy way: a gzipped tarball (eg: tar zcvf ..). Another method: tar .. ; bzip2 ...-> you get a tar.bz2 which has better compression ratio.
Now let's suppose I need to recover data from backup and the tape has some problems at any point. If I had saved separate files (not tarball) it's easy to recover most of the backup (only files on the broken point would be damaged and possibly couldn't be restored). But what does it happen when you have only a one big file (tarball, eg)????? Would you lose more data than the strictly affected by the tape failure? Would you lose all the data?
Have you got any comparative showing which methods are the more robust ones, etc?
I read that .bz2 is more "secure" (=robust) than .gz (maybe in tape howto).
Any ideas / advices are welcome. TIA.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------------------------------------------- To unsubscribe, e-mail: suse-security-unsubscribe@suse.com For additional commands, e-mail: suse-security-help@suse.com
participants (8)
-
engelbert.gruber＠ssg.co.at -
jleroux＠deiknumi.com -
Kurt Seifried -
Maarten van den Berg -
Mark Lutz -
Robert Casties -
RoMaN SoFt / LLFB!! -
Steffen Dettmer