[opensuse] Backup Question
Guys. I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory casee, but I am having problems finding a good example of a back up script. I am looking to make a cpio image of the whole box. Lucky we have a SAN with storage that will be able to hold the back up. So far I have come up with this, but I don't think it best. What I like to ask the group is does anyone have a nice back up script that can share with me, or can you expand on my script. One more question can you do exclude with cpio like tar? I am hoping this simple script can get what I want. ------------------Being Script---------------------------- #!/bin/sh # functions to call on remotedir () { mount /backup } uremotedit () { umount /backup } # Vars for the host=`hostname -f` # this is because the cron runs nightly at midnight we want the day before dat=`date +%m%d%G -d "1 day ago"` dat2=`date +%m-%d-%G` # The meat of the back up remotedir if [ -d == `/backup/ ] then find -d . -print | cpio -ova > /backup/$host.$dat.cpio else logger "System back up failed on $dat2, backup drive didn't mount" fi uremotedir ------------------End Script------------------------------ ---------------------------------------- Discover it! Enjoy it! Share it! openSUSE Linux. ----------------------------------------- openSUSE -- http://en.opensuse.org/User:Terrorpup openSUSE Ambassador openSUSE Member skype -- terrorpup twitter -- terrorpup friendfeed -- http://friendfeed.com/terrorpup Come join me at the Atlanta Linux Fest, September 19th, 2009 http://atlantalinuxfest.org/. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday September 10 2009, Chuck Payne wrote:
Guys.
I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory casee, but I am having problems finding a good example of a back up script. I am looking to make a cpio image of the whole box. Lucky we have a SAN with storage that will be able to hold the back up.
I don't think there's any reason to prefer cpio over tar. It's a less-used and more antiquated format. TAR also permits more compressions schemes (in fact, beyond the ones it will invoke itself based on options like 'j' or 'z', it can run arbitrary external compression commands if you wish). I recommend you stick with your tar-based scheme, if its suits you. Depending on your needs, an rsync approach might be a better choice. One thing I've taken to doing is augmenting my automated backup based on CrashPlan with a rotating (currently just every other day) rsync snapshots of critical directories. It's a lot easier to recover a lost or clobbered file from one of these than it is to poke around in the GUI for CrashPlan, but CrashPlan keeps history, is secure, compresses its backup archives, handles multiple backup destinations (including drives and remote systems), etc., so is still good to have as baseline backup mechanism.
...
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Randall R Schulz wrote:
TAR also permits more compressions schemes (in fact, beyond the ones it will invoke itself based on options like 'j' or 'z', it can run arbitrary external compression commands if you wish).
Or just feed the output from cpio through your favourite compressor. I find cpio very useful for some things, particularly when I have file names in a file or from stdin. The output can be made tar-compatible. /Per -- Per Jessen, Zürich (22.6°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen said the following on 09/10/2009 10:30 AM:
Randall R Schulz wrote:
TAR also permits more compressions schemes (in fact, beyond the ones it will invoke itself based on options like 'j' or 'z', it can run arbitrary external compression commands if you wish).
Or just feed the output from cpio through your favourite compressor.
I find cpio very useful for some things, particularly when I have file names in a file or from stdin. The output can be made tar-compatible.
Historically, CPIO could handle device i-nodes whereas TAR couldn't. Does this still hold? And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command? All to often I find that I want to backup _some_ of a directory tree, not all of it, such as the RCS files but not the originals. Using 'find' or the like is wonderful for this. But then again, my regualr backup is disk-to-disk-to-offline with the first part being done with 'rsync' or perhaps even a LVM copy. -- Always do your best. What you plant now, you will harvest later. - Og Mandino -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Anton Aylward wrote:
And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command?
I feel pretty certain it uses an external command - I'm sure I've seen the forked process in a 'ps axf' display.
All to often I find that I want to backup _some_ of a directory tree, not all of it, such as the RCS files but not the originals. Using 'find' or the like is wonderful for this.
+1. /Per -- Per Jessen, Zürich (22.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, Sep 10, 2009 at 11:01 AM, Per Jessen <per@opensuse.org> wrote:
Anton Aylward wrote:
And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command?
I feel pretty certain it uses an external command - I'm sure I've seen the forked process in a 'ps axf' display.
All to often I find that I want to backup _some_ of a directory tree, not all of it, such as the RCS files but not the originals. Using 'find' or the like is wonderful for this.
+1.
/Per
-- Per Jessen, Zürich (22.8°C)
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Would it be better to use tar with the cpio? find -d . -print0 | cpio --null -ova -H tar -F /backup/$host.$dat.tar If that better, then could I use bzip to make it smaller? -- ---------------------------------------- Discover it! Enjoy it! Share it! openSUSE Linux. ----------------------------------------- openSUSE -- http://en.opensuse.org/User:Terrorpup openSUSE Ambassador openSUSE Member skype -- terrorpup twitter -- terrorpup friendfeed -- http://friendfeed.com/terrorpup Come join me at the Atlanta Linux Fest, September 19th, 2009 http://atlantalinuxfest.org/. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Would it be better to use tar with the cpio? find -d . -print0 | cpio --null -ova -H tar -F /backup/$host.$dat.tar If that better, then could I use bzip to make it smaller?
bzip, for much higher CPU utilization, creates only very marginally smaller archives. I've found it is best just to stick with gzip. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Adam Tauno Williams wrote:
Would it be better to use tar with the cpio? find -d . -print0 | cpio --null -ova -H tar -F /backup/$host.$dat.tar If that better, then could I use bzip to make it smaller?
bzip, for much higher CPU utilization, creates only very marginally smaller archives. I've found it is best just to stick with gzip.
It does depend A LOT on what you're archiving. For logfiles, I started using lzma a while back - it's slow in compressing, but does it extremely well. For general backup archives probably less so. /Per -- Per Jessen, Zürich (22.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2009-09-10 at 17:01 +0200, Per Jessen wrote:
Anton Aylward wrote:
And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command?
I feel pretty certain it uses an external command - I'm sure I've seen the forked process in a 'ps axf' display.
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqpWn4ACgkQtTMYHG2NR9V11gCcCEH3B73R3XB+877Jvrrs68WU 2J0An1ZPt5dxj+wpTQzoHmgAKd4DywR0 =gDT/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, Sep 10, 2009 at 3:58 PM, Carlos E. R. <robin.listas@telefonica.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2009-09-10 at 17:01 +0200, Per Jessen wrote:
Anton Aylward wrote:
And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command?
I feel pretty certain it uses an external command - I'm sure I've seen the forked process in a 'ps axf' display.
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups.
- -- Cheers, Carlos E. R.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux)
iEYEARECAAYFAkqpWn4ACgkQtTMYHG2NR9V11gCcCEH3B73R3XB+877Jvrrs68WU 2J0An1ZPt5dxj+wpTQzoHmgAKd4DywR0 =gDT/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
This question might be a bit off target, is Amanda something i could use to back up these boxes, can it do images or does it need tape. I am currently read the "Unix System Administrator Handbook" because I read it had a cool chapter on backups. -- ---------------------------------------- Discover it! Enjoy it! Share it! openSUSE Linux. ----------------------------------------- openSUSE -- http://en.opensuse.org/User:Terrorpup openSUSE Ambassador openSUSE Member skype -- terrorpup twitter -- terrorpup friendfeed -- http://friendfeed.com/terrorpup Come join me at the Atlanta Linux Fest, September 19th, 2009 http://atlantalinuxfest.org/. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2009-09-10 at 17:01 +0200, Per Jessen wrote:
Anton Aylward wrote:
And as for the compression, is the code or that actually in the TAR binary or does it pipe through an external command?
I feel pretty certain it uses an external command - I'm sure I've seen the forked process in a 'ps axf' display.
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups.
I think the way to look at that is: a) if the compression fails, the tar'ing will fail, therefore your backup will fail. b) if the compression fails, you've got other things to worry about than a missing backup. /Per -- Per Jessen, Zürich (14.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2009-09-11 at 07:43 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups.
I think the way to look at that is:
a) if the compression fails, the tar'ing will fail, therefore your backup will fail. b) if the compression fails, you've got other things to worry about than a missing backup.
It can fail silently, so that the user doesn't notice unless he tries a test recover as part of the backup procedure. Or, the backup media can fail (read error) several months later when you need to do a recovery. Even if that error only affects a few bytes, you can loose the entire tar. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqtVPEACgkQtTMYHG2NR9V3zwCfWJVsOW/f1rWlq3tEzmtCPilh C3oAnjVTSHv/fwDEUhKOLKnfaldrbxfu =Erq6 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
On Friday, 2009-09-11 at 07:43 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups.
I think the way to look at that is:
a) if the compression fails, the tar'ing will fail, therefore your backup will fail. b) if the compression fails, you've got other things to worry about than a missing backup.
It can fail silently, so that the user doesn't notice unless he tries a test recover as part of the backup procedure.
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem.
Or, the backup media can fail (read error) several months later when you need to do a recovery. Even if that error only affects a few bytes, you can loose the entire tar.
It's really just another variation of (b) - the backup media or the physical storage location is not safe/trustworthy. /Per -- Per Jessen, Zürich (11.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Carlos E. R. wrote:
On Friday, 2009-09-11 at 07:43 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Which means that if the compression/decompression fails, it may corrupt the entire backup. I don't trust compressed tars for backups.
I think the way to look at that is:
a) if the compression fails, the tar'ing will fail, therefore your backup will fail. b) if the compression fails, you've got other things to worry about than a missing backup.
It can fail silently, so that the user doesn't notice unless he tries a test recover as part of the backup procedure.
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem.
Or, the backup media can fail (read error) several months later when you need to do a recovery. Even if that error only affects a few bytes, you can loose the entire tar.
It's really just another variation of (b) - the backup media or the physical storage location is not safe/trustworthy.
/Per
Actually that last point is valid. You should never ever actually trust a tape. Buy good tapes, buy good tape drives, keep the tape drive cleaned and don't mistreat the tapes, use bit-level verification** during backus, And despite all that _still_ never actually depend on any single tape. Because the fact is there is no such thing as any mechanical magnetic media that is really dependable, and tape is among the worst, even though there is nothing better. You have to make many tapes and constantly monitor the backup/verify summary reports, and cycle out old tapes before they start failing, and still you need to just be prepared that when you need to restore, maybe the most recent tape won't be good and you'll be restoring from a day or more earlier. And if you have incrimental tapes and the most recent master is bad, maybe you'll be restoing from weeks in the past. That's just the way it is. Given that, if your backup/restore program can't skip over errors in media where the data is compressed, then it's exceedingly valid to avoid using compression, because it could be the difference between a 2 week old restore, or a 1 hour old restore with a single random file somewhere on the system that is either missing or has a few bytes of corruption that can probably be dealt with easily and even save that whole file minus one database record or something. However, good backups software _can_ skip over media errors even with compression. You lose a little more data since you lose the whole block that had the error, which may contain several small files or a good size chunk of a single larger file, and you may lose that entire larger file instead of a piece of it. Also, tape drives have compression in hardware that's turned on by default anyways, so you don't gain much by doing your own software compression. If you've got the cpu to burn it can do a better job by being smarter about only bothering to compress stuff that will compress and not bothering to try to compress zip tgz bz2 png jpg etc etc etc files. All that said, I use compression. It would be wildly impractical not to. Use raid, to reduce your chances of even needing to ever restore from a backup in the first place, use multiple backups like another copy maintained by rsync on some remote server to again reduce your chances of ever needing to restore from a tape, use good hardware & software to reduce your chances of having a problem the day you do have to restore from tape. And go ahead and use compression so that you can actually do full backups every night and so that you can even _affford_ a tape drive and data channel subsystem that handle the job in a reasonable amount of time each night. I think it's silly and causes you more pain and loss and risk than what you are attempting to avoid by not using compression, but, the point about media errors IS valid even with the best technology available, and the point about the special anti-synergy of media errors in concert with compression, IS valid. **(means the backup program actually reads the entire tape back after writing and compares it to the disk, because you physically can't know that the tape media accepted the write until you actually read it back.) -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
Per Jessen wrote:
Or, the backup media can fail (read error) several months later when you need to do a recovery. Even if that error only affects a few bytes, you can loose the entire tar.
It's really just another variation of (b) - the backup media or the physical storage location is not safe/trustworthy.
/Per
Actually that last point is valid. You should never ever actually trust a tape. Buy good tapes, buy good tape drives, keep the tape drive cleaned and don't mistreat the tapes, use bit-level verification** during backus, And despite all that _still_ never actually depend on any single tape. Because the fact is there is no such thing as any mechanical magnetic media that is really dependable, and tape is among the worst, even though there is nothing better.
AFAIR, IBM 3480 and STK 9840 tapes come with a 25-year data retrieval warranty. For longterm storage, I would still advocate keeping two copies, but that would be more to guard against other accidents. /Per -- Per Jessen, Zürich (14.5°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Actually that last point is valid. You should never ever actually trust a tape. Buy good tapes, buy good tape drives, keep the tape drive cleaned and don't mistreat the tapes, use bit-level verification** during backus, And despite all that _still_ never actually depend on any single tape. Because the fact is there is no such thing as any mechanical magnetic media that is really dependable, and tape is among the worst, even though there is nothing better. AFAIR, IBM 3480 and STK 9840 tapes come with a 25-year data retrieval warranty. For longterm storage, I would still advocate keeping two copies, but that would be more to guard against other accidents.
I hear the claim against tape all the time: "tape is among the worst" and I always call "BOGUS!". Tape is an amazingly durable archive media. I've been in charge of a vault full of tapes - failures are very rare and frequently recoverable. People having problems with tape simply haven't been maintaining their archive correctly (cycling out old tapes, maintaining the tape drives, etc...). I've read from tapes that looked like they had been to hell and back - without fail. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Adam Tauno Williams wrote:
I hear the claim against tape all the time: "tape is among the worst" and I always call "BOGUS!". Tape is an amazingly durable archive media. I've been in charge of a vault full of tapes - failures are very rare and frequently recoverable. People having problems with tape simply haven't been maintaining their archive correctly (cycling out old tapes, maintaining the tape drives, etc...). I've read from tapes that looked like they had been to hell and back - without fail.
Quite so. Many years ago, when I was a computer tech, the operators had a machine just for testing tape integrity. If the tape didn't meet specs, it was tossed. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, 2009-09-15 at 10:30 -0400, James Knott wrote:
Adam Tauno Williams wrote:
I hear the claim against tape all the time: "tape is among the worst" and I always call "BOGUS!". Tape is an amazingly durable archive media. I've been in charge of a vault full of tapes - failures are very rare and frequently recoverable. People having problems with tape simply haven't been maintaining their archive correctly (cycling out old tapes, maintaining the tape drives, etc...). I've read from tapes that looked like they had been to hell and back - without fail. Quite so. Many years ago, when I was a computer tech, the operators had a machine just for testing tape integrity. If the tape didn't meet specs, it was tossed.
This is especially true when compared to the failure rates seen in optical meda - which is flat-out a *terrible* archive format. CD and DVD is dramatically shorter than most expectations. The storage conditions recommended for optical media [constant 10°C - 15°C and 20% - 50% relative humidity] is much narrower than for tape; and all burned optical media is extremely sensitive to UV radiation. I couldn't even begin to list the number of times I've been at customer sites where data from CD/DVD was not recoverable - fortunately most are smart enough to keep multiple [as in 3 or more] copies of all data archived to optical. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Adam Tauno Williams wrote:
Actually that last point is valid. You should never ever actually trust a tape. Buy good tapes, buy good tape drives, keep the tape drive cleaned and don't mistreat the tapes, use bit-level verification** during backus, And despite all that _still_ never actually depend on any single tape. Because the fact is there is no such thing as any mechanical magnetic media that is really dependable, and tape is among the worst, even though there is nothing better.
AFAIR, IBM 3480 and STK 9840 tapes come with a 25-year data retrieval warranty. For longterm storage, I would still advocate keeping two copies, but that would be more to guard against other accidents.
I hear the claim against tape all the time: "tape is among the worst" and I always call "BOGUS!". Tape is an amazingly durable archive media. I've been in charge of a vault full of tapes - failures are very rare and frequently recoverable. People having problems with tape simply haven't been maintaining their archive correctly (cycling out old tapes, maintaining the tape drives, etc...). I've read from tapes that looked like they had been to hell and back - without fail.
I did say there is also nothing really better, and I did not say that I've had "problems" with tapes. I have had to deal with bad tapes and bad drives but it's a simple fact of life that the tape drive is the first thing to go in just about every standalone back room server. The dust kills it , and if not that then the simple fact of it being the most moving-part. Even when the on-site people are really good about actually using the cleaning tape. Even when the particular site isn't all that dusty. It's simply by far the weakest, most delicate and easily broke and first to wear out piece of equipment in the server. Something has to be, no matter hoiw much you spend, something has to be the weakest part compared to the rest. Next comes the fans and then the power supplies usually. Doesn't matter how much you want to spend on higher quality hardware. The highest quality server with the highest quality tape driver and fans, the tape drive or fans are still the first part in that server to go. The other fact is, tapes get kicked around in cars, filled with dust or dirt/sand, exposed to moisture, exposed to heat. Hard drives are bolted inside of the machine and aren't moved or dunked in water and any dirt or dust sits harmlessly on the exterior of the hard drive case not inside where the moving parts and delicate media surfaces are. The data that travels through a network card or that sit in ram doesn't really have any mechanical problems to worry about. Tapes suffer all of this better than most other possibilities, and it's why they are the the best choice in many cases. However they have their points, and then they have their points. You can drop a tape without destroying it, not so much with say a removable hard drive. You can dunk a dvd in sea water for a month without harming it, not so much with a tape! You can read a wad of data off of a backup servers raid array, or write to it, hundreds of thousands of times without degrading or wearing out the media or the drive. With a tape you get _hundreds_ of cycles if you're lucky. That is all I mean by tapes being among the worst forms of data media, yet also there being nothing better. Think of it in terms of the chances of a given bit of data suviving a transfer onto and off of some given media. The most reliable tape system in the universe can't compare with any ordinary hard drive or stick of ram or network card. Those other things handle millions of transactions absolutely bit-level flawlessly, every _day_. If you bought a hundred tapes probably 5 would be bad or go bad withing a few cycles. Of the good remaining good ones, the best one probably couldn't survive anywhere near 500 read/write cycles. If you want to take the concept of a tape system as a whole, including the process of cycling out old tapes and old tape drives, well, gee, that only works by dint of a human manuall, actively replacing parts continuously, being knowledgeable and always doing the right thing. That's kind of a ridiculous amount of overhead compared to a hard drive that can read and write a zillion times perfectly all by itself for free for some number of years. And even WITH that human maintenance staff, you still can't claim anywhere near the same level of reliability, because simply that much time hasn't yet passed since the first tape drive was ever invented, to match the number of read/write transactions, even if they were all perfect, which they weren't, to compare with what a cheap ide hard drive does in probably one year. What's better? Depends on your requirements. Every object could be said to be the best possible way to do the exact combination of things that object does. So, that's kind of meaningless. But, for example, If I have a server with live data, and I have a tape drive a and a tape that are known good so-far, and a neighboring server on the network with a raid array and network that are known good so-far. And you tell me: "pick one, write a tape using the backup software of your choice, or copy the data to the neighboring server using the software of your choice, and the if we read back that copy and there is any bit wrong, you will die." I will rsync to the neighboring server without blinking. No contest. But if you have to carry a couple terabytes on a hiking trip through the jungle or the desert or antarctica, without wired or radio transmission. Or place it into a lock box in a vault in a bank, or hide it on your person, then I would choose two copies on tapes over any sort of removable hard drives or current tech optical media. Flash drives are still a question mark to me. I'd take a tape over typical usb thumb drive any day, on the basis of construction quality. But, It's perfectly possible to construct an ssd drive that's embedded in epoxy potting and encased in a sturdy shell that could take far more abuse than any tape, but I don't know how possible it is to protect against static or EM , even with optosolators. Actually given the choice I'd probably do all three. One copy on tape, on copy on blueray, one copy on ssd. They each survive and don't survive different things. If I had to administer a lot of data on a lot of servers scattered around the country in remote colo facilities, and I had enough network connectivity and enough copies of server hardware and enough hard drive space, I'd maintain multiple copies of the data on multiple servers at multiple remote locations from each other with rsync, and employ rolling tar scripts or filesystems that have snapshot capabilities. Providing me with not only multiple copies of data, not only with those copies all being off-site and indeed separate from each other as well, not only with history, but also with ready-to-go, already up & running and configured backup servers to actually run this backup data, and backup network connections to reach them, indeed whole backup hosting facilities ie, power, a/c, etc and, all this without needing any staff to insert and remove tapes nor exotic expensive and rack-space-eating tape libraries. Most colo facilities do offer tape backup as a service so sure, you could pay a service fee and not have to worry about staff, tapes, worn out drives, or rack space, but, my way works equally well in colo facilities or in customer sites or in my own office or indeed basement, and it's all much more immediately available, provides more functionality and flexibility and faster response times and on top of it all, costs less. Tapes have their place, but they are not the last word in data security, disaster prevention, or disaster recovery. -- blw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 15 September 2009 18:31:43 Brian K. White wrote:
SNIP
That's a very interesting post - as someone who does it all with 2-disk raid array, another disk to protect against rm -rf and hardware replacements every 3 years or so. I'd like it if we could crank up the quality on this list with more such posts, and all show a bit more tolerance to and empathy with users of $thatotherdesktop, cranks, n00bs, and the rtfm-dyslexic while we're at it.
Actually given the choice I'd probably do all three. One copy on tape, on copy on blueray, one copy on ssd. They each survive and don't survive different things.
This sounds a lot like a techie version of the old englishman, irishman, scotsman joke genre. Anyone with more wit than me want to deliver a punchline? Oh and while I'm here, remember the openSUSE conference starts in 2 days' time. Keep an eye on the lists and the websites for what's going on and slides/summaries/videos of the goings-on, and if you're not able to attend, tell us what would make it more relevant and useful for you. Will -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Will Stephenson wrote:
On Tuesday 15 September 2009 18:31:43 Brian K. White wrote:
SNIP
That's a very interesting post - as someone who does it all with 2-disk raid array, another disk to protect against rm -rf and hardware replacements every 3 years or so.
I'd like it if we could crank up the quality on this list with more such posts, and all show a bit more tolerance to and empathy with users of $thatotherdesktop, cranks, n00bs, and the rtfm-dyslexic while we're at it.
It's difficult to maintain any particular quality on a list with such a wide scope of users and situations. To get a desired quality, we'd have to narrow the scope. As for tolerance and empathy, this list is doing quite well I think. I can certainly think of a few others where terms such as RTFM are used a lot more. /Per -- Per Jessen, Zürich (13.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
I have had to deal with bad tapes and bad drives but it's a simple fact of life that the tape drive is the first thing to go in just about every standalone back room server. The dust kills it , and if not that then the simple fact of it being the most moving-part. Even when the on-site people are really good about actually using the cleaning tape. Even when the particular site isn't all that dusty. It's simply by far the weakest, most delicate and easily broke and first to wear out piece of equipment in the server.
I agree with all of the above, but a server sat in a broom cupboard somewhere doesn't quite fit in my operating strategy :-) Reading the rest of your posting, I think we are looking at two different ends of the scale wrt tape usage. Like others, I have spent quite a few years of my professional life working with tape. Most recently a few years at StorageTek writing system software for tape libraries/robots.
The most reliable tape system in the universe can't compare with any ordinary hard drive or stick of ram or network card.
Quite the contrary. The most reliable tape system in the universe will easily outlast any ordinary hard drive. Even the r/w head of an STK drive has a guaranteed lifetime of 8.5 years. An ordinary harddrive will die after five years, on average.
If you want to take the concept of a tape system as a whole, including the process of cycling out old tapes and old tape drives, well, gee, that only works by dint of a human manuall, actively replacing parts continuously, being knowledgeable and always doing the right thing. That's kind of a ridiculous amount of overhead compared to a hard drive that can read and write a zillion times perfectly all by itself for free for some number of years.
You might be right, but when we're talking about tape, it's because we're talking about reliable storage, not read/write operations. Brian, take any ordinary harddrive today and write a file to it. I'll do the same to a 9840 tape. Then we store them in a dry place at 10-20C for 10 years. I'll bet I can still retrieve the file, and I'll also bet that you can't. (I've already tried both). /Per -- Per Jessen, Zürich (13.0°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
The most reliable tape system in the universe can't compare with any ordinary hard drive or stick of ram or network card. Quite the contrary. The most reliable tape system in the universe will easily outlast any ordinary hard drive. Even the r/w head of an STK drive has a guaranteed lifetime of 8.5 years. An ordinary harddrive will die after five years, on average.
Agree.
If you want to take the concept of a tape system as a whole, including the process of cycling out old tapes and old tape drives, well, gee, that only works by dint of a human manuall, actively replacing parts continuously, being knowledgeable and always doing the right thing. That's kind of a ridiculous amount of overhead compared to a hard drive that can read and write a zillion times perfectly all by itself for free for some number of years. You might be right, but when we're talking about tape, it's because we're talking about reliable storage, not read/write operations. Brian, take any ordinary harddrive today and write a file to it. I'll do the same to a 9840 tape. Then we store them in a dry place at 10-20C for 10 years. I'll bet I can still retrieve the file, and I'll also bet that you can't. (I've already tried both).
Agree, my money is on the tape. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Brian K. White wrote:
I have had to deal with bad tapes and bad drives but it's a simple fact of life that the tape drive is the first thing to go in just about every standalone back room server. The dust kills it , and if not that then the simple fact of it being the most moving-part. Even when the on-site people are really good about actually using the cleaning tape. Even when the particular site isn't all that dusty. It's simply by far the weakest, most delicate and easily broke and first to wear out piece of equipment in the server.
I agree with all of the above, but a server sat in a broom cupboard somewhere doesn't quite fit in my operating strategy :-)
Reading the rest of your posting, I think we are looking at two different ends of the scale wrt tape usage. Like others, I have spent quite a few years of my professional life working with tape. Most recently a few years at StorageTek writing system software for tape libraries/robots.
The most reliable tape system in the universe can't compare with any ordinary hard drive or stick of ram or network card.
Quite the contrary. The most reliable tape system in the universe will easily outlast any ordinary hard drive. Even the r/w head of an STK drive has a guaranteed lifetime of 8.5 years. An ordinary harddrive will die after five years, on average.
If you want to take the concept of a tape system as a whole, including the process of cycling out old tapes and old tape drives, well, gee, that only works by dint of a human manuall, actively replacing parts continuously, being knowledgeable and always doing the right thing. That's kind of a ridiculous amount of overhead compared to a hard drive that can read and write a zillion times perfectly all by itself for free for some number of years.
You might be right, but when we're talking about tape, it's because we're talking about reliable storage, not read/write operations.
Brian, take any ordinary harddrive today and write a file to it. I'll do the same to a 9840 tape. Then we store them in a dry place at 10-20C for 10 years. I'll bet I can still retrieve the file, and I'll also bet that you can't. (I've already tried both).
/Per
How many write-read cycles does the best tape in the world accomplish before it fails? How many write-read cycles does even an ordinary hard drive accomplish before it fails? That is what I mean. It is simply one metric among many. The fact that a tape may be better by some other metric doesn't change this one. The chances of any given write and then read operation completing successfully are astronomically higher for any of the fast things like ram and hard drives, than any of the slow things like tape, simply because the hard drive actually has been proven by writing and reading a million transactions. No tape can claim that. You write to a tape infinitesimally fewer times. Even if the success rate is 100%, the sample size is still so ridiculously smaller that you can not call a tape which successfully writes/reads 300 times in it's entire life time, anywhere near as reliable as a device which successfully writes/reads 300 times _per second_, for years. You are presenting an entirely different, also valid metric. A very good tape probably has a greater chance of reading without any errors after sitting around for 20 years than a hard drive. Greater but still nowhere near great. Even 99% chance is not great at all compared to what a hard drive does every second. And really, who knows, maybe that tape and the hard drive have exactly the same chance after sitting around 20 years. Any hard drives that are currently 20 years old are nothing like the ones today, and why should a hard drive degrade any more in 20 years sitting unpowered on a climate controlled shelf than a tape and tape drive? What? I shouldn't count the drive? In 20 years they will be manufacturing and selling new LTO or any other drives that can read any of the current types of tapes? Even backwards compatibility has not gone that far so far so I don't know how you can bank on that happening more in the future. Where is the current tape drive made today that can read any tape that was made 20 years ago? Even exotic high end situations where its not unreasonable to have such drive custom manufactured just for you, doesn't always get what it needs in time to save the data. What about all that priceless NASA sattelite and remote probe research data thats disintegrating _as we speak_ on tapes? I said essentially "don't trust a tape" Someone attempted to counter that with essentially "that's bogus, the entire process of using tapes is perfectly reliable" The second statement has no bearing at all on the first and in no way contradicts it. In fact the process of using tapes has at it's core the very precept of trusting any individual tape as little as possible. You make many copies, you read each copy back as soon as you write it, you aggressively cycle out tapes while they are still practically brand new in that they've only been used a handful of times (50 to 200). Those are not policies that reflect a high trust in the base media. You don't buy a new hard drive every 200 writes, or even every 200 days with countless writes per day. You could also say that monks hand transcribing manuscripts on vellum by candle light are more reliable than tape or hard drives or anything else, and no one can prove otherwise. Because monks and cross-verified manuscripts have already existed for thousands of years and tapes have not. Fate accomplis. But of course the human and the vellum are not reliable. Only the entire process of many humans checking each others work, and many copies of the data to compare against each other, is reliable. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
How many write-read cycles does the best tape in the world accomplish before it fails? How many write-read cycles does even an ordinary hard drive accomplish before it fails?
That is what I mean. It is simply one metric among many.
Sure - we could also compare the physical size of the two, but it would be an equally inappropriate metric, IMHO.
The fact that a tape may be better by some other metric doesn't change this one.
True.
You are presenting an entirely different, also valid metric. A very good tape probably has a greater chance of reading without any errors after sitting around for 20 years than a hard drive. Greater but still nowhere near great. Even 99% chance is not great at all compared to what a hard drive does every second.
And really, who knows, maybe that tape and the hard drive have exactly the same chance after sitting around 20 years.
_I_ know. The tape wins hands down. There are many industries where documents and data must be kept for 10/20/25 years, sometimes even longer. The latter are usually microfilmed (or todays equivalent), the former are stored on tape.
Any hard drives that are currently 20 years old are nothing like the ones today,
It doesn't matter, but they're actually surprisingly similar. Spinning platters of metal with a magnetic coating.
and why should a hard drive degrade any more in 20 years sitting unpowered on a climate controlled shelf than a tape and tape drive?
Because the harddrive is full of electronics which has been designed for a very specific MTBF. For instance, it will have SMD electrolytic capacitors that, depending on the materials used, _will_ break whether they're powered on or not. Harddrives were never designed to last 25 years, whereas tapes were.
What? I shouldn't count the drive? In 20 years they will be manufacturing and selling new LTO or any other drives that can read any of the current types of tapes?
Even backwards compatibility has not gone that far so far so I don't know how you can bank on that happening more in the future.
Where is the current tape drive made today that can read any tape that was made 20 years ago?
http://www.sun.com/storage/tape_storage/tape_drives/9840/ (not _any_ tape, but those with which the drive is comptabile of course). /Per -- Per Jessen, Zürich (16.3°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Sep 16, 2009 at 12:30 PM, Per Jessen <per@opensuse.org> wrote:
Brian K. White wrote:
How many write-read cycles does the best tape in the world accomplish before it fails? How many write-read cycles does even an ordinary hard drive accomplish before it fails?
That is what I mean. It is simply one metric among many.
Sure - we could also compare the physical size of the two, but it would be an equally inappropriate metric, IMHO.
The fact that a tape may be better by some other metric doesn't change this one.
True.
You are presenting an entirely different, also valid metric. A very good tape probably has a greater chance of reading without any errors after sitting around for 20 years than a hard drive. Greater but still nowhere near great. Even 99% chance is not great at all compared to what a hard drive does every second.
And really, who knows, maybe that tape and the hard drive have exactly the same chance after sitting around 20 years.
_I_ know. The tape wins hands down. There are many industries where documents and data must be kept for 10/20/25 years, sometimes even longer. The latter are usually microfilmed (or todays equivalent), the former are stored on tape.
Tape is far more reliable than disk if both are sitting on a shelf. As you say it was designed for it. Disk was not. Most larger corporations that are moving to disk based backup don't put disks on a shelf (like HOSO users often do). They maintain them with power on in a raid setup. Then they routinely scan them for failed drives and replace the drives as they fail. In my opinion, that drastically raises the cost of keeping backups on disk and I am still a big fan of tape based backups, but a well maintained raid array should be equally reliable IMO. fyi: at our office, we actually do keep a lot of data sitting on powered off disks, but we also make tape backups we send offsite of the same data. We've had about 10 hard drives fail in the last year. ie. when we pulled the drives off the shelf and tried to use them, they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3. Greg -- Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer Preservation and Forensic processing of Exchange Repositories White Paper - <http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html> The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
_I_ know. The tape wins hands down. There are many industries where documents and data must be kept for 10/20/25 years, sometimes even longer. The latter are usually microfilmed (or todays equivalent), the former are stored on tape. Tape is far more reliable than disk if both are sitting on a shelf. As you say it was designed for it. Disk was not. Most larger corporations that are moving to disk based backup don't put disks on a shelf (like HOSO users often do). They maintain them with power on in a raid setup. Then they routinely scan them for failed drives and replace the drives as they fail.
Absolutely true - this is exactly what we do: disk to disk to tape. And the disk backup is a rolling archive, it contains snapshots of the last 120 days. Tapes cycle to off-site and allow us to go back to quarters, annuals, etc... [which isn't optional, you are pretty much legally obligated to have that].
In my opinion, that drastically raises the cost of keeping backups on disk and I am still a big fan of tape based backups, but a well maintained raid array should be equally reliable IMO.
Disagree with the reduction in the "cost of keeping backups" [if we are just comparing technologies]. Online archival is expensive - you are constantly paying for power and cooling. And power required by the number of drives required for an adequate archival system is not trivial. It is however worth the convenience as it reduces the *labor* of retrieval from archive - and labor is always the most expensive component of any system.
fyi: at our office, we actually do keep a lot of data sitting on powered off disks, but we also make tape backups we send offsite of the same data. We've had about 10 hard drives fail in the last year. ie. when we pulled the drives off the shelf and tried to use them,
Our experiments had the same result: archival to cold disks is a terrible strategy.
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3.
Ditto, again. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2009-09-16 at 13:34 -0400, Adam Tauno Williams wrote:
On Wednesday, 2009-09-16 at 12:49 -0400, Greg Freemyer wrote:
Our experiments had the same result: archival to cold disks is a terrible strategy.
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3.
Ditto, again.
Are there tape systems that may be used for, say, 30 years, by home users or very small sites or professionals? I mean, not data centers with big purses. What can we use? - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqxRosACgkQtTMYHG2NR9U4xQCgjH6PzSmh5MDQ+zSq1VYKd4uv bHMAn3QgHaMspDz0hepPXPcGvnybn8gJ =Eov5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
On Wednesday, 2009-09-16 at 13:34 -0400, Adam Tauno Williams wrote:
On Wednesday, 2009-09-16 at 12:49 -0400, Greg Freemyer wrote:
Our experiments had the same result: archival to cold disks is a terrible strategy.
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3.
Ditto, again.
Are there tape systems that may be used for, say, 30 years, by home users or very small sites or professionals? I mean, not data centers with big purses. What can we use?
I doubt if you'll find anything. For one thing, the demand for long term storage doesn't really exist amongst home users or one-man shops. Yes, I'm sure lots of people would like to keep their precious digital photos and videos that long, but that desire doesn't itself create a demand. Not yet anyway. The other thing is that the manufacturers who supply the consumer/SMB market have to keep supplying, and therefore intentionally make things break or improve the technology, but omitting backward compatibility. /Per -- Per Jessen, Zürich (14.7°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Carlos E. R. wrote:
On Wednesday, 2009-09-16 at 13:34 -0400, Adam Tauno Williams wrote:
On Wednesday, 2009-09-16 at 12:49 -0400, Greg Freemyer wrote:
Our experiments had the same result: archival to cold disks is a terrible strategy.
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3.
Ditto, again.
Are there tape systems that may be used for, say, 30 years, by home users or very small sites or professionals? I mean, not data centers with big purses. What can we use?
I doubt if you'll find anything. For one thing, the demand for long term storage doesn't really exist amongst home users or one-man shops. Yes, I'm sure lots of people would like to keep their precious digital photos and videos that long, but that desire doesn't itself create a demand. Not yet anyway. The other thing is that the manufacturers who supply the consumer/SMB market have to keep supplying, and therefore intentionally make things break or improve the technology, but omitting backward compatibility.
/Per
Some common formats that are generally backwards compatible, the latest version no longer reads the earliest version, and several that were quite popular and seemingly set for life at the time, have simply gone from the earth with the single company that made them. That's one thing I hope LTO helps. Most (all?) other tape formats are owned by some manufacturer, and no other manufacturer may make the drives without licensing and permission from the original producer. Where lto is intentionally open design that any manufacturer may produce drives at any time. 30 years later, it won't be in the hands of any one company, if they even still exist, to decide if a drive that can read LTO-1 media is made. Demand for a drive that can read some really old tape may drop so low that the original company can't be bothered, but there is usually some other company willing to fill any demand as long as it's legally and financially possible. Although I wish they'd come out with a dat or vxa sized format that's open too. lto is fine for the data center but kind of a lot to ask the owner of a small business to put in his pocket every night and every morning. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2009-09-17 at 17:29 -0400, Brian K. White wrote:
Per Jessen wrote:
Carlos E. R. wrote:
On Wednesday, 2009-09-16 at 13:34 -0400, Adam Tauno Williams wrote:
On Wednesday, 2009-09-16 at 12:49 -0400, Greg Freemyer wrote:
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3.
Ditto, again.
Are there tape systems that may be used for, say, 30 years, by home users or very small sites or professionals? I mean, not data centers with big purses. What can we use?
I doubt if you'll find anything. For one thing, the demand for long term storage doesn't really exist amongst home users or one-man shops. Yes, I'm sure lots of people would like to keep their precious digital photos and videos that long, but that desire doesn't itself create a demand. Not yet anyway. The other thing is that the manufacturers who supply the consumer/SMB market have to keep supplying, and therefore intentionally make things break or improve the technology, but omitting backward compatibility.
Some common formats that are generally backwards compatible, the latest version no longer reads the earliest version, and several that were quite popular and seemingly set for life at the time, have simply gone from the earth with the single company that made them.
That's one thing I hope LTO helps.
Most (all?) other tape formats are owned by some manufacturer, and no other manufacturer may make the drives without licensing and permission from the original producer.
Where lto is intentionally open design that any manufacturer may produce drives at any time.
Since this conversation I had a look on LTO drives, and.., they are quite expensive: 800..1200€... that's a lot for me (home). The tape is relatively cheap, though. Well, about half the price of a HD that size (and reasonable security requires two copies). <http://www.alternate.es/html/categoryListing.html?cat1=026&cat2=075&cat3=000&&tn=HARDWARE&l1=Dispositivos&l2=Copia+seguridad%2C+backup&l3=LTO&>
30 years later, it won't be in the hands of any one company, if they even still exist, to decide if a drive that can read LTO-1 media is made. Demand for a drive that can read some really old tape may drop so low that the original company can't be bothered, but there is usually some other company willing to fill any demand as long as it's legally and financially possible.
I heard that NASA has hard problems reading their own old tapes from old "deep space" missions. Not that old, considering what they study. One of the problems was getting drives to read them, or even computers for those drives. It is a real problem. Currently, the typical/practical solution is to store the data somewhere else before the media is too obsolete. Similar to old librarians with papyrus. Progress.
Although I wish they'd come out with a dat or vxa sized format that's open too. lto is fine for the data center but kind of a lot to ask the owner of a small business to put in his pocket every night and every morning.
- -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkrD4soACgkQtTMYHG2NR9VGDwCffWyLTo24YTSaEa8rt49hn6Nr DEcAnjALeUaZwIc+GO0ZFKK0+BwC+iPT =kaVq -----END PGP SIGNATURE-----
Carlos E. R. wrote:
Since this conversation I had a look on LTO drives, and.., they are quite expensive: 800..1200€... that's a lot for me (home).
Backing up to tape, whether or not for long-term storage is more of a business requirement, I think. The cost of tape drives+media must be seen in relation to the value of the benefits they provide, which in a business setting will always be higher than in a home setting. /Per -- Per Jessen, Zürich (13.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
they would not work reliably. None of the tapes we have retrieved from offsite have failed. Our offsite tapes are mostly a combination of LTO-1 and LTO-3. Ditto, again. Are there tape systems that may be used for, say, 30 years, by home users or very small sites or professionals? I mean, not data centers with big
Our experiments had the same result: archival to cold disks is a terrible strategy purses. What can we use?
I've been using DLT tape at home for a long time. Drives can be got pretty cheap on E-Bay and the media is reasonably priced. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2009-09-14 at 08:28 +0200, Per Jessen wrote:
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem.
Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago. I still have backups in 360 KB floppies that work... it had compression and low level error recovery codes. Not portable, of course. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkquyC0ACgkQtTMYHG2NR9UMFQCeONxSh/TeKkFWFjVdCEsbGhnb 55sAn1CBsNal7sdRPdaGtxzoURKkzXHU =3x+R -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2009-09-14 at 08:28 +0200, Per Jessen wrote:
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem.
Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago.
Okay, that's an interesting point - I don't think I have heard anyone complain about tars reliability before. What do you see as a more reliable tool/utility then? /Per -- Per Jessen, Zürich (9.9°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem. Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago. Okay, that's an interesting point - I don't think I have heard anyone complain about tars reliability before. What do you see as a more reliable tool/utility then?
I disagree with the claim that tar is "unreliable". I've been a UNIX admin for 16 years and I can easily count the number of failed tar archives I've encountered. I think the issue is that tar is not "trusted" - a 0 exit status doesn't mean your archive is any good. Tar also won't tell you if an archive has been modified since it was created [another kind of "untrusted"]. I'm not aware of any "trusted" (verified + secured) Open Source backup solutions - but I take that as a vote for the "reliability" of tar. Clearly it is sufficiently "reliable" for the massive number of people using it. I anyone knows of one [that isn't a wrapper around tar! :)] I'd be very interested. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen pecked at the keyboard and wrote:
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2009-09-14 at 08:28 +0200, Per Jessen wrote:
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem. Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago.
Okay, that's an interesting point - I don't think I have heard anyone complain about tars reliability before. What do you see as a more reliable tool/utility then?
/Per
I used Microlite's BackupEdge and never had a problem with restores even to bare metal. It also supports DVD media. -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2009-09-14 at 08:28 +0200, Per Jessen wrote:
That is only a variation of (b) above - if you can't trust your backup procedure, a missing backup not your primary problem.
Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago.
Okay, that's an interesting point - I don't think I have heard anyone complain about tars reliability before. What do you see as a more reliable tool/utility then?
It's not that tar has an unsuspected achilles heel or weakness.** Tar is fine. It's just that tar lacks any extra strength. Tar is fine as long as the underlying media is perfect. The comparison is, the iso9660 and udf filesystems contain extra data similar to the mnp5 error correcting stream that modems use, or the zmodem & kermit filetransfer protocols. (I would say also like raid5 but the striping across multiple disks part of raid5 doesn't apply and may confuse the issue) They can suffer some amount of media / communication errors without losing the payload data, and if there is too much media/trasmission loss to recreate the data from the ECC data, they at least detect and know which bits are garbage instead of delivering them as good, and recover and resume as soon as the media/transmission clears up. Because, CD's, DVD's, telephone lines, even local serial lines, are not perfect media by a long shot. In fact all the low-level protocols for networking and media storage have some form of error detection and correction built-in, so that you the user or higher-level protocol do not have to worry about it. But, backups and tars are a special case. Sometimes you treat a tar or a compressed tar as a simple file in a filesystem. In that case, your "media" is perfect. If it's not, as was said, you have bigger problems. But really the media isn't in fact perfect. Merely the disk has automatic bad block detection and remapping, and the raid controller or software raid maintains extra data to detect and transparently correct media errors so that you never see them at the application level. Similarly, tcp/ip garantees that you at the application level never see any of the garbage that's actually on the wire. the network cards and the tcp/ip stack all do whatever amount of checksumming and retrying is required transparently and present you with a magically perfect communication channel. In these contexts, you can simply handle plain data any way you want and always trust the media 100%. Things like compression aren't dangerous because the media is perfect. But, in the particular case of backups, you also sometimes write these things directly to raw media. In that case everything changes drastically. Now your file better have redundancy and error compensation built in because there is no magic lower layer protecting your data from the imperfect media. All the potential problems that compression can amplify, apply to encryption exactly the same. If you restore a plain file with some garbage in the middle, you can probably still actually use that file and the customer would very much rather have 99% of his database file with a few bad records instead of nothing. But, (in the simple case) encrypt that same file and then inflict the same exact damage on it, and it's impossible to decrypt it. You've lost the entire file. ...Unless you take special steps to mitigate exactly that scenario, and you make your encrypter or compressor work with defined and self-contained blocks of data so that you can lose a block and yet still decrypt the rest, or embed some other form of error compensation. It's not quite as bad as that though. At least for simple compression, bzip2, gzip, and plain old zip all have recovery tools that can sometimes salvage all the good parts of a damaged compressed tar. As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs. Years and years ago there was a simple thing called ctar that was some free code on usenet, that a few different companies went on to enhance and build long-running commercial products on, (ctar, lone-tar, backupedge, (at least?), ctar is no longer sold, but lonetar and backupedge are, and are far far advanced since then) . ctar had a very simple form of compression error mitigation which was the simple fact that when you told ctar to use compression, what it did was it compressed each file individually, seperately, inside the archive, while the archive itself was not technically compressed. The archive was essentially a plain old tar, well, enhanced but compatible, with every file inside compressed with "compress" ("ncompress" today), with no .Z extension added to the filenames. You could unpack a ctar archive with most ordinary tar programs, you'd just have to uncompress each file yourself afterwards. The point is though that even that very simple inversion of the normal process mitigated loss by the fact that, in that kind of archive, when the media was damaged, you only lost as little as one affected file, because the compression only spanned one file at a time. **Actually tar does have one problem in that it doesn't handle all forms of files. It goes beyond the already mentioned ACL's & other extended attributes. Most tar implimentations can't even handle fifo's, device nodes, hard links, sometimes not even symlinks. One reason certain things use cpio many times is because most systems' cpio can, even very old systems. Star is one free tar implementation that can handle all that stuff, and lots of other extended attributes, and long path & filenames, but not every possible new kind of extended attribute or metadata. It's been around long enough that it exists on most commonly used platforms so it's reasonably safe to use it's enhanced features even if portability is a concern. you just have to be willing to get star for any target system. It makes every attempt to be backwards compatible with ordinary tar imlementations so that if unpacked with a plain tar, you will get most or all of your data, just not any acls or other extended attributes that your tar program doesn't understand. Backupedge, ctar, and lonetar are all like that too. But Star doesn't have the stuff that xar/dar/parchive does for error-correction, I was still just talking about extended attributes and long filenames and stuff here. wow sorry for the book... -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2009-09-15 at 11:04 -0400, Brian K. White wrote:
Per Jessen wrote:
Carlos E. R. wrote:
Well, what I mean is that a compressed tar is not reliable as a backup procedure. I don't know what is reliable in Linux, but tar isn't. It is much less reliable than, for example, the old pctools backup from central point software was twenty years ago.
Okay, that's an interesting point - I don't think I have heard anyone complain about tars reliability before. What do you see as a more reliable tool/utility then?
I do not know :-( The old Pcbackup for MSDos from pctools was reliable (per the definition below), but had other problems, like incompatible version upgrades. And of course, it is closed source, and probably not available. I would like to find something similar for Linux, and free. An example: I have an old backup made in a set of about 80 floppies (the 360 KB type) which is still fully retrievable, even though it has read errors: the software correct them.
It's not that tar has an unsuspected achilles heel or weakness.** Tar is fine. It's just that tar lacks any extra strength. Tar is fine as long as the underlying media is perfect.
Exactly. That's the point. There is no integrity check, there is no "forward" error correction...
In fact all the low-level protocols for networking and media storage have some form of error detection and correction built-in, so that you the user or higher-level protocol do not have to worry about it. But, backups and tars are a special case.
Exactly.
Sometimes you treat a tar or a compressed tar as a simple file in a filesystem. In that case, your "media" is perfect. If it's not, as was said, you have bigger problems. But really the media isn't in fact perfect. Merely the disk has automatic bad block detection and remapping, and the raid controller or software raid maintains extra data to detect and transparently correct media errors so that you never see them at the application level. Similarly, tcp/ip garantees that you at the application level never see any of the garbage that's actually on the wire. the network cards and the tcp/ip stack all do whatever amount of checksumming and retrying is required transparently and present you with a magically perfect communication channel. In these contexts, you can simply handle plain data any way you want and always trust the media 100%. Things like compression aren't dangerous because the media is perfect.
Exactly.
But, in the particular case of backups, you also sometimes write these things directly to raw media. In that case everything changes drastically. Now your file better have redundancy and error compensation built in because there is no magic lower layer protecting your data from the imperfect media.
Exactly :-)
All the potential problems that compression can amplify, apply to encryption exactly the same. If you restore a plain file with some garbage in the middle, you can probably still actually use that file and the customer would very much rather have 99% of his database file with a few bad records instead of nothing. But, (in the simple case) encrypt that same file and then inflict the same exact damage on it, and it's impossible to decrypt it. You've lost the entire file.
Exactly again :-)
...Unless you take special steps to mitigate exactly that scenario, and you make your encrypter or compressor work with defined and self-contained blocks of data so that you can lose a block and yet still decrypt the rest, or embed some other form of error compensation.
It's not quite as bad as that though. At least for simple compression, bzip2, gzip, and plain old zip all have recovery tools that can sometimes salvage all the good parts of a damaged compressed tar.
I understand that zip has verification and perhaps error recovery fields, and the compression is an integral part of the archive. But not targz. I have heard of cases where a single error destroyed the possibility of uncompressing the archive, and a large part if not all was unrecoverable. There are independent tools to attempt repair, but they do not always work.
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs.
I wonder why not on free software. ...
The archive was essentially a plain old tar, well, enhanced but compatible, with every file inside compressed with "compress" ("ncompress" today), with no .Z extension added to the filenames. You could unpack a ctar archive with most ordinary tar programs, you'd just have to uncompress each file yourself afterwards.
The point is though that even that very simple inversion of the normal process mitigated loss by the fact that, in that kind of archive, when the media was damaged, you only lost as little as one affected file, because the compression only spanned one file at a time.
True. There was a script included with SuSE some versions back that did that with cpio or afio, I think. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqv6ZIACgkQtTMYHG2NR9UwogCfXR8gYhbG8XHu9waFdp1QXErB j5UAnRwo1QW0XKtWS6jg/eV2JeUtQwIT =B6JF -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs.
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
On Tue, Sep 15, 2009 at 3:57 PM, Boyd Stephen Smith Jr. <bss@iguanasuicide.net> wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs.
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
Has anyone gotten Amanda to write to disk? I found this on Novell site and would love to use it. http://www.novell.com/coolsolutions/feature/18050.html -- ---------------------------------------- Discover it! Enjoy it! Share it! openSUSE Linux. ----------------------------------------- openSUSE -- http://en.opensuse.org/User:Terrorpup openSUSE Ambassador openSUSE Member skype -- terrorpup twitter -- terrorpup friendfeed -- http://friendfeed.com/terrorpup Come join me at the Atlanta Linux Fest, September 19th, 2009 http://atlantalinuxfest.org/. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2009-09-15 at 14:57 -0500, Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs.
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked.
And they have those features we are refering to, like automatic error detection and recovery? - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqv8rAACgkQtTMYHG2NR9W9BACgjh/9BA/krYrOZM+JbLrX+Ksk VBsAoJTK4Ui9uoWR0l9zbOHIMji//cVm =5SrD -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 15 September 2009 15:01:50 Carlos E. R. wrote:
On Tuesday, 2009-09-15 at 14:57 -0500, Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked.
And they have those features we are refering to, like automatic error detection and recovery?
Based on my read of Brain's mail, which you have quoted twice: Yes. If you'd like to read the relevant parts of Brain's mail, it is (still) quoted above. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2009-09-15 at 15:11 -0500, Boyd Stephen Smith Jr. wrote:
And they have those features we are refering to, like automatic error detection and recovery?
Based on my read of Brain's mail, which you have quoted twice: Yes.
If you'd like to read the relevant parts of Brain's mail, it is (still) quoted above.
Which says:
too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I take the "...?" as meaning that he doesn't know. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkqwF3wACgkQtTMYHG2NR9W9xQCfQmeVra9HCpbzFb8mcdbH/zxU wOIAoJE/wxdqin00iv/+i/GcRiSp8TA/ =eR1+ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 15 September 2009 17:38:50 Carlos E. R. wrote:
too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I take the "...?" as meaning that he doesn't know.
I took it ("...?") as "and other programs I don't know about". -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
Carlos E. R. wrote:
too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I take the "...?" as meaning that he doesn't know.
No it meant more programs that I don't happen to know about in addition to those. There are obviously lots of archiving utilities that each have some quirky special feature that nothing else has and I have no idea what they all are. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 15:01:50 Carlos E. R. wrote:
On Tuesday, 2009-09-15 at 14:57 -0500, Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked.
And they have those features we are refering to, like automatic error detection and recovery?
Based on my read of Brain's mail, which you have quoted twice: Yes.
If you'd like to read the relevant parts of Brain's mail, it is (still) quoted above.
Yeah, and it looks like I have to eat that particular statement. I was mixing thoughts a little , thinking about the ability to back up extended attributes and acls, and everything else that tar can't, and the capacity for fault-tolerance. But that's no excuse. Bacula has a checksum in it's block header but that's just to detect and contain errors, not actually recover from them. Several things, almost everything, has some ability to do that, detect a media error and skip past it and resume restoring good data beyond it, even with compression. Some things can try dd_rescue technique of just plain re-reading the same bad spot over and over hoping to eventually get a good read by luck or build up the data cumulatively from the multiple passes. But I can't find anything really that adds in ECC data except parchive. Again. No excuse for the sloppy statement. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 15:01:50 Carlos E. R. wrote:
On Tuesday, 2009-09-15 at 14:57 -0500, Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?)
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked.
And they have those features we are refering to, like automatic error detection and recovery?
Based on my read of Brain's mail, which you have quoted twice: Yes.
If you'd like to read the relevant parts of Brain's mail, it is (still) quoted above.
... But I can't find anything really that adds in ECC data except parchive.
Here's another. http://users.softlab.ntua.gr/~ttsiod/rsbep.html It compiled effortlessly but I haven't tested it yet. I came across a few references to a standalone util called simply "ecc" which seems to have disappeared from the face of the earth. I can't tell yet if it was because of patent issues or simply something else made it obsolete. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday, 2009-09-15 at 14:57 -0500, Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs. I wonder why not on free software. Amanda and Bacula are free software the last time I checked. And they have those features we are refering to, like automatic error detection and recovery?
No, all these use a backend for the actual archive process. If the backend you use is trusted then the resulting solution is trusted. As I recall the *default* backend for most of these solutions is tar, but you can change it [in most cases, I believe]. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Although not free I really prefer to use Backup Edge http://www.microlite.com/ It has saved me time and time again. I require my clients to get it or have something similar. For all that it does, I really have not seen any Open Source Equiv. It just works. It verifies my backup regardless of media choosen. -- Boyd Gerber <gerberb@zenez.com> 801 849-0213 ZENEZ 1042 East Fort Union #135, Midvale Utah 84047 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Boyd Stephen Smith Jr. wrote:
On Tuesday 15 September 2009 14:22:50 Carlos E. R. wrote:
As for tools designed to embed data to assist in recovering from or at least minimizing loss of data due to media errors, including in concert with compression: In the free world right off the top of my head, dar and xar. And dar just uses parchive which you could use seperately with something else too. And any of the official backup programs (amanda, zmanda, bru, bacula, arkeia, ...?) And basically any of the commercial programs.
I wonder why not on free software.
Amanda and Bacula are free software the last time I checked.
Seems silly to quote something that is already quoted right there above but... "In the free world right off the top of my head, dar and xar." Dar itself has checksumming so it can detect media errors and contain their scope to just the affected files and not the whole archive, including with encryption. But doesn't go as far as adding ECC data to actually compensate and magically produce lost data. However it works with a sepoerate add-on utility called parchive which does exactly that. Better yet, youy can use parchive with pretty much any other utility you want, not just dar. xar is newer and does all that stuff within itself. Both attempt to handle all the stuff new filesystems and new kernels support that old file formats have no provisions for because the ideas hadn't been thought of yet when tar and cpio were developed. Star is a funny case where he tries to squeeze every possible bit of functionality out of the latest tar specifications, making full use of the allowances for vendor or future-use extensions. The fact that mostly no one elses tar implementations does this except the commercial "supertars" like lonetar and backupedge, isn't a problem with the file format. I love star. I wonder if there is a neat clean way to add ECC data within the existing framework the way star manages to handle all the new and extended file metadata within it? If not, Well there is always the parchive route which has really not much wrong with it that I can see. http://dar.linux.free.fr/doc/Features.html : "DATA PROTECTION: dar relies on the Parchive program for data protection against media errors." http://parchive.sourceforge.net/ : "Parchive: Parity Archive Volume Set The original idea behind this project was to provide a tool to apply the data-recovery capability concepts of RAID-like systems to the posting and recovery of multi-part archives on Usenet." http://code.google.com/p/xar/ : I must have misremembered something about xar. I see nothing there about ECC data. It certainly seems like that could be just one of many possible uses for it's extensively extensible format, so perhaps what I'm remembering is some discussion about that, not any actual feature currently implimented. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
... wow sorry for the book...
Thanks for this book! I needed this kind of information, since I'm not an experienced admin. John Perry -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, 2009-09-10 at 07:21 -0700, Randall R Schulz wrote:
Guys.
I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory casee, but I am having problems finding a good example of a back up script. I am looking to make a cpio image of the whole box. Lucky we have a SAN with storage that will be able to hold the back up. I don't think there's any reason to prefer cpio over tar. It's a less-used and more antiquated format. TAR also permits more compressions schemes (in fact, beyond the ones it will invoke itself
On Thursday September 10 2009, Chuck Payne wrote: based on options like 'j' or 'z', it can run arbitrary external compression commands if you wish). I recommend you stick with your tar-based scheme, if its suits you. Depending on your needs, an rsync approach might be a better choice.
Ditto on the use of cpio. But I'd recommend star over tar. Use star's exustar format and you include ACLs and metadata in your backup which [sadly] tar still cannot do. star -c -acl H=exustar bs=8k fs=128m f=$BACKUPFILE -bz \ --list=/etc/BackupFileList.txt --dodesc The GNU backup/restore utilities are lagging behind the features of current file-systems.
One thing I've taken to doing is augmenting my automated backup based on CrashPlan with a rotating (currently just every other day) rsync snapshots of critical directories.
Agree. With the --acls and --xattrs current versions of rsync will include the meta-data as well. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 10 September 2009 09:37:04 Adam Tauno Williams wrote:
On Thu, 2009-09-10 at 07:21 -0700, Randall R Schulz wrote:
On Thursday September 10 2009, Chuck Payne wrote:
I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory
I don't think there's any reason to prefer cpio over tar. I recommend you stick with your tar-based scheme, if its suits you. Depending on your needs, an rsync approach might be a better choice.
Ditto on the use of cpio. But I'd recommend star over tar. Use star's exustar format and you include ACLs and metadata in your backup which [sadly] tar still cannot do.
I suggest pax over any other archiver. It speaks at least 2 formats (cpio and tar; both specified by SUSv2) and most implementations support additional formats (e.g. pax; specified by SUSv3). It is the preferred replacement for tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
Boyd Stephen Smith Jr. wrote:
tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998.
Which hasn't stopped rpm and initramfs from using cpio :-) /Per -- Per Jessen, Zürich (22.6°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 10 September 2009 11:12:20 Per Jessen wrote:
Boyd Stephen Smith Jr. wrote:
tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998.
Which hasn't stopped rpm and initramfs from using cpio :-)
While I could certainly be wrong, I believe rpm (both the application and file format) was around before cpio was marked "LEGACY". Not breaking things is often reason enough to continue to use "LEGACY" software. In any case, the file formats are not marked "LEGACY" in SUS, just the "cpio" and "tar" utilities. The file formats are documented under the "pax" utility and will (I imagine) probably be supported for quite a while. The tools that build RPMs and initRAMfs files could be updated to use pax, assuming the processes that consume the data are expecting standard cpio format files, and not some odd GNU (or other non-standard) variant. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
Boyd Stephen Smith Jr. wrote:
On Thursday 10 September 2009 11:12:20 Per Jessen wrote:
Boyd Stephen Smith Jr. wrote:
tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998.
Which hasn't stopped rpm and initramfs from using cpio :-)
While I could certainly be wrong, I believe rpm (both the application and file format) was around before cpio was marked "LEGACY".
I looked, but I couldn't find anything about when rpm came about. I also think it's pre-dates the LEGACYfication.
Not breaking things is often reason enough to continue to use "LEGACY" software.
In any case, the file formats are not marked "LEGACY" in SUS, just the "cpio" and "tar" utilities.
Ah, that's an interesting twist.
The tools that build RPMs and initRAMfs files could be updated to use pax,
What's interesting about initramfs is that the cpio format was introduced with 2.6, so quite late really. /Per -- Per Jessen, Zürich (22.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, 2009-09-10 at 10:53 -0500, Boyd Stephen Smith Jr. wrote:
On Thu, 2009-09-10 at 07:21 -0700, Randall R Schulz wrote:
On Thursday September 10 2009, Chuck Payne wrote:
I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory I don't think there's any reason to prefer cpio over tar. I recommend you stick with your tar-based scheme, if its suits you. Depending on your needs, an rsync approach might be a better choice. Ditto on the use of cpio. But I'd recommend star over tar. Use star's exustar format and you include ACLs and metadata in your backup which [sadly] tar still cannot do. I suggest pax over any other archiver. It speaks at least 2 formats (cpio and tar; both specified by SUSv2) and most implementations support additional
On Thursday 10 September 2009 09:37:04 Adam Tauno Williams wrote: formats (e.g. pax; specified by SUSv3). It is the preferred replacement for tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998.
Does pax archive ACLs and extended attributes? The man page doesn't mention those issues at all; actually from the man page it doesn't look impressive at all - still stupid filename and path length limitations, no mention of UNICODE support, etc... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 10 September 2009 12:25:25 Adam Tauno Williams wrote:
On Thu, 2009-09-10 at 10:53 -0500, Boyd Stephen Smith Jr. wrote:
On Thursday 10 September 2009 09:37:04 Adam Tauno Williams wrote:
On Thu, 2009-09-10 at 07:21 -0700, Randall R Schulz wrote:
On Thursday September 10 2009, Chuck Payne wrote:
I have about twenty boxes I need to back up. I been told that cpio is go to do that, I currently use tar but by directory by directory
I don't think there's any reason to prefer cpio over tar.
Ditto on the use of cpio. But I'd recommend star over tar.
I suggest pax over any other archiver. It is the preferred replacement for tar or cpio in the Single UNIX Specification. Both tar and cpio have been marked "LEGACY" since at least 1998.
Does pax archive ACLs and extended attributes?
<http://www.opengroup.org/onlinepubs/000095399/utilities/pax.html> documents what pax, tar, and cpio can do *portably*. Any individual implementation may be able to do more. In particular, the pax extended header reserves the "security." prefix for ACLs to be later standardized. A pax implementation could use (e.g.) "gnu.linux-ea" values to (re)store the EAs (which includes ACLs) from a Linux file system that supports them. If you don't care that your script/program depends on very specific and usually not-well-defined features of the particular implementation you happen to be developing against, that's fine with me. I prefer to write scripts and programs I can take to the *BSDs and Mac OS X (UNIX 2003 Certified) without modification. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
Chuck Payne wrote:
So far I have come up with this, but I don't think it best. What I like to ask the group is does anyone have a nice back up script that can share with me, or can you expand on my script. One more question can you do exclude with cpio like tar?
Filter the filenames before you pipe them to cpio. Or adjust your find parameters.
find -d . -print | cpio -ova > /backup/$host.$dat.cpio
I would use: find -d . -print0 | cpio --null -ova > /backup/$host.$dat.cpio to cope with all kinds of weird and wonderful filenames. -- Per Jessen, Zürich (22.4°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, Sep 10, 2009 at 10:24 AM, Per Jessen <per@opensuse.org> wrote:
Chuck Payne wrote:
So far I have come up with this, but I don't think it best. What I like to ask the group is does anyone have a nice back up script that can share with me, or can you expand on my script. One more question can you do exclude with cpio like tar?
Filter the filenames before you pipe them to cpio. Or adjust your find parameters.
find -d . -print | cpio -ova > /backup/$host.$dat.cpio
I would use:
find -d . -print0 | cpio --null -ova > /backup/$host.$dat.cpio
to cope with all kinds of weird and wonderful filenames.
-- Per Jessen, Zürich (22.4°C)
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Thanks Per I will try that out. Randall, this is meant a temp solution. We are set up Netback up, but until then we have to be back to recovery these boxes should anything go pear shape. Thanks I will look into CrashPlan. -- ---------------------------------------- Discover it! Enjoy it! Share it! openSUSE Linux. ----------------------------------------- openSUSE -- http://en.opensuse.org/User:Terrorpup openSuSE Ambassador openSuSE Member skype -- terrorpup twitter -- terrorpup friendfeed -- http://friendfeed.com/terrorpup Come join me at the Atlanta Linux Fest, September 19th, 2009 http://atlantalinuxfest.org/. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (14)
-
Adam Tauno Williams -
Anton Aylward -
Boyd Lynn Gerber -
Boyd Stephen Smith Jr. -
Brian K. White -
Carlos E. R. -
Chuck Payne -
Greg Freemyer -
James Knott -
John E. Perry -
Ken Schneider - openSUSE -
Per Jessen -
Randall R Schulz -
Will Stephenson