Trying to make a near Perfect Backup System
Hello all, I would like some advice on what would be considered as a "bullet-proof" backup system. The system I am currently using is as follows: some P4 5X 200GB HDD configured as an LVM using ReiserFS (2X SATA, 3X IDE) 1X 20GB HDD holding the OS: SuSE 9.2 Backup Software: Rsync-based shell script run from cron every 24 hours to backup the entire division. Most of the rsync script was gathered from: http://www.mikerubel.org/computers/rsync_snapshots/ I only backup our division, not the entire organisation - that is the IT department's problem. Currently we have no large-capacity tape drive for off-site storage, so that is an obvious MUST-HAVE. I am considering the HP Ultrium 100-200GB tape device for this, should I be going in another direction with tapes? The above PC works very well. However, it's greatest weakness is disk-failure. We recently had the backup system die due to disk failure (there were no spare 200GB disks available in our IT section at the time too :( ), and at the same time one of our most important data analysis pc's also had a disk failure. I could not recover data from the backup LVM, so we had effectively lost all data. Fortunately we were able to recover data from the analysis PC disk via a data recovery specialist. Since we have 40+ pc's in the division, the likelihood of both the backup system and a crucial PC failing at the same time is remote, but it happened. Not surprisingly I was immediately in the "hot-seat" to answer why it happened and why there was no plan for this etc... So, my questions on how do I make a "bullet-proof" backup system are these: 1. I need a tape device for off-site storage, are the LTO Ultrium devices good enough? 2. Should have a 'twin' of the above PC so I have a backup of the backup, since I can't fit more disks into the box? 3. Would a RAID system be better (I have not used RAID yet though) - would that need to be hardware RAID? 4. I have read a little on SUN Storedge systems, can these be used with Linux, and are they better than anything else? 5. So far I have only been using shell scripts for backup, should I be considering commercial packages, if so which are "in-favour" at present? Sorry for the epic posting, but the aforementioned disk failures have prompted the investigation into a "bullet-proof" backup system (and quite rightly so). Many many thanks for any assistance. -dale -- Dale Roblin Council for Geoscience South Africa Tel: +27 (0) 12 841-1492 Fax: +27 (0) 12 841-1424 Disclaimer Legal Notice: By having opened and read this electronic mail, you are deemed to have understood and accepted all disclaimers and conditions pertaining to electronic mail emanating from, and received by The Council for Geoscience, further detail of which may be viewed at the following hyperlink: http://www.geoscience.org.za/disclaimer.htm
A long post deserves a long answer: On 7/14/05, Dale L Roblin wrote:
Hello all,
I would like some advice on what would be considered as a "bullet-proof" backup system. The system I am currently using is as follows:
some P4 5X 200GB HDD configured as an LVM using ReiserFS (2X SATA, 3X IDE) 1X 20GB HDD holding the OS: SuSE 9.2 Backup Software: Rsync-based shell script run from cron every 24 hours to backup the entire division. Most of the rsync script was gathered from: http://www.mikerubel.org/computers/rsync_snapshots/
I only backup our division, not the entire organisation - that is the IT department's problem.
Currently we have no large-capacity tape drive for off-site storage, so that is an obvious MUST-HAVE. I am considering the HP Ultrium 100-200GB tape device for this, should I be going in another direction with tapes?
The above PC works very well. However, it's greatest weakness is disk-failure. We recently had the backup system die due to disk failure (there were no spare 200GB disks available in our IT section at the time too :( ), and at the same time one of our most important data analysis pc's also had a disk failure. I could not recover data from the backup LVM, so we had effectively lost all data. Fortunately we were able to recover data from the analysis PC disk via a data recovery specialist.
Since we have 40+ pc's in the division, the likelihood of both the backup system and a crucial PC failing at the same time is remote, but it happened. Not surprisingly I was immediately in the "hot-seat" to answer why it happened and why there was no plan for this etc...
So, my questions on how do I make a "bullet-proof" backup system are these:
Are you prepared to spend time and money on this project? I would guess that you will need to spend between $10K (US) and $50K (US) and a lot of time too. The more "bullet-proof" the more expensive and time-consuming. Just the software for a project like this could cost over $20K if you go with the market leader (Veritas Net Backup).
1. I need a tape device for off-site storage, are the LTO Ultrium devices good enough?
I just saw this year plus old write-up on tape drives. Probably worth your time to read: http://www.clipper.com/research/TCG2004040.pdf Anyway, personally I like the LTO, but you talk about the LTO 100/200. I assume you are talking about a LTO-1 drive with 100 GB native, 200 GB compressed? FYI: The amount of compressibility varies greatly based on the data type. Above you show having 1 TB of disk on your backup server. Is that big enough to hold a full backup? If not, with LTO-1 you will need a small autoloader or tape library. Small ones are not outrageously expensive and a lot of backup software know how to manage them. Alternatively, there are tape drives out there that will hold almost a TB (compressed) and I like the idea of getting the full backup on a single tape. In particular, the LTO-3 has been out about 6 months and is likely to be supported for many years. (400native/800compressed). (LTO had 75% of the market in 2004, so the product line is very unlikely to go away.) The largest capacity drive I know of is the Sony SAIT-2 (1 TB Native, 2.6 TB Compressed). (I have no first hand experience with this drive). It was supposed to be released in 2005, but I don't know if they are shipping or not. Also I expect they will be very, very expensive. (ie. $10-15K US each)
2. Should have a 'twin' of the above PC so I have a backup of the backup, since I can't fit more disks into the box?
Assuming you are going to use a disk to disk to tape backup scheme it would be very unusual for a department level backup system to have fully redundant systems. OTOH, if you have the resources to duplicate everything, that would be great. I would use SLES 9 with Linux-HA and DRDB to setup a HA cluster of the two machines and DRDB mirroring the data between the computers live.
3. Would a RAID system be better (I have not used RAID yet though) - would that need to be hardware RAID?
Absolutely you need RAID. It sounds like you have a five disk based solution currently. I suspect that if any one drive fails, you lose it all. If true the odds of your failure are 5 times the odds of a single disk failure. Very Bad. Linux has great software RAID, but I prefer hardware RAID and it really is not that expensive. When building a backup server like this, read speed is the most critical aspect. (ie. Tape drives eat lots of data very quickly, your disk system has to be fast to keep up.) I would use RAID 5. For this kind of load it should be faster than RAID 1, and might be faster than RAID 10.
4. I have read a little on SUN Storedge systems, can these be used with Linux, and are they better than anything else?
What do you want to do with that? Just build your own solution.
5. So far I have only been using shell scripts for backup, should I be considering commercial packages, if so which are "in-favour" at present?
If you decide to look at commercial solutions, I would look at HPs Data Protector. Especially for a solution like this, it is pretty cost effective. (ie. HP charges by the number tape drives you are using, not by the number of computers being backed up.) DP also supports the disk-disk-tape concept, but I must admit to not having set that up yet myself.
Sorry for the epic posting, but the aforementioned disk failures have prompted the investigation into a "bullet-proof" backup system (and quite rightly so).
Before your done you will be writing several epics. Companies spend lots of money on backup systems. I was talking to friend that works at a very large company. He said they produce 30-40,000 tapes per week!!!!!
Many many thanks for any assistance.
-dale
-- Dale Roblin
Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century
Dale L Roblin <daler@geoscience.org.za> writes:
1. I need a tape device for off-site storage, are the LTO Ultrium devices good enough?
I've been using an "HP StorageWorks Ultrium 460" with 400 GB Ultrium LTO 2 data cartridges on a PC with SUSE Linux 9.1 without any problem for about 1 year.
3. Would a RAID system be better (I have not used RAID yet though) - would that need to be hardware RAID?
I think a HW RAID 5 (with a spare disk on a shelf) is a good solution. I use 3ware controllers (http://www.3ware.com/) which are directly supported by Linux.
4. I have read a little on SUN Storedge systems, can these be used with Linux, and are they better than anything else?
If I remember it right, these system can be managed remotely and more servers can be connected to one array. This is quite good for HA solutions but the price is higher. I would use it for the primary storage but not for a backup.
5. So far I have only been using shell scripts for backup, should I be considering commercial packages, if so which are "in-favour" at present?
I think the answer depends on many factors. The advantage of simple scripts is that they are cheap and you can tune them as you wish. The disadvantage is that if something goes wrong then you may be the only person who can restore data from the backup since other people may not have appropriate skills. -- A.M.
On Thursday 14 July 2005 18:22, Alexandr Malusek wrote:
3. Would a RAID system be better (I have not used RAID yet though) - would that need to be hardware RAID?
I think a HW RAID 5 (with a spare disk on a shelf) is a good solution. I use 3ware controllers (http://www.3ware.com/) which are directly supported by Linux.
Definitely, RAID5 with 3ware. You can have 12 SATA disks per controller. With 250 GB disks RAID5, and 1 spare, you get 2.5 TB. And you can put more than one controller in a computer. You can make your own (need big power sources, maybe more than one, and motherboard with PCI64), or buy ready made (recommended). Search google or contact 3ware and ask them to recommend a ready made NAS.
participants (4)
-
Alexandr Malusek -
Dale L Roblin -
Greg Freemyer -
Silviu Marin-Caea