On Wed November 12 2008 2:57:34 am Per Jessen wrote:
Just thinking out loud:
when a drive in a 2x500Gb RAID1 array fails and is replaced, how long does it take for the array to recover? I can't help worry about the length of time in degraded mode - the longer, the higher the risk of a second drive failure. What do people do to overcome this - run 3- or 4-drive RAID1?
I recently had a drive fail on a 4 drive RAID. Actually (as both an experiment and because it works well), the drive that failed was part of 3 separate MD Raid groups. The first was for the ROOT system, the 2nd was for /home and the last was a 1.3TB RAID 5 that used not only the 1st two drives common to the / and /home arrays (RAID 1s) but the 2 extra drives of the 4 devices. As it turned out, the /dev/sda crashed so badly that even BIOS didn't even recognize the drive existed. This machine was/is primarily used as my file server for my home LAN so I often don't even look at the machine or directly log into it. One day, I did (remotely) log into it and the machine seemed a little sluggish (not much) and I was going to do a system update (from Yast2) of many of the programs. I don't know why, but at some point, I decided to reboot it which is a rarity as they normally run 24x7 for months on end, but when it didn't allow me to reconnect remotely, I went into the other room where the machine physically resides, turned on the monitor and saw it had booted into a maintenance/rescue mode instead of the normal boot. That is where I found one of the drives had failed. To my horror, it was one of the drives with the RAID1 / (root) and /boot and also part of the /home array also, and feared the worse. I replaced the drive and the system booted quite normally and completely. I decided to check the array status with mdadm and other utilities and found that none of the arrays were using the new drive. I then realized I needed to repartition the new drive the same as it was on the defective drive and mdadm -a the drive back to the 3 arrays affected. For root-boot and the home array, the -a and the subsequent rebuild wass almost so fast that it was done before my finger left the <return> key. I thought something was wrong because it was so fast, but it was fine. Then I tackled the 1.3TB raid 5 and did the same mdadm -a on it and it immediately came back. Checking the status showed it was being rebuilt (albeit still available for use during the rebuild time) and the time to completion was 132 minutes for the 1.3TB of data. It completed totally successsfully in just about 2 hours and 15 minutes for a total time to rebuild all three MD RAID devices. I don't know how long it had been running in degraded mode and the 2:15 total repair time was something that made me a believer. Before my stroke, I had taught OS theory including RAID to college students but it was only a theoretical thing about how much RAID was actually worth. Now, it isn't theoretical anymore. RAID is NOT a backup mechanism, good backups still are the only 'protection' for recovery of lost or damaged data, but RAID has convinced me that it is a pretty bulletproof HARDWARE solution for the reliability issue. It often *can* eliminate the need to actually use a backup because it helps prevent data damage in the first place (due to hardware, not pilot, errors) It is an integrity mechinism, not a backup mechanism, but if the harware is 'solid', and pilot error is removed, often backups never get used because the data is always available. Backups are primarily 'cockpit' error recovery, RAID does a great job of providing hardware protection against loss. So, it is no longer theoretical; you need RAID both on the primary AND backup hardware devices. That gives you the best fault tolerance, both human and hardware. That 2:15 hour lesson is one well learned and I wish I still taught because I would now add another 'lab exercise' for my students and provide them the same valuable lesson this 65 year old coot just learned. Richard
/Per
-- /Per Jessen, Zürich