[opensuse] RAID5 & Power outages
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue. Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....) Anybody knows if the raid5 (md) would survive this scenario? Regards, Ciro -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Friday 2007-10-26 at 00:16 -0300, Ciro Iriarte wrote:
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue.
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
It should. It might, actually: no filesystem likes a power outage, and no raid will protect you from that. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFHIcU6tTMYHG2NR9URAmlMAJkBQPHZPYVf+ep0Ekvv+pTnreqMtwCdELN4 018uZgBrELbLawsNzmkUjXs= =fW8y -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
2007/10/26, Carlos E. R. <robin.listas@telefonica.net>:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Friday 2007-10-26 at 00:16 -0300, Ciro Iriarte wrote:
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue.
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
It should.
It might, actually: no filesystem likes a power outage, and no raid will protect you from that.
- -- Cheers, Carlos E. R. I expect it to survive in the sense that i don't have to reinit the raid losing all the data... A filesysten check procedure requires a working RAID...
Ciro -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Friday 2007-10-26 at 23:04 -0300, Ciro Iriarte wrote:
It should.
It might, actually: no filesystem likes a power outage, and no raid will protect you from that.
I expect it to survive in the sense that i don't have to reinit the raid losing all the data... A filesysten check procedure requires a working RAID...
Ah, I see what you mean. I can't certify that. I think it survives, though. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFHIxJutTMYHG2NR9URAunAAJ4rk1BglIkRhZbZiotfou82OZnOYQCeIrWt qBjYtWxSiDK22X8o0NLc5RA= =b56v -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Ciro Iriarte wrote:
I expect it to survive in the sense that i don't have to reinit the raid losing all the data... A filesysten check procedure requires a working RAID...
Working = fully operational or operating in degraded mode, i.e. being rebuilt or with one dead disk. /Per Jessen, Zürich -- http://www.spamchek.com/ - your spam is our business. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Ciro Iriarte wrote:
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
A RAID5 will survive a single-disk failure. I believe RAID6 can deal with a dual-disk failure, but you'll need more disks. /Per Jessen, Zürich -- http://www.spamchek.com/ - your spam is our business. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
2007/10/26, Per Jessen <per@computer.org>:
Ciro Iriarte wrote:
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
A RAID5 will survive a single-disk failure. I believe RAID6 can deal with a dual-disk failure, but you'll need more disks.
/Per Jessen, Zürich
Well, i'm not talking about a phisical failure, but logical failure of the array, the kind of failure that would require a rebuild of the array... Regards, Ciro
Ciro Iriarte wrote:
2007/10/26, Per Jessen <per@computer.org>:
Ciro Iriarte wrote:
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
A RAID5 will survive a single-disk failure. I believe RAID6 can deal with a dual-disk failure, but you'll need more disks.
/Per Jessen, Zürich
Well, i'm not talking about a phisical failure, but logical failure of the array, the kind of failure that would require a rebuild of the array...
Ah, then the answer is - maybe, maybe not. I'm sure it could quite possibly survive, but I also know for a fact it doesn't always. /Per Jessen, Zürich -- http://www.spamchek.com/ - your spam is our business. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 10/25/07, Ciro Iriarte <cyruspy@gmail.com> wrote:
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue.
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
The raid should definately survive, but the portion you are writing at the time could easily be corrupt. Raid5 typically has a 64k per disk data unit, but your issues are going to really be at the 4k page level I think. In particular, when you write/overwrite a page the kernel has to read the original page of data and the checksum page of data. Update both and write them both back out. There is really no way the OS can ensure that both writes actually make it to disk simultaneously. So if you have a power outage while writing to a raid5 the odds are high that one of the 2 updates failed to make it to disk. The good news, is the array itself should be fine. It just the single raid5 stripe that is corrupted. You likely won't find out about the problem until you lose a disk. But even then it will just be a few pages that are impacted. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
2007/10/26, Greg Freemyer <greg.freemyer@gmail.com>:
On 10/25/07, Ciro Iriarte <cyruspy@gmail.com> wrote:
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue.
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
The raid should definately survive, but the portion you are writing at the time could easily be corrupt.
I thought that that kind of corruption would trigger the array rebuild or logical fail...
Raid5 typically has a 64k per disk data unit, but your issues are going to really be at the 4k page level I think.
In particular, when you write/overwrite a page the kernel has to read the original page of data and the checksum page of data. Update both and write them both back out. There is really no way the OS can ensure that both writes actually make it to disk simultaneously.
So if you have a power outage while writing to a raid5 the odds are high that one of the 2 updates failed to make it to disk. The good news, is the array itself should be fine. It just the single raid5 stripe that is corrupted. You likely won't find out about the problem until you lose a disk. But even then it will just be a few pages that are impacted.
So?, i will just heal itself?, continue to work with some corrupted files (corrupted stripe effect)?
Greg
Regards, Ciro -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Ciro Iriarte wrote:
2007/10/26, Greg Freemyer <greg.freemyer@gmail.com>:
On 10/25/07, Ciro Iriarte <cyruspy@gmail.com> wrote:
Anybody has experience with linux Raid5 and power outages?, i currently have two 250GB disks in a RAID1 array in my main workstation. When there's any power outage the array is rebuilt but given it's a mirror it's not a major issue.
Now i would like to add 4 500GB disks in a secondary array with raid5 but i'm afraid i would lose the array if there's any outage (have to mention a UPS in my letter to Santa....)
Anybody knows if the raid5 (md) would survive this scenario?
The raid should definately survive, but the portion you are writing at the time could easily be corrupt.
I thought that that kind of corruption would trigger the array rebuild or logical fail...
Raid5 typically has a 64k per disk data unit, but your issues are going to really be at the 4k page level I think.
In particular, when you write/overwrite a page the kernel has to read the original page of data and the checksum page of data. Update both and write them both back out. There is really no way the OS can ensure that both writes actually make it to disk simultaneously.
So if you have a power outage while writing to a raid5 the odds are high that one of the 2 updates failed to make it to disk. The good news, is the array itself should be fine. It just the single raid5 stripe that is corrupted. You likely won't find out about the problem until you lose a disk. But even then it will just be a few pages that are impacted.
So?, i will just heal itself?, continue to work with some corrupted files (corrupted stripe effect)?
Hi Ciro, Try to think of a RAID Array as a single partition. You put a filesystem "on top" of a partition or a RAID Array. If the power fails, the -partition- DOES "survive". The filesystem may have inconsistencies but it is probably recoverable. It is the same principle ( almost ) with a Soft-RAID array. If you have frequent power loss, IMHO, you should activate RAID write-intent bitmapping. The command is mdadm /dev/mdX -Gb internal If you are afraid about a RAID's inconsistency by issuing "echo check >> /sys/block/mdX/md/sync_action" and check for failures wiht "cat /sys/block/mdX/md/mismatch_cnt". If there are failures, correct it with "echo repair >> /sys/block/mdX/md/sync_action" You can also put it on a cron script... Hope it helps, Rui
Greg
Regards, Ciro
-- Rui Santos http://www.ruisantos.com/ Veni, vidi, Linux! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
2007/10/27, Rui Santos <rsantos@ruisantos.com>:
Hi Ciro,
Try to think of a RAID Array as a single partition. You put a filesystem "on top" of a partition or a RAID Array.
I know it's logically a partition/disk, but the important bit there is "logically", a real partition won't melt down or break into multiple unusable peaces.
If the power fails, the -partition- DOES "survive". The filesystem may have inconsistencies but it is probably recoverable. It is the same principle ( almost ) with a Soft-RAID array.
If you have frequent power loss, IMHO, you should activate RAID write-intent bitmapping. The command is mdadm /dev/mdX -Gb internal
That's what i'm talking about, i wonder if that has a performance penalty. This is the result of the last power outage: mainwks:~ # cat /proc/mdstat Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [linear] md0 : active raid1 sda1[0] sdb1[1] 104320 blocks [2/2] [UU] md1 : active raid1 sda3[0] sdb3[1] 241987008 blocks [2/2] [UU] [==>..................] resync = 11.0% (26760192/241987008) finish=315.0min speed=11384K/sec I assume that would look really nasty on a big raid5...
If you are afraid about a RAID's inconsistency by issuing "echo check >> /sys/block/mdX/md/sync_action" and check for failures wiht "cat /sys/block/mdX/md/mismatch_cnt". If there are failures, correct it with "echo repair >> /sys/block/mdX/md/sync_action"
You can also put it on a cron script...
Hope it helps, Rui
I will look further on the bitmapping topic. Thanks a lot Ciro -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Ciro Iriarte wrote:
I assume that would look really nasty on a big raid5... A UPS battery costs less then 80 euro. It gives your computer 3 minutes to shut down properly.
Kind regards Philippe -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Carlos E. R. -
Ciro Iriarte -
Greg Freemyer -
Per Jessen -
Philippe Landau -
Rui Santos