[opensuse] strange mdraid problem
Hello: For some reason (probably a loose cable) all of my mdraid type 1 (mirror) arrays became degraded. After replugging the cables I have resynchronized most of the arrays but not all of them. The arrays that have been resynched work nicely but those devices which I haven't resynched act strange: at every boot either one or the other disk of the array is assembled to the array. That is, one time /dev/md8 has /dev/sda8 assembled and /dev/sdb8 kicked off and the other time /dev/md8 has /dev/sdb8 assembled and /dev/sda8 kicked off. By the time I discovered this I've had different data on the two disks of the array. My questions: 1. If an array becomes degraded and only one device is assembled to it, the other one should be kicked off consequently after this, until the array has been resynched or the missing deviced is readded manually. Why then one or the other device is assembled to the array at each boot? 2. How can I force mdraid to assemble only the same one of the devices to the degraded array? (For example /dev/sda8 to /dev/md8 and not /dev/sdb8.) Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
07.12.2015 19:58, Istvan Gabor пишет:
Hello:
For some reason (probably a loose cable) all of my mdraid type 1 (mirror) arrays became degraded. After replugging the cables I have resynchronized most of the arrays but not all of them. The arrays that have been resynched work nicely but those devices which I haven't resynched act strange: at every boot either one or the other disk of the array is assembled to the array. That is, one time /dev/md8 has /dev/sda8 assembled and /dev/sdb8 kicked off and the other time /dev/md8 has /dev/sdb8 assembled and /dev/sda8 kicked off. By the time I discovered this I've had different data on the two disks of the array.
My questions:
1. If an array becomes degraded and only one device is assembled to it, the other one should be kicked off consequently after this, until the array has been resynched or the missing deviced is readded manually. Why then one or the other device is assembled to the array at each boot?
If only one piece is visible, how can mdraid know it is stale?
2. How can I force mdraid to assemble only the same one of the devices to the degraded array? (For example /dev/sda8 to /dev/md8 and not /dev/sdb8.)
Only by wiping out the wrong one. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkovírta:
07.12.2015 19:58, Istvan Gabor пишет:
Hello:
For some reason (probably a loose cable) all of my mdraid type 1 (mirror) arrays became degraded. After replugging the cables I have resynchronized most of the arrays but not all of them. The arrays that have been resynched work nicely but those devices which I haven't resynched act strange: at every boot either one or the other disk of the array is assembled to the array. That is, one time /dev/md8 has /dev/sda8 assembled and /dev/sdb8 kicked off and the other time /dev/md8 has /dev/sdb8 assembled and /dev/sda8 kicked off. By the time I discovered this I've had different data on the two disks of the array.
My questions:
1. If an array becomes degraded and only one device is assembled to it, the other one should be kicked off consequently after this, until the array has been resynched or the missing deviced is readded manually. Why then one or the other device is assembled to the array at each boot?
Thanks, Andrei.
If only one piece is visible, how can mdraid know it is stale?
OK. How do I know that only one device is visible? I have several arrays and both devices are visible and assembled in the resynched group. What you're saying is that in case of ~10 arrays (all have been resynched after the failure) both devices are always visible (dev/sda* and /dev/sdb*) but in case of not synched (and only in not synched) either /dev/sda* or /dev/sdb* is visible alternatively at different boots. How can I confirm that this causes the problem?
2. How can I force mdraid to assemble only the same one of the devices to the degraded array? (For example /dev/sda8 to /dev/md8 and not /dev/sdb8.)
Only by wiping out the wrong one.
You mean wiping out the raid info block (raid sector)? Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/07/2015 11:52 AM, Istvan Gabor wrote:
OK. How do I know that only one device is visible? I have several arrays and both devices are visible and assembled in the resynched group. What you're saying is that in case of ~10 arrays (all have been resynched after the failure) both devices are always visible (dev/sda* and /dev/sdb*) but in case of not synched (and only in not synched) either /dev/sda* or /dev/sdb* is visible alternatively at different boots. How can I confirm that this causes the problem?
Istvan, I had a similar issue where I had a disk controller that was flaky. I still do not know exactly how it happened, but apparently on one boot, the array booted into degraded mode and did not see the other disk. When that occurred, it continued to write to the good disk as it normally would. On next boot, the other disk re-appeared and dmraid was stuck. It saw both metadata saying they were fine and if the event counts are not that far off, it doesn't know which is the good disk. It should kick one out and continue on the one with the most recent event. To recover, you 'fail' and 'remove' the bad device (or the one dmraid thinks is bad), Make sure you fail the *correct* partition, e.g.: # mdadm /dev/md1 --fail /dev/sdb5 --remove /dev/sdb5 mdadm: set device faulty failed for /dev/sdb5: No such device *note:* since mdadm has already kicked the drive, you will receive the 'No such device' warning above (this is normal). Then re-'add' the device: # mdadm /dev/md1 --add /dev/sdb5 mdadm: re-added /dev/sdb5 That will start the resync. Good luck. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi All,
Just to let a little contribution here:
Istvan, if your array contains any kind of Write-intent bitmap, as
most created by YaST possess, the solution provided David alone will
not (i think) rebuild your whole array, but only the "missing" sectors
that where changed since it was last removed. You can check if your
array has Write-intent bitmap by issuing:
~# cat /proc/mdstat
Then observe the output. Here is an example of two arrays, one WITHOUT
bitmap (/dev/md0), and another WITH bitmap (/dev/md2).
md0 : active raid1 sdb1[3] sda1[2]
127988 blocks super 1.0 [2/2] [UU]
md2 : active raid1 sdb3[3] sda3[2]
1462913296 blocks super 1.0 [2/2] [UU]
bitmap: 11/11 pages [44KB], 65536KB chunk
See the bitmap line on md2? That means you have a Write-intent bitmap
associated with that array.
If you do not have any Write-intent bitmap, David solution will
rebuild your whole array.
If not, you have to solutions:
1) Remove the Write-intent bitmap from the array, by issuing:
~# mdadm --grow -b none (replace question mark)
Then, carry out David instructions.
You may than re-add the bitmap, with:
~# mdadm --grow -n internal
NOTE: This will create an internal Write-intent bitmap
2) Carry out David instructions (if you think the array did not
startup in the order you wanted it to, according to David's
explanations).
Then issue a full rebuild on the array, by issuing:
~# echo repair >> /sys/block/md?/md/sync_action (replace question mark)
Hope this helps,
Best regards,
Rui
On Tue, Dec 8, 2015 at 5:35 AM, David C. Rankin
On 12/07/2015 11:52 AM, Istvan Gabor wrote:
OK. How do I know that only one device is visible? I have several arrays and both devices are visible and assembled in the resynched group. What you're saying is that in case of ~10 arrays (all have been resynched after the failure) both devices are always visible (dev/sda* and /dev/sdb*) but in case of not synched (and only in not synched) either /dev/sda* or /dev/sdb* is visible alternatively at different boots. How can I confirm that this causes the problem?
Istvan,
I had a similar issue where I had a disk controller that was flaky. I still do not know exactly how it happened, but apparently on one boot, the array booted into degraded mode and did not see the other disk. When that occurred, it continued to write to the good disk as it normally would. On next boot, the other disk re-appeared and dmraid was stuck. It saw both metadata saying they were fine and if the event counts are not that far off, it doesn't know which is the good disk. It should kick one out and continue on the one with the most recent event.
To recover, you 'fail' and 'remove' the bad device (or the one dmraid thinks is bad), Make sure you fail the *correct* partition, e.g.:
# mdadm /dev/md1 --fail /dev/sdb5 --remove /dev/sdb5 mdadm: set device faulty failed for /dev/sdb5: No such device
*note:* since mdadm has already kicked the drive, you will receive the 'No such device' warning above (this is normal).
Then re-'add' the device:
# mdadm /dev/md1 --add /dev/sdb5 mdadm: re-added /dev/sdb5
That will start the resync. Good luck.
-- David C. Rankin, J.D.,P.E.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- Rui Santos Veni, Vidi, Linux -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
07.12.2015 20:52, Istvan Gabor пишет:
Andrei Borzenkovírta:
07.12.2015 19:58, Istvan Gabor пишет:
Hello:
For some reason (probably a loose cable) all of my mdraid type 1 (mirror) arrays became degraded. After replugging the cables I have resynchronized most of the arrays but not all of them. The arrays that have been resynched work nicely but those devices which I haven't resynched act strange: at every boot either one or the other disk of the array is assembled to the array. That is, one time /dev/md8 has /dev/sda8 assembled and /dev/sdb8 kicked off and the other time /dev/md8 has /dev/sdb8 assembled and /dev/sda8 kicked off. By the time I discovered this I've had different data on the two disks of the array.
My questions:
1. If an array becomes degraded and only one device is assembled to it, the other one should be kicked off consequently after this, until the array has been resynched or the missing deviced is readded manually. Why then one or the other device is assembled to the array at each boot?
Thanks, Andrei.
If only one piece is visible, how can mdraid know it is stale?
OK. How do I know that only one device is visible? I have several
I think similar topic was discussed relatively recently, may be even by you. What version of openSUSE do you have? IIRC in 13.1 (and surely before) one of startup scripts forcibly started degraded array; so if there was any delay in discovering devices this could result in described behavior.
arrays and both devices are visible and assembled in the resynched group. What you're saying is that in case of ~10 arrays (all have been resynched after the failure) both devices are always visible (dev/sda* and /dev/sdb*) but in case of not synched (and only in not synched) either /dev/sda* or /dev/sdb* is visible alternatively at different boots. How can I confirm that this causes the problem?
I would start with kernel logs, it could show relative timing of events.
2. How can I force mdraid to assemble only the same one of the devices to the degraded array? (For example /dev/sda8 to /dev/md8 and not /dev/sdb8.)
Only by wiping out the wrong one.
You mean wiping out the raid info block (raid sector)?
Thanks,
Istvan
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Andrei Borzenkov -
David C. Rankin -
Istvan Gabor -
Rui Santos