https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c1 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #1 from Neil Brown 2012-05-07 09:01:47 UTC --- Thanks for the report. I suspect this is a known problem, the fix for which isn't in 12.1 yet. I'll get you an rpm to test, but it might be a couple of days... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c3 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |volker3204@paradise.net.nz --- Comment #3 from Neil Brown 2012-05-21 04:55:13 UTC --- Probably easiest to try the mdadm from factory http://download.opensuse.org/repositories/Base:/System/openSUSE_Factory/x86_... If that makes a difference you can either just still with that or I might be able to schedule an update for 12.1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c5 --- Comment #5 from Volker Kuhlmann 2012-06-26 18:57:14 NZST --- Sorry Neil NOT fixed. I recompiled the package from factory: mdadm-3.2.5-66.1.x86_64 when installing it, the initrds were re-created so I didn't put in any extra effort there. But it's not the rootfs array that has a problem, it's the /home array. After the first reboot a degraded array was created - expected. I resynced that by adding the second disk and waited until that was finished. After a reboot (or maybe two) the array showed up correctly. After another reboot the array was degraded again: /proc/mdstat Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4] md127 : active raid1 sda6[2] 99546668 blocks super 1.0 [2/1] [_U] Another reboot later two arrays were again created: /proc/mdstat Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4] md125 : active raid1 sdb6[3] 99546668 blocks super 1.0 [2/1] [U_] md127 : active (auto-read-only) raid1 sda6[2] 99546668 blocks super 1.0 [2/1] [_U] This array is used for /home. The other array uses sd[ab]5, is used for /, and doesn't show this split problem. There is no entry in syslog that suggests there might be a hardware error. The same hardware has been rock-solid for successsive openSUSE versions for 5 years. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c7 --- Comment #7 from Volker Kuhlmann 2012-06-27 20:04:15 NZST --- Thanks Neil! If you have an mdadm fixing the second problem I'll be happy to test. Re first problem: [0:0:0:0] disk ATA SAMSUNG HD103UJ 1AA0 /dev/sda [4:0:0:0] disk Generic IC1210 CF 1.9C /dev/sdb [4:0:0:1] disk Generic IC1210 MS 1.9C /dev/sdc [4:0:0:2] disk Generic IC1210 MMC/SD 1.9C /dev/sdd [4:0:0:3] disk Generic IC1210 SM 1.9C /dev/sde [5:0:0:0] disk ATA ST3120026A 3.06 /dev/sdf sda reliably remains to be the Samsung. The Seagate PATA randomly gets inserted as sdb (before) or sdf (after the card reader drives). The card reader is internal; USB connected. My uneducated guess for the delay of sd[bf] is that the kernel has pushed initialisation of PATA right to the end, after SATA and USB. Combined with systemd there's not much telling of timing... I don't see any boot script affecting access to sd[bf]. That disk is old, and contains 2 swap partitions and 2 raid partitions (for / and /home). The rootfs surely must be started first, it never has any problems, and it would start sd[bf] IIUC. Both raids are raid1, booting is with setup --stage2=/boot/grub/stage2 --force-lba (hd0) (hd0,4) setup --stage2=/boot/grub/stage2 --force-lba (hd1) (hd1,4) quit The only non-SuSE init scripts I use are using these dependencies # Provides: boot.local-rc # Required-Start: $local_fs boot.rootfsck # Should-Start: boot.quota boot.cleanup # Required-Stop: $null # Should-Stop: $null # Default-Start: B # Default-Stop: and they call env SYSTEMD_NO_WRAP=1 /etc/init.d/boot.cleanup to get the usual cleanup behaviour with systemd, and they create a few files in /tmp. All of this is operates on the rootfs, not on /home. The /home raid is the one which isn't behaving. The details of the home array are identical to the ones for the rootfs, but I had to create them manually. When I installed oS 12.1 (fresh install) yast was giving me some problems with superblock versions. I didn't want 0.9, and 1.2 didn't work so I had to manually force the use of 1.0. The system boots from the rootfs raid1. (I must be using this type of setup for almost 10 years now.) If there's any other useful information I can provide I'd be happy to. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c9 --- Comment #9 from Neil Brown 2012-09-12 07:53:36 UTC --- I've changed my mind. Don't try adding "udevadm settle" - it won't help (and it is already there anyway). You still need to change /etc/init.d/boot.md though. Please find the line near the top which reads: # Require-Start: boot.udev boot.rootfsck and add another word to the end (after a space): udev-trigger This will ensure that new events have been triggered before boot.md runs, so that the "udevadm settle" can wait for them, and md won't try to assemble incomplete arrays. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c11 --- Comment #11 from Swamp Workflow Management 2012-10-04 20:08:40 UTC --- openSUSE-RU-2012:1291-1: An update that has two recommended fixes can now be installed. Category: recommended (low) Bug References: 760859,772286 CVE References: Sources used: openSUSE 12.1 (src): mdadm-3.2.2-4.9.1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c13 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |NEEDINFO InfoProvider| |volker3204@paradise.net.nz --- Comment #13 from Neil Brown 2012-10-07 22:38:03 UTC --- hmm.. it seems this is more subtle than I thought. Can you try removing /lib/udev/64-md-raid.rules from the initrd? The easiest way to do this is to edit /lib/mkinitrd/scripts/setup-udev.sh and remove the line 64-md-raid.rules \ and then run "mkinitrd". This will make sure the initrd doesn't even try to assemble the 'home' array. I think that might be the problem - it gets assembled too early before all the devices are available. I understand you might not be able to try this immediately, but if you could let me know when you might be able to try, that would be helpful. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c15 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |NEEDINFO InfoProvider| |volker3204@paradise.net.nz --- Comment #15 from Neil Brown 2012-10-09 01:23:10 UTC --- Well, that wasn't it... or maybe... 64-md-raid.rules does a few different things. I want to disable some of them but apparently not all. Could you try editing /lib/udev/rules.d/64-md-raid.rules (maybe take a copy first for safety) and comment out every line that contains "--incremental". Then run "mkinitrd" and try again? Also what is in the "etc/mdadm.conf" file in your initrd? You included it in the original report, but the latest mdadm package tries to avoid using names like "md126" - I just want to check that it is getting that right. The changing of devices names is expected and shouldn't cause any problems. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c17 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |ASSIGNED --- Comment #17 from Neil Brown 2012-10-11 04:55:52 UTC --- Encouraging! I'm still not 100% sure what it going on. Is that mdadm.conf that you gave the one from the root filesystem or the one from the initrd? The easiest way to get the one from the initrd is: zcat /boot/initrd | cpio -i --to-stdout etc/mdadm.conf I'm I correct that your 2 raid1 arrays both use the same two devices? It might be useful to see config/md.sh from the initrd too: zcat /boot/initd | cpio -i --to-stdout config/md.sh -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c19 --- Comment #19 from Neil Brown 2012-10-18 01:53:41 UTC --- Thanks for the extra details. I've run out of ideas. Everything looks right - it should work properly. I'm quite sure that nothing is going wrong inside the initrd. It really looks like there is some race between "mdadm --incremental" being run by udev, and "mdadm -A" being run by boot.md. However the addition of "udev-trigger" to the "Should-Start" line, combined with the "udevadm settle" call already in boot.md, should prevent any race. The latest change you made prevent udev from running "mdadm --incremental" so there is now no chance of a race. I've made a couple of changes to upstream mdadm to improve the interaction between "-I" and "-A" but they are too big to push into 12.2 (or 12.1) without good reason. I'll make sure they get into 12.3 (or whatever comes next). For now I think you'll just have to manage with your current work-around. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c21 Volker Kuhlmann changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Severity|Normal |Major --- Comment #21 from Volker Kuhlmann 2012-10-26 18:47:07 NZDT --- Drats, here's a different problem. On a more modern box I use a similar setup - 12.1, 2x SATA disk, raid1 each for / and /home. Yesterday morning I rebooted, and there was a raid array failure I didn't pay too much attention to, and which I couldn't solve from the root shell, so I rebooted again. No trouble that I noticed then, but it must have come up with md1 (/home) with only 1 active disk. Mid day was a power cut, followed by a boot in the evening. I was missing files I had created that morning, and found that md1 was running with only 1 active disk - this time the *OTHER* one, or at least only that explains it - my missing files were on the non-active disk. It took me a few hours to clean up from that. Just in case I reverted to mdadm-3.2.2-4.4.1.x86_64. Now I rebooted twice for testing, all fine, though I'm unsure what bitmap: 1/2 pages [4KB], 65536KB chunk means. I then tried mdadm-3.2.2-4.9.1.x86_64.rpm again. I rebooted twice, and each time md1 came up with only 1 active disk. Reverting back to mdadm-3.2.2-4.4.1 fixes that problem. Increasing importance of this bug - while it may help the issue with SATA/PATA raid, for SATA raid it seems dangerous. And a raid implementation randomly starting with 1 out of 2 disks is less than useless (it's begging for data loss). The update to mdadm-3.2.2-4.9.1 was a little while ago, and yesterday I first noticed the problem. There have been at least one more reboot, and it's theoretically possible I failed to notice running on only 1 disk. I attach some status info from when running with mdadm-3.2.2-4.9.1. smartctl shows no disk problem - this is a software problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c23 --- Comment #23 from Volker Kuhlmann 2012-10-26 18:48:18 NZDT --- Created an attachment (id=510989) --> (http://bugzilla.novell.com/attachment.cgi?id=510989) raid details -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c25 --- Comment #25 from Volker Kuhlmann 2012-10-26 18:49:22 NZDT --- Created an attachment (id=510991) --> (http://bugzilla.novell.com/attachment.cgi?id=510991) /proc/mdstat -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c27 --- Comment #27 from Volker Kuhlmann 2012-11-01 21:02:12 NZDT --- Installed the updates from yesterday including the dbus packages. Rebooted, failure to run md1 altogether - this is on the machine with 2 SATA disks in RAID1, and with the previous mdadm-3.2.2-4.4.1.x86_64 Rebooted again, md1 is started, but with only 1 disk. Rebooted some more, it appears md1 is started with only one disk, and it's random which one that is. I am guessing that the boot failure resulted from neither disk being available for md1. The previous mdadm package was needed to make md1 start with both disks, but it seems that's no longer the case. This whole thing smells like a race problem, and it seems irrelevant whether one of the disks is PATA. dbus may be involved, I don't see any other system-level package change since I last tested it 6 days ago. It also looks like that Linux md raid is currently severely broken :-( -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #510990|application/octet-stream |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c29 --- Comment #29 from Neil Brown 2012-11-29 04:00:42 UTC --- That work around feels a lot like a hack. It is little more than adding something like sleep 5 in there. Maybe the problem is async device discovery - a phrase that I think I've heard but don't know the details of. The current approach assumes that all devices have been discovered by the kernel when "udevadm trigger" is run by systemd, and so once 'udevadm settle' has run, all the devices will be visible. But if discovery happen asynchronously (rather than during the module 'init' process) then I cannot see any way to reliably wait for them. And without that, we cannot boot reliably. I'll see if I can find someone to ask about this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=760859 https://bugzilla.novell.com/show_bug.cgi?id=760859#c31 --- Comment #31 from Volker Kuhlmann 2013-03-14 15:26:23 NZDT --- This problem has nothing to do with PATA disks and continues to exist on current hardware. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.