https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c1 kk zhang changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kkzhang@novell.com AssignedTo|bnc-team-screening@forge.pr |puzel@suse.com |ovo.novell.com | --- Comment #1 from kk zhang 2012-03-21 05:12:38 UTC --- Hi，puzel.could you please look at this?I am not sure whether it is right to assign it to you.Feel free to reassign it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c2 --- Comment #2 from Wilfred van Velzen 2012-03-21 15:38:39 UTC --- With the help of https://en.wikipedia.org/wiki/Mdadm, I found the right commands to (re)assemble the raid arrays in the emergency console, in this state:

mdadm --stop /dev/md0 mdadm --stop /dev/md2

mdadm --assemble /dev/md0 /dev/sda1 --run mdadm --add /dev/md0 /dev/sdb1 [repeated for all the arrays]

And wait for the synchronization to finish, before rebooting. Checking this with:

cat /proc/mdstat

So it isn't an unrecoverable state, but still rather inconvenient. Specially if the machine is at a co-location and you have to drive for a X amount of time to go to the location to find out what happend and fix it... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c4 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|wvvelzen@gmail.com | --- Comment #4 from Wilfred van Velzen 2012-03-22 08:58:59 UTC --- (In reply to comment #3)

It sounds like you don't have an mdadm.conf in your initrd. Did you convert this from non-RAID to RAID without rerunning mkinitrd ??

The raid setup was created during installation of opensuse...

could you: cd /tmp zcat /boot/initrd | cpio -idv

and see if "/tmp/etc/mdadm.conf" gets created? If it does, please attach it.

It is: AUTO -all ARRAY /dev/md2 metadata=1.0 name=linux:2 UUID=bf722ec4:89570cf1:4dd162d9:8c34c87b ARRAY /dev/md0 metadata=1.0 name=linux:0 UUID=fd8ef37a:564a9254:72876573:b839300c It's different from the one in /etc though: DEVICE containers partitions ARRAY /dev/md0 UUID=fd8ef37a:564a9254:72876573:b839300c ARRAY /dev/md1 UUID=ada7fa3a:9242009a:cd9f63b7:a3d1b3d3 ARRAY /dev/md2 UUID=bf722ec4:89570cf1:4dd162d9:8c34c87b ARRAY /dev/md3 UUID=a938e1fc:3a39681c:8f8d2c1b:789e6370

If it doesn't please mkinitrd then run the above commands again and see if an mdadm.conf is there.

I did this anyway, but the contents of /tmp/etc/mdadm.conf, remain the same. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c6 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|wvvelzen@gmail.com | --- Comment #6 from Wilfred van Velzen 2012-04-17 07:59:48 UTC --- (In reply to comment #5)

Thanks.

I think the problem might be that /lib/udev/rules.d/64-md-raid.rules

in the initrd is causing problems.

This file doesn't exist on the system: nnn:/etc/udev/rules.d # ls -l total 252 -rw-r--r-- 1 root root 571 Oct 29 19:37 51-lirc.rules -rw-r--r-- 1 root root 13890 Oct 30 07:23 55-hpmud.rules -rw-r--r-- 1 root root 161732 Mar 22 18:11 55-libsane.rules -rw-r--r-- 1 root root 692 Oct 30 07:23 56-hpmud_support.rules -rw-r--r-- 1 root root 46551 Mar 22 18:11 56-sane-backends-autoconfig.rules -rw-r--r-- 1 root root 2016 Oct 29 20:13 70-kpartx.rules -rw-r--r-- 1 root root 532 Dec 15 15:15 70-persistent-cd.rules -rw-r--r-- 1 root root 948 Mar 28 11:09 70-persistent-net.rules -rw-r--r-- 1 root root 182 Oct 29 20:13 71-multipath.rules -rw-r--r-- 1 root root 93 Oct 29 20:09 99-iwlwifi-led.rules

To confirm this I'd like you to create an initrd without this file and see what happens.

I think the easiest might be to edit /lib/mkinitrd/scripts/setup-udev.sh (after taking a copy just to be safe) and remove the line:

64-md-raid.rules \

and the run mkinitrd again.

Could you try that and let me know the result?

So this file isn't included in mkinitrd any way...? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c8 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|wvvelzen@gmail.com | --- Comment #8 from Wilfred van Velzen 2012-04-17 09:03:04 UTC --- (In reply to comment #7)

Look in /lib/udev/rules.d, not /etc/udev/rules.d

Ah, sorry, I should have looked more careful...

(confusing, isn't it... don't look in /run/udev/rules.d whatever you do :-)

It's empty? Anyway... I edited /lib/mkinitrd/scripts/setup-udev.sh, ran mkinitrd. And checked with 'zcat /boot/initrd | cpio -idv', what was in '/tmp/etc/mdadm.conf'. It still contains just the two ARRAY lines for /dev/md2 and /dev/md0, as shown before. Regardless I tried the cold-remove-one-raid-disk test. But I still get to the emergency console, like before, when I start the server in this state. After this when I reboot with the two disks. It boots but the array is in degraded mode: :/proc # cat mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md1 : active raid1 sda2[0] sdb2[2] 522228 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk md3 : active raid1 sda5[0] sdb5[2] 248460152 blocks super 1.0 [2/2] [UU] bitmap: 1/2 pages [4KB], 65536KB chunk md2 : active raid1 sda3[0] 41946040 blocks super 1.0 [2/1] [U_] bitmap: 1/1 pages [4KB], 65536KB chunk md0 : active (auto-read-only) raid1 sda1[0] sdb1[2] 2095092 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none> I don't think this happened before, but this is easily fixed, with some mdadm commands... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c10 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium Status|NEEDINFO |NEW InfoProvider|wvvelzen@gmail.com | --- Comment #10 from Wilfred van Velzen 2012-05-11 11:05:28 UTC --- (In reply to comment #9)

And you checked that "64-md-raid.rules" wasn't in the initrd in any directory?

Yep.

Very strange.

Would it be possible for me to get a copy of your initrd? It might be too big to attach, in which case you need to find somewhere to upload it that I could grab it from.

http://www.sercom.nl/tmp/initrd.zip -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c12 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |wvvelzen@gmail.com --- Comment #12 from Neil Brown 2012-05-21 04:37:22 UTC --- Thanks... still cannot find any smoking guns. I wonder if the fact that md2 turned up degraded after your last attempt is significant. It could suggest that removing 64-md-raid.rules did make an important difference, but then some other problem interfered with successful boot. It certainly does seem to imply the array array was assembled with only one device and then accessed. Could you try again with one disk missing and get another photo of the emergency console screen? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c14 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|wvvelzen@gmail.com | --- Comment #14 from Wilfred van Velzen 2012-05-21 13:19:15 UTC --- (In reply to comment #12)

Could you try again with one disk missing and get another photo of the emergency console screen?

I did a 'zypper up', did a reboot, and made sure the state of the array was ok. It was: /proc # cat mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md3 : active raid1 sda5[0] sdb5[2] 248460152 blocks super 1.0 [2/2] [UU] bitmap: 0/2 pages [0KB], 65536KB chunk md1 : active raid1 sda2[0] sdb2[2] 522228 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk md2 : active raid1 sda3[0] sdb3[2] 41946040 blocks super 1.0 [2/2] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk md0 : active (auto-read-only) raid1 sda1[0] sdb1[2] 2095092 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none> (Except for the auto-read-only state of the raid that contains the swap partition. But I think this is a different issue) After this I did the test with 1 disk removed. This got me to the emergency mode again (see the new attachement). After adding the disk and rebooting the state of the array again was degraded again, like before, for the root partion: /proc # cat mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md1 : active raid1 sda2[0] sdb2[2] 522228 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk md3 : active raid1 sda5[0] sdb5[2] 248460152 blocks super 1.0 [2/2] [UU] bitmap: 0/2 pages [0KB], 65536KB chunk md2 : active raid1 sda3[0] 41946040 blocks super 1.0 [2/1] [U_] bitmap: 1/1 pages [4KB], 65536KB chunk md0 : active (auto-read-only) raid1 sda1[0] sdb1[2] 2095092 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c16 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |wvvelzen@gmail.com --- Comment #16 from Neil Brown 2012-05-22 04:57:34 UTC --- Could you please install the appropriate rpm from here: https://build.opensuse.org/package/binaries?package=mdadm&project=home%3Aneilbrown%3Abranches%3AopenSUSE%3A12.1%3AUpdate&repository=standard revert the change you made to /lib/mkinitrd/scripts/setup-udev.sh run "mkinitrd", and then see if the problem is fixes? I've added calls to "mdadm -Irs" at appropriate points. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c18 --- Comment #18 from Neil Brown 2012-05-23 12:16:59 UTC --- Created an attachment (id=492101) --> (http://bugzilla.novell.com/attachment.cgi?id=492101) rpm to test Sorry, I thought it was public.. I wonder how I make it public. Anyway, hopefully attaching it here will work. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c20 Christian Boltz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-beta@cboltz.de --- Comment #20 from Christian Boltz 2012-05-23 20:12:04 CEST --- (In reply to comment #18)

Sorry, I thought it was public.. I wonder how I make it public.

AFAIK, home:*:branches:* have the publish flag disabled by default. Enable it in the project config - or just use osc getbinaries to download the RPMs. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c22 Wilfred van Velzen changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|wvvelzen@gmail.com | --- Comment #22 from Wilfred van Velzen 2012-05-24 12:16:51 UTC --- (In reply to comment #21)

Thanks for testing and the screen shot. I think we are getting there.

I suspect that if you switched back to sysvinit now it would work.

I didn't try that.

The remaining problem is an interaction with systemd.

[...]

If this works as expected I'll ask a maintenance update with these fixes.

This worked as expected. The server now booted fine with the one disk removed!

Thanks for your help in isolating this (and sorry that it has taken 2 month!).

No problem, I'm used to much worse response time in bugzilla. :-/ This wasn't even a bug you will notice during normal operation. And I'm happy to help improve openSUSE! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c24 --- Comment #24 from Bernhard Wiedemann 2012-06-05 08:00:09 CEST --- This is an autogenerated message for OBS integration: This bug (752869) was mentioned in https://build.opensuse.org/request/show/123595 Factory / mdadm -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=752869 https://bugzilla.novell.com/show_bug.cgi?id=752869#c26 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #26 from Neil Brown 2012-06-05 07:53:28 UTC --- Thanks. Update submitted. Closing bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.