https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c Christian Boltz changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #516596|application/octet-stream |text/plain mime type| | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c2 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |INVALID --- Comment #2 from Neil Brown 2012-12-12 05:27:16 UTC --- If you are referring to md3 then the logs suggest: [ 13.478064] md: bind<sda6> [ 13.478999] md: kicking non-fresh sdb6 from array! [ 13.479017] md: unbind<sdb6> [ 13.481045] md: export_rdev(sdb6) [ 13.483721] md/raid1:md3: active with 1 out of 2 mirrors that sdb6 had failed previously and was not included for that reason - perfectly correct. Why it failed I cannot tell without /var/log/messages. So it all appears to be working correctly -> invalid. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c4 Christian Boltz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|VERIFIED |REOPENED Resolution|INVALID | --- Comment #4 from Christian Boltz 2013-01-10 18:16:39 CET --- Happened again (Factory from 2012-12-25, mdadm-3.2.6-2.2.x86_64) - and this time I have logs :-) The relevant part from /var/log/messages are (from booting): 2013-01-09T12:08:18.137440+01:00 geeko kernel: [ 14.418724] md: md1 stopped. 2013-01-09T12:08:18.137442+01:00 geeko kernel: [ 14.608502] md: bind<sdb3> 2013-01-09T12:08:18.137444+01:00 geeko kernel: [ 14.608549] md: could not open unknown-block(8,3). 2013-01-09T12:08:18.137446+01:00 geeko kernel: [ 14.608552] md: md_import_device returned -16 2013-01-09T12:08:18.137448+01:00 geeko kernel: [ 14.638234] md/raid1:md1: active with 1 out of 2 mirrors 2013-01-09T12:08:18.137449+01:00 geeko kernel: [ 14.638664] created bitmap (254 pages) for device md1 2013-01-09T12:08:18.137451+01:00 geeko kernel: [ 14.646312] md1: bitmap initialized from disk: read 16 pages, set 0 of 520167 bits 2013-01-09T12:08:18.137456+01:00 geeko kernel: [ 14.666434] md1: detected capacity change from 0 to 136358617088 2013-01-09T12:08:18.137459+01:00 geeko kernel: [ 14.668951] md1: unknown partition table On the previous boot, md1 was assembled with both disks, and the log does not contain anything that could explain why it broke. BTW: Do you need something from the missing disk or can I re-integrate it into the array? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c6 --- Comment #6 from Christian Boltz 2013-02-09 19:12:19 CET --- (In reply to comment #5)

Hopefully you didn't wait for me and have already re-integrated the missing disk into the array. If not, feel free to do so.

I'm just doing it _again_ :-( - this time sdb3 was missing from md1 (are you interested in fresh logs? - note to myself: 2013-02-09T11:39:33) Besides your ideas what could go wrong, would an unclear umount/shutdown be an explanation for what I'm seeing? md1 is my /home partition and also contains /var (/var is a symlink to /home/sys-var/). I'm quite sure I've seen umounting md1 failing at shutdown because it's still in use, but I don't know if this happens always or only sometimes (and it's probably too late to be logged). BTW: The resync is usually extremely fast (120 G in 2 minutes) - is there a "list of changed sectors" that avoids a full resync? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c8 --- Comment #8 from Neil Brown 2013-02-19 03:42:24 UTC --- Created an attachment (id=525153) --> (http://bugzilla.novell.com/attachment.cgi?id=525153) alternate mdadm to test I looked at the code again, and "unknown-block()" is exactly what is expected, so that doesn't tell us anything. So it is simply an EBUSY problem, meaning a race with mdadm somehow. Closing the race doesn't seem easy without a lot of code change (which will happen in the next upstream release, but isn't suitable for openSUSE yet), but retrying should be just as successful and a lot easier. So I've modified the code to retry if it gets EBUSY. It only retries 5 times, but that should be plenty. Could you please install that attached rpm: rpm -U mdadm..... and see if the problem recurs. I suspect we'll have to wait several weeks before we can be sure it is fixed... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c10 --- Comment #10 from Christian Boltz 2013-02-20 21:44:08 CET --- BTW: Does your patched test package log anything if it retries because of EBUSY? If not, please add it - grepping for something like "EBUSY - retrying $device" is easier than just waiting for a random bug not to appear ;-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c12 --- Comment #12 from Christian Boltz 2013-02-26 21:03:45 CET --- Just happened again :-( The previous boot looks good (I'm logging the content of /proc/mdstat from boot.local): 2013-02-25T17:48:01.742859+01:00 geeko boot.local[3338]: md1 : active raid1 sda3[0] sdb3[2] 2013-02-25T17:48:01.743058+01:00 geeko boot.local[3338]: 133162712 blocks super 1.0 [2/2] [UU] and there's also no sign of problems in the log later. FYI: I grepped for "md1|sda3|sdb3" + context lines On the next boot, only sda3 was used for md1: 2013-02-26T12:02:46.684946+01:00 geeko kernel: [ 1.704779] sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 > 2013-02-26T12:02:46.684948+01:00 geeko kernel: [ 1.705234] sd 0:0:0:0: [sda] Attached SCSI disk 2013-02-26T12:02:46.684950+01:00 geeko kernel: [ 1.706459] sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 sdb7 > 2013-02-26T12:02:46.684952+01:00 geeko kernel: [ 1.706884] sd 1:0:0:0: [sdb] Attached SCSI disk [...] 2013-02-26T12:02:46.685507+01:00 geeko kernel: [ 17.016099] md: md1 stopped. 2013-02-26T12:02:46.685510+01:00 geeko kernel: [ 17.018210] md: bind<sda3> 2013-02-26T12:02:46.685512+01:00 geeko kernel: [ 17.044953] md/raid1:md1: active with 1 out of 2 mirrors 2013-02-26T12:02:46.685517+01:00 geeko kernel: [ 17.045083] created bitmap (254 pages) for device md1 2013-02-26T12:02:46.685520+01:00 geeko kernel: [ 17.052109] md1: bitmap initialized from disk: read 16 pages, set 0 of 520167 bits 2013-02-26T12:02:46.685522+01:00 geeko kernel: [ 17.134417] md1: detected capacity change from 0 to 136358617088 2013-02-26T12:02:46.685524+01:00 geeko kernel: [ 17.335114] md1: unknown partition table No sign of an error message :-( but the line "bind<sdb3>" is missing. This still leaves the question why sdb3 was not added - any idea where you could add/log another error message? ;-) Another boot later, I get an error message (as expected, because sdb3 is outdated in the meantime): 2013-02-26T16:54:09.493860+01:00 geeko kernel: [ 11.950498] md: bind<sda3> 2013-02-26T16:54:09.493865+01:00 geeko kernel: [ 11.951345] md: kicking non-fresh sdb3 from array! 2013-02-26T16:54:09.493867+01:00 geeko kernel: [ 11.951353] md: unbind<sdb3> 2013-02-26T16:54:09.493869+01:00 geeko kernel: [ 11.952442] md: export_rdev(sdb3) 2013-02-26T16:54:09.493871+01:00 geeko kernel: [ 11.953904] md/raid1:md1: active with 1 out of 2 mirrors -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c14 Christian Boltz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|suse-beta@cboltz.de | --- Comment #14 from Christian Boltz 2013-02-27 13:38:11 CET --- (In reply to comment #13)

Could you add "-vv" to the mdadm command in /etc/init.d/boot.md

Done.

Am I correct in thinking that this never happens to the "root" array, only to other arrays what would be assembled once the system has booted?

IIRC you are correct (but I'm not 100% sure). When first repoting the problem, it affected /boot and /testroot, but in the last months typically my encrypted /home (which also contains /var via symlinks) breaks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c16 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #526231|0 |1 is obsolete| | --- Comment #16 from Neil Brown 2013-03-03 23:55:44 UTC --- Created an attachment (id=527872) --> (http://bugzilla.novell.com/attachment.cgi?id=527872) new rpm with extra EBUSY loop I missed a spot where it can fail with EBUSY. please "rpm -U" this rpm and try again. Thanks for your patience. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c18 --- Comment #18 from Christian Boltz 2013-03-08 01:12:34 CET --- With the latest package, I get several boot.md[335]: EBUSY - retrying dev_open of /dev/sda5 boot.md[335]: EBUSY - retrying dev_open of /dev/sdb5 which is not really surprising - sd*5 is part of md2, which is my root partition and already mounted by initrd. (Nevertheless, wouldn't it make sense to skip checking partitions that are already in use by an active raid array?) It also looks like your changes prevented a failure: Mar 05 12:13:33 geeko.boltz boot.md[316]: mdadm: looking for devices for /dev/md1 [...] Mar 05 12:13:35 geeko.boltz boot.md[316]: EBUSY - retrying dev_open of /dev/sdb3 [...] Mar 05 12:13:36 geeko.boltz boot.md[316]: mdadm: /dev/sdb3 is identified as a member of /dev/md1, slot 1. Mar 05 12:13:36 geeko.boltz boot.md[316]: mdadm: /dev/sda3 is identified as a member of /dev/md1, slot 0. Mar 05 12:13:36 geeko.boltz boot.md[316]: mdadm: added /dev/sdb3 to /dev/md1 as 1 Mar 05 12:13:36 geeko.boltz boot.md[316]: mdadm: added /dev/sda3 to /dev/md1 as 0 Mar 05 12:13:37 geeko.boltz boot.md[316]: mdadm: /dev/md1 has been started with 2 drives. and also helped two days later (Mar 07 12:14:51) with similar log messages. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c20 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jslaby@suse.com --- Comment #20 from Neil Brown 2013-03-12 02:46:31 UTC --- *** Bug 805415 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=805415 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c22 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |volker3204@paradise.net.nz --- Comment #22 from Neil Brown 2013-03-13 04:06:31 UTC --- *** Bug 760859 has been marked as a duplicate of this bug. *** http://bugzilla.novell.com/show_bug.cgi?id=760859 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c24 --- Comment #24 from Volker Kuhlmann 2013-03-14 15:33:27 NZDT --- I've just had a major filesystem corruption because of this. Back in SuSE 9.2 Linux used to just work... Anyway it's essential to re-attach the missing disks and waiting for the sync to finish before rebooting, or otherwise next time the *other* disk might be the sole array and it's up the creek very quickly. My mistake was to not notice the degraded array for a week. all packagea are current (oS 12.1, but same problem). I've put some more logging into place, and the symptoms are always that the md1 raid1 array (md0 for the root fs is never a problem) is randonmly created with one disk, the other disk, both disks, or no disk. In the last case there's a huge timeout (for mounting the non-existing disk) before entering emergency mode. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c26 --- Comment #26 from Volker Kuhlmann 2013-03-14 15:41:13 NZDT --- Created an attachment (id=529614) --> (http://bugzilla.novell.com/attachment.cgi?id=529614) init script 1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c28 --- Comment #28 from Volker Kuhlmann 2013-03-14 15:42:42 NZDT --- Created an attachment (id=529616) --> (http://bugzilla.novell.com/attachment.cgi?id=529616) sysconfig addition -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c30 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |fcrozat@suse.com --- Comment #30 from Neil Brown 2013-03-14 07:10:36 UTC --- Ahh.. I might have it at last This bug is very similar to bug 772286 which we fixed by adding # Should-Start: udev-trigger to /etc/init.d/boot.md. However it seems that udev-trigger isn't a service any more. Rather there is a systemd-udev-trigger.service So maybe we just need to change "udev-trigger" in boot.md to "systemd-udev-trigger". A bit of experimentation suggests that changing # Should-Start: boot.scsidev boot.multipath udev-trigger in /etc/init.d/boot.md to # Should-Start: boot.scsidev boot.multipath systemd-udev-trigger does seem to change the behaviour as I would expect. I don't have any problems with assembling arrays (as the problem is very sensitive to particular timing of various events) so I cannot be certain that this fixes the problem. If someone who does experience the problem could make this change and see if it fixes the problem, that would be great. Frederic: apart from wanting to let you know that I'm a bit grumpy about this name change (not your fault exactly, but I wanted to complain to someone) I notice that the /usr/lib/systemd/system/systemd-udev-trigger.service file looks wrong based on the man page (and a quick look at the code). It contains: ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger --type=devices --action=add However the body of "ExecStart" is not a shell script, but a command and some args, so this passes ";" and "/usr/bin/udevadm" etc as extra args to /usr/bin/udevadm. For "Type=oneshot" services you are allowed multiple ExecStart, so this should be ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ExecStart=/usr/bin/udevadm trigger --type=devices --action=add This change might also be needed for md arrays to be started properly but I'm not 100% sure. The upstream code has this bug. Would you follow it up with upstream, or would you rather that I did? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c32 --- Comment #32 from Volker Kuhlmann 2013-03-14 23:17:20 NZDT --- Re comment #30 Unfortunately adding udev-trigger to # Should-Start: udev-trigger DOES NOT fix the problem. My older SATA/PATA system still has the line ENV{ID_FS_TYPE}=="linux_raid_member", ACTION=="add", RUN+="/sbin/mdadm --incremental $env{DEVNAME}" in /lib/udev/rules.d/64-md-raid.rules commented out. Otherwise it's up to all updates of 12.1, and it seems reliable. My system with 2 SATA has all updates, no changes, and raid1 is @*^$ed (still 12.1 too). My additional scripts may look excessive but have the deciding advantage that they are guaranteed to work. Their warning features will also be necessary for any future tests (I don't want to get screwed again). Of about 8 reboots for testing today the problem showed in the first 3 and then disappeared. I hate that. Is there anything that could adapt to a changed system configuration (straws, grasp)? Over the next few weeks I expect to upgrade to 12.3, until then I can test the change -ExecStart=/sbin/udevadm trigger --type=subsystems --action=add ; /sbin/udevadm trigger --type=devices --action=add +ExecStart=/sbin/udevadm trigger --type=subsystems --action=add +ExecStart=/sbin/udevadm trigger --type=devices --action=add in /lib/systemd/system/udev-trigger.service -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c34 --- Comment #34 from Frederic Crozat 2013-03-14 12:05:17 UTC --- (In reply to comment #33)

Btw is it possible to get the boot console(!) log into a file? Setting ENFORCE_BLOGD="yes" in sysconfig/boot does nothing. There is no boot.msg file.

Use journalctl -b to get all the logs from current boot.

So it looks a bit like you want to get that broken udev file (and its name in boot.md) fixed in 12.3 yesterday...

We'll ship a maintenance update on relevant distributions once we are sure all the fixes are tested, of course :) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c36 --- Comment #36 from Frederic Crozat 2013-03-15 08:30:05 UTC --- (In reply to comment #35)

journalctl: Command not found.

sorry, I forgot you are on 12.2, use systemd-journalctl instead.

Everyone's interested in the latest, but I have to test this on oS 12.1 for the time being. (Do I put this down to yet another case of half-baked banana software like systemd being let lose on the guinee pigs ooops opensuse users...??? :-)

These kind of comments won't really help motivating people to help you.. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c38 --- Comment #38 from Frederic Crozat 2013-03-15 10:28:13 UTC --- This bug is mixing a lot of various versions. For 12.1, there is no journal. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c40 --- Comment #40 from Frederic Crozat 2013-03-15 16:24:00 UTC --- After more tests, there is no bug in systemd, ExecStart should correctly handle ";". But if at one moment "systemctl daemon-reload" is run, the status output might become incomplete, which might give the impression only one process was launched : systemd-udev-trigger.service - udev Coldplug all Devices Loaded: loaded (/usr/lib/systemd/system/systemd-udev-trigger.service; static) Active: active (exited) since Fri, 2013-03-15 12:48:43 CET; 4h 29min ago Docs: man:udev(7) man:systemd-udevd.service(8) Process: 308 ExecStart=/usr/bin/udevadm trigger --type=devices --action=add (code=exited, status=0/SUCCESS) Process: 306 ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/systemd-udev-trigger.service would become : systemd-udev-trigger.service - udev Coldplug all Devices Loaded: loaded (/usr/lib/systemd/system/systemd-udev-trigger.service; static) Active: active (exited) since Fri, 2013-03-15 12:48:43 CET; 4h 29min ago Docs: man:udev(7) man:systemd-udevd.service(8) Main PID: 308 (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/systemd-udev-trigger.service -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c42 --- Comment #42 from Volker Kuhlmann 2013-04-05 10:09:23 NZDT --- Thanks Neil, much appreciated. I added that strace, with an -f as well. The output isn't very long. Of 4 reboots, 1 was OK, 1 was a total failure, 2 were degraded. My diff, the strace output, and the md status are in files in the attached tar file. If you'd like me to run more and different tests no problem, and I'll be a bit faster. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c44 --- Comment #44 from Neil Brown 2013-04-10 01:26:49 UTC --- Thanks Volker. Unfortunately it doesn't get me much closer. "udevadm settle" seems to be behaving exactly correctly. Yet something related to it isn't working. So I'm still stuck. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c46 --- Comment #46 from Volker Kuhlmann 2013-04-12 17:18:03 NZST --- It occurred to me that stracing the mdadm run by boot.md would give only a small part of the picture, and logging *ALL* invocations of mdadm including location and arguments would be more interesting. So I put a wrapper script around /sbin/mdadm. The problem appears to be a race between udev and boot.md. The -IRs and -A -s... are run by boot.md, the others by udev. This gives a degraded array: + : mdadm --detail --export /dev/md0 + : mdadm -IRs + : mdadm --examine --export /dev/sdb5 + : mdadm --examine --export /dev/sda5 + : mdadm --examine --export /dev/sda3 + : mdadm --examine --export /dev/sdb3 + : mdadm -A -s -c /etc/mdadm.conf + : mdadm --incremental /dev/sdb3 + : mdadm --incremental /dev/sdb5 + : mdadm --incremental /dev/sda5 + : mdadm --incremental /dev/sda3 This works fine: + : mdadm --detail --export /dev/md0 + : mdadm -IRs + : mdadm -A -s -c /etc/mdadm.conf + : mdadm --examine --export /dev/sdb3 + : mdadm --examine --export /dev/sda3 + : mdadm --examine --export /dev/sdb5 + : mdadm --examine --export /dev/sda5 + : mdadm --incremental /dev/sdb3 + : mdadm --incremental /dev/sdb5 + : mdadm --incremental /dev/sda3 + : mdadm --incremental /dev/sda5 So udev stoidly examines all partitions and then runs incremental. Meanwhile boot.md has already run incremental and at some point assembles the arrays. What I don't understand is why both of them are doing the same job. How much do they interfere with each other? And where/when? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c48 --- Comment #48 from Volker Kuhlmann 2013-04-12 17:22:03 NZST --- Created an attachment (id=534871) --> (http://bugzilla.novell.com/attachment.cgi?id=534871) reboot 2 ok Remember that I am running boot.md-check and boot.md-retry after boot.md so I get a reliable system (except when it sinkholes itself into emergency mode because md1 is running with 0/2). My boot.md still should-starts udev-trigger because it's 12.1. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c50 --- Comment #50 from Neil Brown 2013-04-29 06:05:46 UTC --- Hi Volker. Your latest details seem to just confirm that udev is doing thing when I believe that it shouldn't. The call to "udevadm settle" earlier in boot.md should wait for all those udev events to happen. It very much seems like it does. Only they still come, which seems to suggest that something else is triggering new 'add' events, though I cannot imaging what. Could you try something new for me? in boot.md, add the line /sbin/udevadm control --stop-exec-queue before $mdadm_BIN -IRs $mdadm_BIN -A -s -c $mdadm_CONFIG # a status of 2 is not an error test $? -eq 0 -o $? -eq 2 rc_status -v and add /sbin/udevadm control --start-exec-queue just after that section. This should ensure that udev and boot.md don't race. It might slow boot by 5 seconds but that might be better than races. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c52 --- Comment #52 from Volker Kuhlmann 2013-05-03 18:58:00 NZST --- Created an attachment (id=537803) --> (http://bugzilla.novell.com/attachment.cgi?id=537803) Log files of reboot 2 (comment #51) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c54 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |volker3204@paradise.net.nz --- Comment #54 from Neil Brown 2013-05-14 04:40:19 UTC --- Thanks for all the logs. It seems that the events for all the sda devices are starting *after* the "udevadm settle" call. The dependency on udev-trigger should prevent that. Could you try something else? In boot.md, before the "udevadm settle" call, call udevadm control --log-priority=debug udevadm trigger --sysname-match='sd[ab]*' That should trigger just the devices we want, and not waste time triggering everything else. It should also put lots of 'udevd' messages in the /var/log/messages file, which might be useful. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c56 --- Comment #56 from Volker Kuhlmann 2013-06-04 04:55:19 NZST --- Created an attachment (id=542405) --> (http://bugzilla.novell.com/attachment.cgi?id=542405) Log files for comment #55 README-3 with detailsa bout the log files, boot.md script used with diff from original, syslog, mdadm traces. /var/log/messages had all hylafax-related lines removed (they're long, and some of it is private) but is otherwise complete. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c58 Christian Boltz changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|WORKSFORME | --- Comment #58 from Christian Boltz 2013-06-11 07:03:29 CEST --- Please allow me to disagree ;-) This happened to me me on several 12.3 machines (with very different hardware), so I'd guess it's quite likely to hit more people. In other words: please create a maintenance update for 12.3. The package from comment #16 fixes the problem for me - maybe you can just use this package (+ the initscript changes that helped Volker) for the 12.3 update. BTW: Is this fixed in factory? I still have a lock on mdadm, but I don't want to keep it forever ;-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c60 --- Comment #60 from Neil Brown 2013-06-12 22:34:50 UTC --- I've submitted update requests for Factory and 12.3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=793954 https://bugzilla.novell.com/show_bug.cgi?id=793954#c62 Benjamin Brunner changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #62 from Benjamin Brunner 2013-06-24 08:37:00 CEST --- Update released for openSUSE 12.3. Resolved fixed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.