https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c A R changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugzilla.novell.com | |/show_bug.cgi?id=849752 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c A R changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|boot failure after 12.3 -> |boot failure after 12.3 -> |13.1 upgrade if /boot is on |13.1 upgrade if /boot is on |RAID |RAID; mdadm stack? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c3 --- Comment #3 from A R 2013-11-24 06:09:01 UTC --- @Andrey I can add the systemd.log_level=debug parameter. I can't actually enter emergency mode during a failed boot process; I see the msg advising to enter emergency mode, but can't get to it. I can grab serial console output, if that's sufficient. Is it? And/or, I can boot, fail, reboot to a resuce disk, chroot to this env, and perhaps get at the system.journal that way. What's preferred? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c5 --- Comment #5 from A R 2013-11-24 15:52:37 UTC ---

Could you attach /usr/lib/systemd/system/emergency.service

cat /usr/lib/systemd/system/emergency.service # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Emergency Shell Documentation=man:sulogin(8) DefaultDependencies=no Conflicts=shutdown.target Before=shutdown.target [Service] Environment=HOME=/root WorkingDirectory=/root ExecStartPre=-/bin/plymouth quit ExecStartPre=-/bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\\nsystem logs, "systemctl reboot" to reboot, "systemctl default" to try again\\nto boot into default mode.' ExecStart=-/usr/sbin/sulogin ExecStopPost=/usr/bin/systemctl --fail --no-block default Type=idle StandardInput=tty-force StandardOutput=inherit StandardError=inherit KillMode=process IgnoreSIGPIPE=no SendSIGHUP=yes cat /usr/lib/systemd/system/emergency.target # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Emergency Mode Documentation=man:systemd.special(7) Requires=emergency.service After=emergency.service AllowIsolate=yes

...

What's preferred? Well, journal may hold some extra output, so would be better to get it.

Here's the boot config title openSUSE SERIAL CONSOLE root (hd0,0) kernel /vmlinuz root=/dev/VG_SVR/LV_SVR_DOM0_ROOT rootfstype=ext4 noresume showopts ide=nodma apm=off edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 x11failsafe vga=0x31a console=tty0 console=ttyS0,57600n8 nomodeset debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug systemd.log_level=debug systemd.log_target=syslog-or-kmsg initrd /initrd

journalctl -D /path/to/var/log/journal -b.

When I exec journalctl -D /var/log/journal/e3d557b51266cc26571ec214493ee60e -b I just get, -- Logs begin at Fri 2013-11-22 12:07:59 PST, end at Sat 2013-11-23 23:27:24 PST. -- Instead, I've captured the output of journalctl -D /var/log/journal/e3d557b51266cc26571ec214493ee60e/system.journal it's here, in 2 parts: part1: http://pastebin.com/m5cCQ32G part2: http://pastebin.com/F9FEqg65 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c7 --- Comment #7 from A R 2013-11-24 17:24:29 UTC --- @Andrey as you suspected, after rm /path/to/.../syslog.service reboot I can now enter emergency mode @ prompt. once there, journalctl -xb -D /var/log/journal/<...> returns: http://pastebin.com/Qp0M0sTh -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c9 --- Comment #9 from A R 2013-11-24 18:01:16 UTC --- @Andrey

I opened bnc#852021 for rsyslog issue.

thanks.

... you can attach output of ...

udevadm info --export-db --> http://pastebin.com/ch9zDjJh cat /etc/fstab cat /proc/mdstat --> http://pastebin.com/fniwz0K1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c11 --- Comment #11 from A R 2013-11-24 18:29:59 UTC ---

Could you also attach /etc/mdadm.conf?

cat /etc/mdadm.conf --> http://pastebin.com/FkZqd7wg -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c13 --- Comment #13 from A R 2013-11-25 02:16:50 UTC --- with existing mdadm.comf Boot to kernel-default, FAIL Boot to kernel-xen, Xen Dom0 FAIL Edit in mdadm.conf - DEVICE ... + #DEVICE ... Boot to kernel-default, SUCCESS Boot to kernel-xen, Xen Dom0 SUCCESS (Launching Xen DomU Guest still has issues, but that's a different problem I suspsect) So, "DEVICE ..." is problematic. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c A R changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|boot failure after 12.3 -> |Opensuse 13.1 boot fails if |13.1 upgrade if /boot is on |'/boot' is on RAID & |RAID; mdadm stack? |mdadm.conf IDs devices | |using "DEVICE ..." keyword -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c16 --- Comment #16 from Neil Brown 2013-12-03 03:10:41 UTC --- Created an attachment (id=569893) --> (http://bugzilla.novell.com/attachment.cgi?id=569893) mdadm RPM for testing This RPM contains enhancements to mdadm and changed to the udev script so that the DEVICE line should be honoured correctly. It also contains the fix for bug 832501. Please test and confirm. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c18 A R changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|ar16@imapmail.org | --- Comment #18 from A R 2013-12-03 04:23:43 UTC --- rpm -qa | grep -i mdadm mdadm-3.3-4.1.2.x86_64 rpm -Uvh --force bug-851993_mdadm-3.3-0.x86_64.rpm vi /etc/mdadm.conf CREATE owner=root group=disk mode=0660 auto=yes HOMEHOST <ignore> ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=ec1... ARRAY /dev/md1 level=raid1 num-devices=2 metadata=1.00 name=<none>:server_root UUID=a1b... ARRAY /dev/md2 level=raid10 num-devices=4 metadata=1.02 name=<none>:server_nas UUID=a0c... shutdown -r now ... uname -a Linux xen01 3.11.6-4-xen #1 SMP Wed Oct 30 18:04:56 UTC 2013 (e6d4a27) x86_64 x86_64 x86_64 GNU/Linux vi /etc/mdadm.conf + DEVICE /dev/disk/by-id/ata-ST3250410AS_9RY1133S-part2 /dev/disk/by-id/ata-ST3250410AS_9RY1127T-part2 + DEVICE /dev/disk/by-id/ata-ST3250410AS_9RY1133S-part1 /dev/disk/by-id/ata-ST3250410AS_9RY1127T-part1 + DEVICE /dev/disk/by-id/ata-SAMSUNG_HD103UJ_S14CD3DT541620-part1 /dev/disk/by-id/ata-SAMSUNG_HD103UJ_S14CD3DT541639-part1 /dev/disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMADR7500124-part1 /dev/disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMADR7975412-part1 CREATE owner=root group=disk mode=0660 auto=yes HOMEHOST <ignore> ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=ec1... ARRAY /dev/md1 level=raid1 num-devices=2 metadata=1.00 name=<none>:server_root UUID=a1b... ARRAY /dev/md2 level=raid10 num-devices=4 metadata=1.02 name=<none>:server_nas UUID=a0c... shutdown -r now ... uname -a Linux xen01 3.11.6-4-xen #1 SMP Wed Oct 30 18:04:56 UTC 2013 (e6d4a27) x86_64 x86_64 x86_64 GNU/Linux cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [linear] md0 : active raid1 sdb1[1] sda1[0] 200704 blocks [2/2] [UU] bitmap: 0/25 pages [0KB], 4KB chunk md2 : active raid10 sdf1[3] sde1[2] sdc1[0] sdd1[1] 1953519616 blocks super 1.2 512K chunks 2 far-copies [4/4] [UUUU] bitmap: 1/466 pages [4KB], 2048KB chunk md125 : active raid1 sdb2[2] sda2[0] 243987048 blocks super 1.0 [2/2] [UU] With your patch, and "DEVICE ..." reinstated in /etc/mdadm.conf, boot's OK & the arrays are correctly assembled & up. Raid device names are correctly pulled from /etc/mdadm.conf for /dev/md0 & /dev/md2, but ignored for /dev/md1 -- which ends up as /dev/md125. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c20 A R changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|ar16@imapmail.org | --- Comment #20 from A R 2013-12-03 04:55:54 UTC ---

It looks like md1 holds your root filesystem - correct?

yes md0 <- "/boot" md1 <- "/" md2 <- additional storage

What does

zcat /boot/initrd | cpio -i --to-stdout etc/mdadm.conf

report? Please double check it gives the same uuid as mdadm -Db /dev/md125

Same UUID returned for both zcat /boot/initrd | cpio -i --to-stdout etc/mdadm.conf AUTO -all ARRAY /dev/md125 metadata=1.0 name=<none>:server_root UUID=a1b...c3d 61591 blocks and mdadm -Db /dev/md125 ARRAY /dev/md125 metadata=1.0 name=<none>:server_root UUID=a1b...c3d -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c22 --- Comment #22 from A R 2013-12-03 06:16:23 UTC ---

So the array is getting called md125 because that is what the mdadm.conf on the initrd say, and the initrd say that because that is what it is currently called.

Catch22, then. But, apparently, not a suprise.

I guess it would be nice to copy the name out of /etc/mdadm.conf onto the initrd. Though given that the name is "server_root", you really want /dev/md_server_root, but that is a little way off working perfectly yet.

Ideally, 'it' does what you 'tell it' to do -- in /etc/mdadm.conf. It's not a big deal, but does cause some head-scratching ...

I don't think I'm comfortable getting the mkintrd to automatically change the name of the root md device as it is too error prone.

Perhaps, semi-auto, then? I.e., a cmd-line flag/option to mdadm that writes the /dev/<name> as spec'd in /etc/mdadm.conf to the mkinitrd?

You could do it manually by: ...

Thanks, noted.

6/ edit the command line to change "root=/dev/md125" to "root=/dev/md0"

just fyi, my root fs is on LVM. grub contains title Xen root (hd0,0) kernel /xen.gz ... module /vmlinuz-xen root=/dev/VG_SVR/LV_SVR_DOM0_ROOT ... module /initrd-xen i.e., NOT "root=/dev/md125". Not sure if LVM mount 'vs' MD assembly is of any issue -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c24 --- Comment #24 from Bernhard Wiedemann 2013-12-05 02:00:22 CET --- This is an autogenerated message for OBS integration: This bug (851993) was mentioned in https://build.opensuse.org/request/show/209450 Factory / mdadm -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c26 Matjaz Godec changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |matjaz.godec@agenda.si Resolution|FIXED | --- Comment #26 from Matjaz Godec 2013-12-27 19:25:47 UTC --- Hello, I've just found similar problem also with updated mdadm (so reopening this BUG report): We have raid0 over raid1 config. Raid0 is /dev/md/8 and it uses md5, md6 and md7 raid1 devices. At boot in 13.1 latest update all raid devices are assembled but md8. In rescue boot, all devices including md8 is correctly assembled. After removing fstab entry and bootinf into the system mdadm --aassemble /dev/md8 works OK, and also lvchange -ay data (LVM on md8 device) is created OK. Our mdadm.conf: DEVICE partitions ARRAY /dev/md/0 metadata=1.0 UUID=4a77b066:af69c365:8e81d105:53cde38b name=linux:0 ARRAY /dev/md/1 metadata=1.0 UUID=f9f70990:55bb2040:8d623e5e:61ff1918 name=linux:1 ARRAY /dev/md/2 metadata=1.0 UUID=ed19ab42:e814ac8d:2b40b865:6e12ab0d name=linux:2 ARRAY /dev/md/4 metadata=1.0 UUID=d1a4ba2b:ecb8142f:506a0bd8:96b0f27a name=xenbackup:4 ARRAY /dev/md/7 metadata=1.0 UUID=e217d13e:e397d0e3:4214de7b:874babc2 name=xenbackup:7 ARRAY /dev/md/3 metadata=1.0 UUID=370c9e5e:859138e5:59b14374:168e3b44 name=xenbackup:3 ARRAY /dev/md/6 metadata=1.0 UUID=56deaee4:c6f0b43b:5fe71825:259403d6 name=xenbackup:6 ARRAY /dev/md/5 metadata=1.0 UUID=06ed4fd4:655320dc:ec4e383b:c1b0b60d name=xenbackup:5 ARRAY /dev/md/8 metadata=1.0 UUID=549a0648:fc3c3076:d3220322:cdaa8d31 name=xenbackup:8 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c28 --- Comment #28 from Neil Brown 2014-01-21 04:22:17 UTC --- Hi Matjaz, could you please try editing /usr/lib/udev/rules.d/64-md-raid-assembly.rules and change ACTION=="add", RUN+="/sbin/mdadm --incremental $devnode --offroot" to ACTION=="add|change", RUN+="/sbin/mdadm --incremental $devnode --offroot" i.e. add '|change' to the 'ACTION==' section. and let me know if that helps. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c30 --- Comment #30 from Stephan Brunsch 2014-01-21 08:29:20 UTC --- I created a mdmadm.conf in the initrd but it did not help chrooting in md did not help. raid is found but cannot be mounted => missing personality ... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c33 --- Comment #33 from Bernhard Wiedemann 2014-01-23 11:00:24 CET --- This is an autogenerated message for OBS integration: This bug (851993) was mentioned in https://build.opensuse.org/request/show/214760 Factory / mdadm -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c36 Stephan Brunsch changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |REOPENED InfoProvider|stephan.brunsch@berlin.de | --- Comment #36 from Stephan Brunsch 2014-03-22 13:39:54 UTC --- got a mdadm.conf in initrd with forcing root-device md1 mkinitrd -f md -v -d /dev/md1 But booting with this conf stops with error message personality for level 1 not loaded. mdadm.conf from initrd: AUTO -all ARRAY /dev/md1 metadata=0.90 UUID=93fbee3e:b52f6a0c:b61deb77:cbb381cb After normal rebooting, md-device has to be stopped and assembled again to get it mounted. Do I have to add raid1 as modul? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=851993 https://bugzilla.novell.com/show_bug.cgi?id=851993#c38 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- InfoProvider|883f6ce9@opayq.com |stephan.brunsch@berlin.de --- Comment #38 from Neil Brown 2014-04-23 23:53:36 UTC --- Where are we with this? Is it still not working? If so, please attach the initrd. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.