On Mon, Jan 9, 2017 at 4:12 PM, Andrei Borzenkov wrote:

On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor wrote:

...

How can I start md service in Leap 42.2?

MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually. If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"

This is udev.log-priority=debug, sorry.

(and omitting "quiet" to be sure) on kernel command line may provide some hints.

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

09.01.2017 18:07, Istvan Gabor пишет:

On Mon, 9 Jan 2017 16:21:34 +0300, Andrei Borzenkov wrote:

...

On Mon, Jan 9, 2017 at 4:12 PM, Andrei Borzenkov wrote:

...

On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor wrote:

...

How can I start md service in Leap 42.2?

MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually.

I looked at 64-md-raid-assembly.rules file but I don't know how to manually run the commands. I guess this is the command you mean:

/sbin/mdadm --incremental --export $devnode --offroot ${DEVLINKS}

I don't know what to take for $devnode and ${DEVLINKS}.

$devnode is name of device that is being scanned (i.e. array component). $DEVLINKS is list of device aliases.

I found that /dev/disk/by-uuid has only a few devices. It should have much more.

It really sounds like some udev problem.

...

...

If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"

This is udev.log-priority=debug, sorry.

...

(and omitting "quiet" to be sure) on kernel command line may provide some hints.

OK, I booted with this kernel parameter. What to look for?

Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/

And a very strange thing happened. I ran cfdisk. When I exited cfdisk (without writing anything) a lot of md devices have been created but only having one disk of the arrays:

Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.

# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0

md7 : inactive sdb7[3](S) 31455164 blocks super 1.0

unused devices:

/dev/disk/by-uuid has been populated too.

What next?

mdadm --examine --scan -vv in addition to journalctl output would be good. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

09.01.2017 21:08, Per Jessen пишет:

Andrei Borzenkov wrote:

...

Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.

It came up with a really weird config - in the config below, md6 and md7 have three drives each, and the only one present is a hot-spare.

The [3] is not total number of components, but rather unique component identifier. It is also *not* position in the RAID array. This identifier may change e.g. after device replacement. Spare status is correct after "mdadm --incremental" with single component. So far it looks pretty OK except I would expect second half of array to be present as well.

...

...

# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0

md7 : inactive sdb7[3](S) 31455164 blocks super 1.0

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Mon, 09 Jan 2017 20:12:30 +0100, Per Jessen wrote:

Andrei Borzenkov wrote:

...

09.01.2017 21:08, Per Jessen пишет:

...

Andrei Borzenkov wrote:

...

Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.

Sorry, I omitted that I only ran cfdisk /dev/sdb. This has triggered only process of sdb devices, one halves of the arrays. Once more the whole process, step by step: # cat /proc/mdstat cat: /proc/mdstat: No such file or directory # cfdisk /dev/sdb QUIT Many devices are generated. # cat /proc/mdstat Personalities : md7 : inactive sdb7[3](S) 31455164 blocks super 1.0 md6 : inactive sdb6[3](S) 20972752 blocks super 1.0 unused devices: <none> # cfdisk /dev/sdc QUIT Many devices are generated again. # cat /proc/mdstat Personalities : [raid1] md7 : active raid1 sdc7[2] sdb7[3] 31455164 blocks super 1.0 [2/2] [UU] md6 : active raid1 sdc6[2] sdb6[3] 20971520 blocks super 1.0 [2/2] [UU] unused devices: <none> In reality I have 18 arrays, but inserted here only two as examples. Andrei, I sent the required log files to your address. Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

10.01.2017 22:32, Istvan Gabor пишет: ...

...

Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/

I tried to upload the output from journalctl -b to susepaste.org but the site haven't accepted it. I guess it's too much to paste in. The whole log file is ~1 MB and it contains 11678 lines. I have sent the log to your email too, I don't know if you've received it.

I did; as I answered in another mail, udev debug output was not that useful and there is nothing standing out between good and bad cases. Also some logs are definitely lost. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 01/11/2017 03:35 PM, Istvan Gabor wrote:

Hi Andrei,

Was the new log file I sent this morning useful?

I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.

Thanks,

Istvan

Istvan, I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2017-01-12 10:03, Istvan Gabor wrote:

...

I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.

Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.

if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-01-12 18:01, Istvan Gabor wrote:

On Thu, 12 Jan 2017 13:06:49 +0100, Carlos E. R. wrote:

...

On 2017-01-12 10:03, Istvan Gabor wrote:

...

...

I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.

Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.

if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation.

fstab only describes which partitions to mount and where. Partition/volume/device recognition doesn't require fstab entry.

Or array mounts, too: /dev/md0 /data/raid xfs defaults,nofail,relatime 1 3 Maybe I misunderstood you :-? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 01/12/2017 03:03 AM, Istvan Gabor wrote:

If the intram/initrd requires to have the mdadm arrays it also means that I have to recreate initram when I set up a new array or remove an array. Doesn't sound good.

It's not that bad. When you add an array to the system, you a enable the mdadm hook in /etc/sysconfig and make sure you have your arrays defined in /etc/mdadm.conf. Once that hook is in place, you only need to rebuild the initramfs once, then whatever is now SuSEconfig, should automatically add that to the rebuild of any future kernel. When you remove the arrays, it's no big deal if the hook is left in place (as long as you change /etc/mdadm.conf and /etc/fstab to reflect the removal). It doesn't hurt to have mdadm check if there are arrays that need to be assembled even if there are none. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 13/01/17 08:10, David C. Rankin wrote:

Just saw Andrei's post -- looks like opensuse doesn't need mdadm hooks in initramfs -- the problem is elsewhere... Glad I didn't jump right in with a linux-raid 4 disk install :)

You NEED mdadm in your initramfs (unless you have a v1 mirror). Bear in mind, though, that I have no mdadm.conf that I can find on my system, and it boots off my mirror fine. The normal sequence of events on a raid boot is for the initramfs to be loaded, mdadm is called to assemble the arrays, and then a switchroot is done to the "/" mount. Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 13/01/17 12:57, Carlos E. R. wrote:

On 2017-01-13 12:48, Wols Lists wrote:

...

Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.

Mmm. initramfs holds kernel modules.

Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically.

/dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect

As I said, AUTOMATIC ASSEMBLY NO LONGER WORKS. It's a kernel feature that was removed some time in the 3.x series. "type raid" no longer means anything - all my partitions are type linux (8300 or 8200). And once you invoke mdadm - which you NEED to do for modern kernels - partition type is ignored. So sorry, your advice *was* correct, but times have moved on and it's been rendered obsolete and redundant :-( Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2017-01-13 18:14, Wols Lists wrote:

On 13/01/17 12:57, Carlos E. R. wrote:

...

On 2017-01-13 12:48, Wols Lists wrote:

...

Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.

Mmm. initramfs holds kernel modules.

Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically.

/dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect

As I said, AUTOMATIC ASSEMBLY NO LONGER WORKS. It's a kernel feature that was removed some time in the 3.x series. "type raid" no longer means anything - all my partitions are type linux (8300 or 8200).

And once you invoke mdadm - which you NEED to do for modern kernels - partition type is ignored.

So sorry, your advice *was* correct, but times have moved on and it's been rendered obsolete and redundant :-(

But I don't invoke mdadm, and they mount. Something may invoke it, but it is not me. 2016-12-24 00:54:38+01:00 - Booting the system now ================================================================================ Linux Telcontar 3.12.62-55-desktop #1 SMP PREEMPT Thu Oct 20 08:47:11 UTC 2016 (b0aa9a6) x86_64 x86_64 x86_64 GNU/Linux <5.6> 2016-12-24 00:54:40 Telcontar rsyslogd - - - [origin software="rsyslogd" swVersion="7.4.7" x-pid="1306" x-info="http://www.rsyslog.com"] start <3.6> 2016-12-24 00:54:40 Telcontar systemd-modules-load 354 - - Inserted module 'sg' <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Started Load Kernel Modules. <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Mounted FUSE Control File System. <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Mounting Configuration File System... <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.195005] raid6: sse2x1 4964 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.217007] raid6: sse2x2 6355 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.239009] raid6: sse2x4 8371 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.244340] raid6: using algorithm sse2x4 (8371 MB/s) <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.249662] raid6: using ssse3x2 recovery algorithm <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.826761] md: raid6 personality registered for level 6 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.832497] md: raid5 personality registered for level 5 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.838093] md: raid4 personality registered for level 4 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.843735] md/raid:md0: device sdb10 operational as raid disk 1 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.849036] md/raid:md0: device sdc15 operational as raid disk 2 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.854467] md/raid:md0: device sda11 operational as raid disk 0 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.860133] md/raid:md0: allocated 3298kB <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.865329] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870550] RAID conf printout: <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870551] --- level:5 rd:3 wd:3 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870553] disk 0, o:1, dev:sda11 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870554] disk 1, o:1, dev:sdb10 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870556] disk 2, o:1, dev:sdc15 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870918] created bitmap (1 pages) for device md0 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.876376] md0: bitmap initialized from disk: read 1 pages, set 0 of 193 bits <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.974409] md0: detected capacity change from 0 to 25777668096 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.997321] Adding 15727612k swap on /dev/sda3. Priority:1 extents:1 across:15727612k FS <3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-01-13 20:43, Wols Lists wrote:

On 13/01/17 19:24, Carlos E. R. wrote:

...

<3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done

mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly.

What kernel? What distro?

The first line in the log printed the kernel version. But I made a mistake, that log is for 13.1.

Oh - and that first line I've quoted says "starting mdadmd monitoring" ... I don't think that's anything to do with starting the arrays, it's firing up the monitor that emails you if anything goes wrong ...

Yes. That's the first appearance of mdadm in the log. The array was mounted earlier. On Leap 42.2: 2016-12-26 20:39:16+01:00 - Booting the system now ================================================================================ Linux Telcontar 4.4.36-8-default #1 SMP Fri Dec 9 16:18:38 UTC 2016 (3ec5648) x86_64 x86_64 x86_64 GNU/Linux <5.6> 2016-12-26 20:39:22 Telcontar rsyslogd - - - [origin software="rsyslogd" swVersion="8.4.0" x-pid="1551" x-info="http://www.rsyslog.com"] start <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 0.000000] microcode: CPU0 microcode updated early to revision 0xa0b, date = 2010-09-28 <3.6> 2016-12-26 20:39:22 Telcontar systemd 1 - - Found device /dev/disk/by-label/raid5. <3.5> 2016-12-26 20:39:22 Telcontar systemd 1 - - data-raid.mount: Directory /data/raid to mount over is not empty, mounting anyway. <3.6> 2016-12-26 20:39:22 Telcontar systemd 1 - - Mounting /data/raid... 0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.758408] REISERFS (device sdc6): Using r5 hash to sort names <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.876010] raid6: sse2x1 gen() 4952 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.944002] raid6: sse2x1 xor() 4806 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.012010] raid6: sse2x2 gen() 5267 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.080008] raid6: sse2x2 xor() 5807 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.148004] raid6: sse2x4 gen() 8218 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.188327] XFS (sdc8): Ending clean mount <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216008] raid6: sse2x4 xor() 6689 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216009] raid6: using algorithm sse2x4 gen() 8218 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216011] raid6: .... xor() 6689 MB/s, rmw enabled <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216012] raid6: using ssse3x2 recovery algorithm <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905538] md: raid6 personality registered for level 6 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905540] md: raid5 personality registered for level 5 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905540] md: raid4 personality registered for level 4 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.932459] audit: type=1400 audit(1482781131.317:37): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_icmp" pid=1 169 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936062] md/raid:md0: device sda11 operational as raid disk 0 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936063] md/raid:md0: device sdc15 operational as raid disk 2 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936064] md/raid:md0: device sdb10 operational as raid disk 1 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.951834] md/raid:md0: allocated 3316kB <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953008] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953008] RAID conf printout: <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953009] --- level:5 rd:3 wd:3 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953010] disk 0, o:1, dev:sda11 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953011] disk 1, o:1, dev:sdb10 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953012] disk 2, o:1, dev:sdc15 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.984767] audit: type=1400 audit(1482781131.369:38): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_ide_smart" pid=1187 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.037347] created bitmap (1 pages) for device md0 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.057130] audit: type=1400 audit(1482781131.441:39): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_load" pid=1193 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.062355] md0: bitmap initialized from disk: read 1 pages, set 0 of 193 bits <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.262738] md0: detected capacity change from 0 to 25777668096 Now, I don't think the timing on syslog is correct, though. Everything has the same timestamp, that can't be, either. If we believe the above as exact, mounting happens earlier than assembly. It is still the kernel who does the assembly, seconds on the boot. dmesg has slightly different info: [ 24.403400] md/raid:md0: device sdb11 operational as raid disk 0 [ 24.403401] md/raid:md0: device sdd15 operational as raid disk 2 [ 24.403403] md/raid:md0: device sdc10 operational as raid disk 1 [ 24.403683] md/raid:md0: allocated 3316kB [ 24.403729] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 So it happens 24 seconds after boot. And mount: [ 27.881141] XFS (md0): Mounting V4 Filesystem [ 35.903167] XFS (md0): Starting recovery (logdev: internal) [ 37.710756] XFS (md0): Ending recovery (logdev: internal) [ 38.404876] XFS (dm-0): Mounting V4 Filesystem [ 38.600968] XFS (dm-0): Starting recovery (logdev: internal) [ 39.208397] XFS (dm-0): Ending recovery (logdev: internal) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-01-13 21:23, Wols Lists wrote:

On 13/01/17 20:09, Carlos E. R. wrote:

...

...

...

What kernel? What distro? The first line in the log printed the kernel version. But I made a mistake, that log is for 13.1.

Ah. Now I know what I'm looking for I can see it ... :-)

:-) And then I posted the equivalent thing for 42.2, same hardware.

But I've been doing a bit of searching trying to find when auto-assembly was deleted. It was deprecated in 2009, but I can't find when it was removed from the kernel. I'm sure it was, however. I'll ask on the raid mailing list, as I've just checked the kernel source and something like it appears to be there - strange ...

(I notice Andrei says it was assembled by udev, which makes sense... there's been a bunch of bugs with mdadm and udev fighting over disks :-)

Could be udev, dunno. The log does not say, the entries are printed by the kernel, though. Could udev be telling the kernel what to do? -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith))

On 2017-01-14 11:00, Wols Lists wrote:

On 14/01/17 02:07, Carlos E. R. wrote:

Just got the correct answer straight from the horse's mouth :-)

auto-assembly is still there. It only works for 0.9 metadata, though.

0.9 is obsolete - if it breaks, it's unlikely to be fixed. And it has other problems, too. I noticed my (gentoo) kernel supports it, but it's also likely to be disabled by default at some point, if it isn't already, so that's another little push.

So you could still be using it, but you can't upgrade your array to a supported raid without moving off it. Typical kernel modernisation imho :-) - leave old features in until they break but effectively abandon them.

Ah... How can one find which version one is using? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-01-14 15:11, Per Jessen wrote:

Carlos E. R. wrote:

...

Ah...

How can one find which version one is using?

# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md127 : active raid1 sdt[0] sdl[1] 1952982848 blocks super 1.2 [2/2] [UU] ^^^^^^^^^

Ah. Telcontar:~ # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb11[4] sdd15[3] sdc10[1] 25173504 blocks super 1.0 level 5, 128k chunk, algorithm 2 [3/3] [UUU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none>

If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly.

It is 1.0. It appears to be assembled by the kernel :-? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

Wols Lists wrote:

On 14/01/17 14:11, Per Jessen wrote:

...

If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly.

ashdown anthony # mdadm --detail /dev/md127

Thanks Wol, that's exactly what I had in mind. /Per -- Per Jessen, Zürich (0.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, 12 Jan 2017 01:04:14 -0600, David C. Rankin wrote:

On 01/11/2017 03:35 PM, Istvan Gabor wrote:

...

Hi Andrei,

Was the new log file I sent this morning useful?

I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.

Thanks,

Istvan

Istvan,

I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)

There is something that I forgot to mention in the list. I have a SATA Seagate HD which use for backups. To decrease debug load I disconnected this drive. When this HD is not attached, the arrays are created correctly at boot. I would say there's some problem with that hard drive, but I've just run a smartctl long test on it last Saturday and it passed, no errors were reported. openSUSE 13.1 and 13.2 systems assemble the arrays correctly even when the Seagate drive is attached. Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, 12 Jan 2017 20:19:41 +0100, Carlos E. R. wrote:

On 2017-01-12 18:12, Istvan Gabor wrote:

...

There is something that I forgot to mention in the list.

I have a SATA Seagate HD which use for backups. To decrease debug load I disconnected this drive. When this HD is not attached, the arrays are created correctly at boot. I would say there's some problem with that hard drive, but I've just run a smartctl long test on it last Saturday and it passed, no errors were reported. openSUSE 13.1 and 13.2 systems assemble the arrays correctly even when the Seagate drive is attached.

Perhaps a duplicate UUID or label.

I doubt it. The Seagate 1 TB HD is independent from the system disks which contain the raid arrays. I experimented a little bit. Replaced the 1 TB Seagate with a 320 GB Western Digital WD3200AVVS drive. In this case the arrays were assembled correctly at boot. The I replaced the WD drive with another 1 TB Seagate (ST1000DM003, same model but different device). With this other Seagate the arrays are not assembled. So it seems that problem it triggered by some device specific property. smartctl -i for the Seagate drives gives: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162 Serial Number: xxxxxxxx LU WWN Device Id: 5 000c50 06c9a272d Firmware Version: CC47 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Jan 12 21:05:45 2017 CET ==> WARNING: A firmware update for this drive is available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled Should I try to update the drive's firmware? I never updated a HD's firmware so far. The question, why oS 13.x systems has no problem with the arrays in the same configuration, still remains. Cheers, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

smartctl -i for the Seagate drives gives:

=== START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162

Should I try to update the drive's firmware? I never updated a HD's firmware so far.

Probably you should try: http://ask.adaptec.com/app/answers/detail/a_id/17241/~/known-issues-with-sea... Make a backup before the firmware update!

12.01.2017 10:04, David C. Rankin пишет:

On 01/11/2017 03:35 PM, Istvan Gabor wrote:

...

Hi Andrei,

Was the new log file I sent this morning useful?

I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.

Thanks,

Istvan

Istvan,

I'll let Andrei decipher the udev messages, but I do have a thought. On Arch

I answered it off list (as I got them off list) - I do not see anything explaining this behavior. It looks like events related to disk are simply lost or udev for some reasons decides to not apply some rules. Unfortunately udev debug logging is both quite voluminous (so you are always in danger to lose some output) and rather non-informative, making it hard to associate particular log line with specific event.

at least, mdadm is added as a hook to the initcpio initramfs creation setup so

You (should) need it only if your boot/hibernate devices are on MD RAID. Otherwise there is no point to add them to initrd.

that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)

It actually works fine with three disks and stops working when the fourth disk is added. Currently I'm afraid I'm out of ideas how to debug it. At least, without being able to reproduce it or having access to this problem configuration. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, 12 Jan 2017 20:12:21 +0000, Wols Lists wrote:

On 12/01/17 17:24, Andrei Borzenkov wrote:

...

...

...

similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into

that mdadm is present in your boot image. If OpenSuSE requires something linux-raid on 42.2

...

yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)

It actually works fine with three disks and stops working when the fourth disk is added.

Currently I'm afraid I'm out of ideas how to debug it. At least, without being able to reproduce it or having access to this problem configuration.

Raid 0.9 can have problems accurately identifying disks - I thought this was fixed in 1.0 (the problem doesn't apply to 1.1 and 1.2). You're not getting any errors or warnings about superblocks, are you?

What version of mdadm are you using? mdadm 4.0 has just been released - it might well pay to upgrade.

Thanks. I don't think it is an mdadm problem, for several reasons: - The arrays are correctly assembled and run well in openSUSE 13.1 and 13.2. - Running cfdisk on the disks which contains the partitions for the arrays triggers mdadm and array assembly immediately. - The problem occurs only if a specific type/model of SATA HD is attached. Thank you again, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

14.01.2017 21:31, Andrei Borzenkov пишет:

Could you once more try both good and bad cases with udev debugging, but now using

udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on

OK, this was more successful. Now we at least see [ 23.647300] systemd-udevd[623]: IMPORT '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' /usr/lib/udev/rules.d/64-md-raid-assembly.rules:33 [ 26.224213] systemd-udevd[623]: Process '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' failed with exit code 1. for all components. This proves that startup sequence is correct. What you can try next - insert "-vvv" into mdadm invocation in udev rules; something like ACTION=="add|change", IMPORT{program}="BINDIR/mdadm -vvv --incremental --export $devnode --offroot ${DEVLINKS}" This hopefully will provide some additional output when mdadm fails and this output should be logged by udev. Run bad case with the same kernel parameters. There is no need to run good case as we know where it fails now. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

16.01.2017 20:05, Andrei Borzenkov пишет:

16.01.2017 20:01, Andrei Borzenkov пишет:

...

14.01.2017 21:31, Andrei Borzenkov пишет:

...

Could you once more try both good and bad cases with udev debugging, but now using

udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on

OK, this was more successful. Now we at least see

[ 23.647300] systemd-udevd[623]: IMPORT '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' /usr/lib/udev/rules.d/64-md-raid-assembly.rules:33 [ 26.224213] systemd-udevd[623]: Process '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' failed with exit code 1.

for all components.

This proves that startup sequence is correct. What you can try next - insert "-vvv" into mdadm invocation in udev rules; something like

ACTION=="add|change", IMPORT{program}="BINDIR/mdadm -vvv --incremental

s@BINDIR@/sbin@ of course; copied directly from mdadm sources.

...

--export $devnode --offroot ${DEVLINKS}"

This hopefully will provide some additional output when mdadm fails and this output should be logged by udev. Run bad case with the same kernel parameters. There is no need to run good case as we know where it fails now.

Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation. Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com). -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

17.01.2017 19:37, Wols Lists пишет:

On 17/01/17 03:36, Andrei Borzenkov wrote:

...

Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation.

Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com).

mdadm is now maintained by Jes Sorensen Jes.Sorensen@redhat.com

Sigh ... bor@bor-Latitude-E5450:~$ osc maintainer -v Base:System mdadm bugowner of Base:System/mdadm : neilbrown Neil Brown maintainer of Base:System/mdadm : michal-m Michal Marek msmeissn Marcus Meissner neilbrown Neil Brown

Neil handed it over at the start of 2016. Put a post on the mailing list and both Jes and Neil will see it (Neil has adopted an "elder statesman" role :-)

Cheers, Wol

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org