[opensuse] How to start mdadm service in openSUSE Leap 42.2?
Hello: I have a few md raid1 sets I used in previous openSUSE versions. In my freshly installed Leap 42.2 the raid sets are not assembled automatically at boot, though I have a correct mdadm.conf file. I can manually assemble the sets using mdadm. # cat /proc/mdstat cat: /proc/mdstat: No such file or directory # cat /etc/mdadm.conf DEVICE containers partitions ARRAY /dev/md/pc:6 UUID=21316afe:1a4dd0bf:50911056:88042a7c ARRAY /dev/md/pc:7 UUID=64e23ea9:7dcb9ee2:7bca71bd:248cc5cf # mdadm -E --scan ARRAY /dev/md/6 metadata=1.0 UUID=21316afe:1a4dd0bf:50911056:88042a7c name=pc:6 ARRAY /dev/md/7 metadata=1.0 UUID=64e23ea9:7dcb9ee2:7bca71bd:248cc5cf name=pc:7 # mdadm -A /dev/md6 /dev/sdb6 /dev/sdc6 mdadm: /dev/md6 has been started with 2 drives. # cat /proc/mdstat Personalities : [raid1] md6 : active raid1 sdb6[3] sdc6[2] 20971520 blocks super 1.0 [2/2] [UU] unused devices: <none> How can I start md service in Leap 42.2? Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor <suseuser04@gmail.hu> wrote:
How can I start md service in Leap 42.2?
MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually. If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug" (and omitting "quiet" to be sure) on kernel command line may provide some hints. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Jan 9, 2017 at 4:12 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor <suseuser04@gmail.hu> wrote:
How can I start md service in Leap 42.2?
MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually. If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"
This is udev.log-priority=debug, sorry.
(and omitting "quiet" to be sure) on kernel command line may provide some hints.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 9 Jan 2017 16:21:34 +0300, Andrei Borzenkov wrote:
On Mon, Jan 9, 2017 at 4:12 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor <suseuser04@gmail.hu> wrote:
How can I start md service in Leap 42.2?
MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually.
I looked at 64-md-raid-assembly.rules file but I don't know how to manually run the commands. I guess this is the command you mean: /sbin/mdadm --incremental --export $devnode --offroot ${DEVLINKS} I don't know what to take for $devnode and ${DEVLINKS}. I found that /dev/disk/by-uuid has only a few devices. It should have much more.
If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"
This is udev.log-priority=debug, sorry.
(and omitting "quiet" to be sure) on kernel command line may provide some hints.
OK, I booted with this kernel parameter. What to look for? And a very strange thing happened. I ran cfdisk. When I exited cfdisk (without writing anything) a lot of md devices have been created but only having one disk of the arrays: # cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0 md7 : inactive sdb7[3](S) 31455164 blocks super 1.0 unused devices: <none /dev/disk/by-uuid has been populated too. What next? Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
09.01.2017 18:07, Istvan Gabor пишет:
On Mon, 9 Jan 2017 16:21:34 +0300, Andrei Borzenkov wrote:
On Mon, Jan 9, 2017 at 4:12 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
On Mon, Jan 9, 2017 at 3:38 PM, Istvan Gabor <suseuser04@gmail.hu> wrote:
How can I start md service in Leap 42.2?
MD devices are expected tp be assembled incrementally by udev rules. See /usr/lib/udev/rules.d/64-md-raid-assembly.rules. Try commands from these rules manually.
I looked at 64-md-raid-assembly.rules file but I don't know how to manually run the commands. I guess this is the command you mean:
/sbin/mdadm --incremental --export $devnode --offroot ${DEVLINKS}
I don't know what to take for $devnode and ${DEVLINKS}.
$devnode is name of device that is being scanned (i.e. array component). $DEVLINKS is list of device aliases.
I found that /dev/disk/by-uuid has only a few devices. It should have much more.
It really sounds like some udev problem.
If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"
This is udev.log-priority=debug, sorry.
(and omitting "quiet" to be sure) on kernel command line may provide some hints.
OK, I booted with this kernel parameter. What to look for?
Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/
And a very strange thing happened. I ran cfdisk. When I exited cfdisk (without writing anything) a lot of md devices have been created but only having one disk of the arrays:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0
md7 : inactive sdb7[3](S) 31455164 blocks super 1.0
unused devices: <none
/dev/disk/by-uuid has been populated too.
What next?
mdadm --examine --scan -vv in addition to journalctl output would be good. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
It came up with a really weird config - in the config below, md6 and md7 have three drives each, and the only one present is a hot-spare.
# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0
md7 : inactive sdb7[3](S) 31455164 blocks super 1.0
-- Per Jessen, Zürich (-0.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
09.01.2017 21:08, Per Jessen пишет:
Andrei Borzenkov wrote:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
It came up with a really weird config - in the config below, md6 and md7 have three drives each, and the only one present is a hot-spare.
The [3] is not total number of components, but rather unique component identifier. It is also *not* position in the RAID array. This identifier may change e.g. after device replacement. Spare status is correct after "mdadm --incremental" with single component. So far it looks pretty OK except I would expect second half of array to be present as well.
# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0
md7 : inactive sdb7[3](S) 31455164 blocks super 1.0
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
09.01.2017 21:08, Per Jessen пишет:
Andrei Borzenkov wrote:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
It came up with a really weird config - in the config below, md6 and md7 have three drives each, and the only one present is a hot-spare.
The [3] is not total number of components, but rather unique component identifier. It is also *not* position in the RAID array. This identifier may change e.g. after device replacement.
ah, I misread it - actually I was thinking component#, not number of devices :-)
Spare status is correct after "mdadm --incremental" with single component. So far it looks pretty OK except I would expect second half of array to be present as well.
I would have expected the array to be running degraded with one disk missing/failed ? No disks present and one hot-spare still seems weird. -- Per Jessen, Zürich (-0.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 09 Jan 2017 20:12:30 +0100, Per Jessen wrote:
Andrei Borzenkov wrote:
09.01.2017 21:08, Per Jessen пишет:
Andrei Borzenkov wrote:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
Sorry, I omitted that I only ran cfdisk /dev/sdb. This has triggered only process of sdb devices, one halves of the arrays. Once more the whole process, step by step: # cat /proc/mdstat cat: /proc/mdstat: No such file or directory # cfdisk /dev/sdb QUIT Many devices are generated. # cat /proc/mdstat Personalities : md7 : inactive sdb7[3](S) 31455164 blocks super 1.0 md6 : inactive sdb6[3](S) 20972752 blocks super 1.0 unused devices: <none> # cfdisk /dev/sdc QUIT Many devices are generated again. # cat /proc/mdstat Personalities : [raid1] md7 : active raid1 sdc7[2] sdb7[3] 31455164 blocks super 1.0 [2/2] [UU] md6 : active raid1 sdc6[2] sdb6[3] 20971520 blocks super 1.0 [2/2] [UU] unused devices: <none> In reality I have 18 arrays, but inserted here only two as examples. Andrei, I sent the required log files to your address. Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 9 Jan 2017 20:35:16 +0300, Andrei Borzenkov wrote:
$devnode is name of device that is being scanned (i.e. array component). $DEVLINKS is list of device aliases.
I found that /dev/disk/by-uuid has only a few devices. It should have much more.
It really sounds like some udev problem.
If commands work, it probably means for some reason rules are not applied, in which case booting with "udev.debug"
This is udev.log-priority=debug, sorry.
(and omitting "quiet" to be sure) on kernel command line may provide some hints.
OK, I booted with this kernel parameter. What to look for?
Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/
I tried to upload the output from journalctl -b to susepaste.org but the site haven't accepted it. I guess it's too much to paste in. The whole log file is ~1 MB and it contains 11678 lines. I have sent the log to your email too, I don't know if you've received it.
And a very strange thing happened. I ran cfdisk. When I exited cfdisk (without writing anything) a lot of md devices have been created but only having one disk of the arrays:
Well, cfdisk likely triggered rescan of devices that in turn triggered events and udev processed them. Why only for half of components is a good question.
# cat /proc/mdstat Personalities : md6 : inactive sdb6[3](S) 20972752 blocks super 1.0
md7 : inactive sdb7[3](S) 31455164 blocks super 1.0
unused devices: <none
/dev/disk/by-uuid has been populated too.
What next?
mdadm --examine --scan -vv
in addition to journalctl output would be good.
I have uploaded it to susepaste.org: http://susepaste.org/02e27f89 Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
10.01.2017 22:32, Istvan Gabor пишет: ...
Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/
I tried to upload the output from journalctl -b to susepaste.org but the site haven't accepted it. I guess it's too much to paste in. The whole log file is ~1 MB and it contains 11678 lines. I have sent the log to your email too, I don't know if you've received it.
I did; as I answered in another mail, udev debug output was not that useful and there is nothing standing out between good and bad cases. Also some logs are definitely lost. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 11 Jan 2017 06:58:21 +0300, Andrei Borzenkov wrote:
10.01.2017 22:32, Istvan Gabor пишет: ...
Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/
I tried to upload the output from journalctl -b to susepaste.org but the site haven't accepted it. I guess it's too much to paste in. The whole log file is ~1 MB and it contains 11678 lines. I have sent the log to your email too, I don't know if you've received it.
I did; as I answered in another mail, udev debug output was not that useful and there is nothing standing out between good and bad cases. Also some logs are definitely lost.
Hi Andrei, Was the new log file I sent this morning useful? I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed. Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/11/2017 03:35 PM, Istvan Gabor wrote:
Hi Andrei,
Was the new log file I sent this morning useful?
I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.
Thanks,
Istvan
Istvan, I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, 12 Jan 2017 01:04:14 -0600, David C. Rankin wrote:
On 01/11/2017 03:35 PM, Istvan Gabor wrote:
Hi Andrei,
Was the new log file I sent this morning useful?
I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.
Thanks,
Istvan
David, thank you.
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.
Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.
Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)
If the intram/initrd requires to have the mdadm arrays it also means that I have to recreate initram when I set up a new array or remove an array. Doesn't sound good. I can try to activate my arrays and make a new initrd. Thanks, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-12 10:03, Istvan Gabor wrote:
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.
Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.
if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On Thu, 12 Jan 2017 13:06:49 +0100, Carlos E. R. wrote:
On 2017-01-12 10:03, Istvan Gabor wrote:
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.
Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.
if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation.
fstab only describes which partitions to mount and where. Partition/volume/device recognition doesn't require fstab entry. Cheers, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-12 18:01, Istvan Gabor wrote:
On Thu, 12 Jan 2017 13:06:49 +0100, Carlos E. R. wrote:
On 2017-01-12 10:03, Istvan Gabor wrote:
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.
Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.
if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation.
fstab only describes which partitions to mount and where. Partition/volume/device recognition doesn't require fstab entry.
Or array mounts, too: /dev/md0 /data/raid xfs defaults,nofail,relatime 1 3 Maybe I misunderstood you :-? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On Thu, 12 Jan 2017 20:17:49 +0100, Carlos E. R. wrote:
On 2017-01-12 18:01, Istvan Gabor wrote:
On Thu, 12 Jan 2017 13:06:49 +0100, Carlos E. R. wrote:
On 2017-01-12 10:03, Istvan Gabor wrote:
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup.
Your supposition is correct. My initram and yast does not have the arrays. When I install a new system I set up only the minimal requirements for it. I install only one partition, the root (/), and don't set up separate partitions. My fstab has only this root partition and a swap partition.
if fstab doesn't mention the array, how would the system know it has to set up the array? Maybe that would trigger initram creation.
fstab only describes which partitions to mount and where. Partition/volume/device recognition doesn't require fstab entry.
Or array mounts, too:
/dev/md0 /data/raid xfs defaults,nofail,relatime 1 3
Maybe I misunderstood you :-?
raid arrays are partition volumes too. In mdraid system they are designated as /dev/md* devices. Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/12/2017 03:03 AM, Istvan Gabor wrote:
If the intram/initrd requires to have the mdadm arrays it also means that I have to recreate initram when I set up a new array or remove an array. Doesn't sound good.
It's not that bad. When you add an array to the system, you a enable the mdadm hook in /etc/sysconfig and make sure you have your arrays defined in /etc/mdadm.conf. Once that hook is in place, you only need to rebuild the initramfs once, then whatever is now SuSEconfig, should automatically add that to the rebuild of any future kernel. When you remove the arrays, it's no big deal if the hook is left in place (as long as you change /etc/mdadm.conf and /etc/fstab to reflect the removal). It doesn't hurt to have mdadm check if there are arrays that need to be assembled even if there are none. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/13/2017 01:56 AM, David C. Rankin wrote:
It's not that bad. When you add an array to the system, you a enable the mdadm hook in /etc/sysconfig and make sure you have your arrays defined in /etc/mdadm.conf. Once that hook is in place, you only need to rebuild the initramfs once, then whatever is now SuSEconfig, should automatically add that to the rebuild of any future kernel. When you remove the arrays, it's no big deal if the hook is left in place (as long as you change /etc/mdadm.conf and /etc/fstab to reflect the removal). It doesn't hurt to have mdadm check if there are arrays that need to be assembled even if there are none.
Just saw Andrei's post -- looks like opensuse doesn't need mdadm hooks in initramfs -- the problem is elsewhere... Glad I didn't jump right in with a linux-raid 4 disk install :) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/01/17 08:10, David C. Rankin wrote:
Just saw Andrei's post -- looks like opensuse doesn't need mdadm hooks in initramfs -- the problem is elsewhere... Glad I didn't jump right in with a linux-raid 4 disk install :)
You NEED mdadm in your initramfs (unless you have a v1 mirror). Bear in mind, though, that I have no mdadm.conf that I can find on my system, and it boots off my mirror fine. The normal sequence of events on a raid boot is for the initramfs to be loaded, mdadm is called to assemble the arrays, and then a switchroot is done to the "/" mount. Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-13 12:48, Wols Lists wrote:
Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.
Mmm. initramfs holds kernel modules. Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically. /dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 13/01/17 12:57, Carlos E. R. wrote:
On 2017-01-13 12:48, Wols Lists wrote:
Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.
Mmm. initramfs holds kernel modules.
Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically.
/dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect
As I said, AUTOMATIC ASSEMBLY NO LONGER WORKS. It's a kernel feature that was removed some time in the 3.x series. "type raid" no longer means anything - all my partitions are type linux (8300 or 8200). And once you invoke mdadm - which you NEED to do for modern kernels - partition type is ignored. So sorry, your advice *was* correct, but times have moved on and it's been rendered obsolete and redundant :-( Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Wols Lists wrote:
On 13/01/17 12:57, Carlos E. R. wrote:
On 2017-01-13 12:48, Wols Lists wrote:
Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.
Mmm. initramfs holds kernel modules.
Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically.
/dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect
As I said, AUTOMATIC ASSEMBLY NO LONGER WORKS. It's a kernel feature that was removed some time in the 3.x series. "type raid" no longer means anything - all my partitions are type linux (8300 or 8200).
Ditto, we've stopped using 0xfd long ago. -- Per Jessen, Zürich (0.4°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-13 18:14, Wols Lists wrote:
On 13/01/17 12:57, Carlos E. R. wrote:
On 2017-01-13 12:48, Wols Lists wrote:
Because linux can no longer assemble arrays itself, that is why the initramfs is needed for anything other said v1 mirror - said mirror looking to the kernel like any other normal file system.
Mmm. initramfs holds kernel modules.
Just remembered. You have to mark partititions to be of raid type in order for the raids to be assembled automatically.
/dev/sdd15 2598531072 2762371071 163840000 78.1G fd Linux raid autodetect
As I said, AUTOMATIC ASSEMBLY NO LONGER WORKS. It's a kernel feature that was removed some time in the 3.x series. "type raid" no longer means anything - all my partitions are type linux (8300 or 8200).
And once you invoke mdadm - which you NEED to do for modern kernels - partition type is ignored.
So sorry, your advice *was* correct, but times have moved on and it's been rendered obsolete and redundant :-(
But I don't invoke mdadm, and they mount. Something may invoke it, but it is not me. 2016-12-24 00:54:38+01:00 - Booting the system now ================================================================================ Linux Telcontar 3.12.62-55-desktop #1 SMP PREEMPT Thu Oct 20 08:47:11 UTC 2016 (b0aa9a6) x86_64 x86_64 x86_64 GNU/Linux <5.6> 2016-12-24 00:54:40 Telcontar rsyslogd - - - [origin software="rsyslogd" swVersion="7.4.7" x-pid="1306" x-info="http://www.rsyslog.com"] start <3.6> 2016-12-24 00:54:40 Telcontar systemd-modules-load 354 - - Inserted module 'sg' <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Started Load Kernel Modules. <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Mounted FUSE Control File System. <3.6> 2016-12-24 00:54:40 Telcontar systemd 1 - - Mounting Configuration File System... <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.195005] raid6: sse2x1 4964 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.217007] raid6: sse2x2 6355 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.239009] raid6: sse2x4 8371 MB/s <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.244340] raid6: using algorithm sse2x4 (8371 MB/s) <0.4> 2016-12-24 00:54:40 Telcontar kernel - - - [ 5.249662] raid6: using ssse3x2 recovery algorithm <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.826761] md: raid6 personality registered for level 6 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.832497] md: raid5 personality registered for level 5 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.838093] md: raid4 personality registered for level 4 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.843735] md/raid:md0: device sdb10 operational as raid disk 1 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.849036] md/raid:md0: device sdc15 operational as raid disk 2 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.854467] md/raid:md0: device sda11 operational as raid disk 0 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.860133] md/raid:md0: allocated 3298kB <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.865329] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870550] RAID conf printout: <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870551] --- level:5 rd:3 wd:3 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870553] disk 0, o:1, dev:sda11 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870554] disk 1, o:1, dev:sdb10 <0.7> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870556] disk 2, o:1, dev:sdc15 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.870918] created bitmap (1 pages) for device md0 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.876376] md0: bitmap initialized from disk: read 1 pages, set 0 of 193 bits <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.974409] md0: detected capacity change from 0 to 25777668096 <0.6> 2016-12-24 00:54:40 Telcontar kernel - - - [ 17.997321] Adding 15727612k swap on /dev/sda3. Priority:1 extents:1 across:15727612k FS <3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 13/01/17 19:24, Carlos E. R. wrote:
<3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done
mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly.
What kernel? What distro? Oh - and that first line I've quoted says "starting mdadmd monitoring" ... I don't think that's anything to do with starting the arrays, it's firing up the monitor that emails you if anything goes wrong ... Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-13 20:43, Wols Lists wrote:
On 13/01/17 19:24, Carlos E. R. wrote:
<3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done
mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly.
What kernel? What distro?
The first line in the log printed the kernel version. But I made a mistake, that log is for 13.1.
Oh - and that first line I've quoted says "starting mdadmd monitoring" ... I don't think that's anything to do with starting the arrays, it's firing up the monitor that emails you if anything goes wrong ...
Yes. That's the first appearance of mdadm in the log. The array was mounted earlier. On Leap 42.2: 2016-12-26 20:39:16+01:00 - Booting the system now ================================================================================ Linux Telcontar 4.4.36-8-default #1 SMP Fri Dec 9 16:18:38 UTC 2016 (3ec5648) x86_64 x86_64 x86_64 GNU/Linux <5.6> 2016-12-26 20:39:22 Telcontar rsyslogd - - - [origin software="rsyslogd" swVersion="8.4.0" x-pid="1551" x-info="http://www.rsyslog.com"] start <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 0.000000] microcode: CPU0 microcode updated early to revision 0xa0b, date = 2010-09-28 <3.6> 2016-12-26 20:39:22 Telcontar systemd 1 - - Found device /dev/disk/by-label/raid5. <3.5> 2016-12-26 20:39:22 Telcontar systemd 1 - - data-raid.mount: Directory /data/raid to mount over is not empty, mounting anyway. <3.6> 2016-12-26 20:39:22 Telcontar systemd 1 - - Mounting /data/raid... 0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.758408] REISERFS (device sdc6): Using r5 hash to sort names <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.876010] raid6: sse2x1 gen() 4952 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 24.944002] raid6: sse2x1 xor() 4806 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.012010] raid6: sse2x2 gen() 5267 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.080008] raid6: sse2x2 xor() 5807 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.148004] raid6: sse2x4 gen() 8218 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.188327] XFS (sdc8): Ending clean mount <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216008] raid6: sse2x4 xor() 6689 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216009] raid6: using algorithm sse2x4 gen() 8218 MB/s <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216011] raid6: .... xor() 6689 MB/s, rmw enabled <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.216012] raid6: using ssse3x2 recovery algorithm <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905538] md: raid6 personality registered for level 6 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905540] md: raid5 personality registered for level 5 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.905540] md: raid4 personality registered for level 4 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.932459] audit: type=1400 audit(1482781131.317:37): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_icmp" pid=1 169 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936062] md/raid:md0: device sda11 operational as raid disk 0 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936063] md/raid:md0: device sdc15 operational as raid disk 2 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.936064] md/raid:md0: device sdb10 operational as raid disk 1 <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.951834] md/raid:md0: allocated 3316kB <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953008] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953008] RAID conf printout: <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953009] --- level:5 rd:3 wd:3 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953010] disk 0, o:1, dev:sda11 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953011] disk 1, o:1, dev:sdb10 <0.7> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.953012] disk 2, o:1, dev:sdc15 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 25.984767] audit: type=1400 audit(1482781131.369:38): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_ide_smart" pid=1187 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.037347] created bitmap (1 pages) for device md0 <0.5> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.057130] audit: type=1400 audit(1482781131.441:39): apparmor="STATUS" operation="profile_load" name="/usr/lib/nagios/plugins/check_load" pid=1193 comm="apparmor_parser" <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.062355] md0: bitmap initialized from disk: read 1 pages, set 0 of 193 bits <0.6> 2016-12-26 20:39:22 Telcontar kernel - - - [ 26.262738] md0: detected capacity change from 0 to 25777668096 Now, I don't think the timing on syslog is correct, though. Everything has the same timestamp, that can't be, either. If we believe the above as exact, mounting happens earlier than assembly. It is still the kernel who does the assembly, seconds on the boot. dmesg has slightly different info: [ 24.403400] md/raid:md0: device sdb11 operational as raid disk 0 [ 24.403401] md/raid:md0: device sdd15 operational as raid disk 2 [ 24.403403] md/raid:md0: device sdc10 operational as raid disk 1 [ 24.403683] md/raid:md0: allocated 3316kB [ 24.403729] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 So it happens 24 seconds after boot. And mount: [ 27.881141] XFS (md0): Mounting V4 Filesystem [ 35.903167] XFS (md0): Starting recovery (logdev: internal) [ 37.710756] XFS (md0): Ending recovery (logdev: internal) [ 38.404876] XFS (dm-0): Mounting V4 Filesystem [ 38.600968] XFS (dm-0): Starting recovery (logdev: internal) [ 39.208397] XFS (dm-0): Ending recovery (logdev: internal) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 13/01/17 20:09, Carlos E. R. wrote:
On 2017-01-13 20:43, Wols Lists wrote:
On 13/01/17 19:24, Carlos E. R. wrote:
<3.6> 2016-12-24 00:55:00 Telcontar systemd 1 - - Starting LSB: mdadmd daemon monitoring MD devices... <3.6> 2016-12-24 00:55:00 Telcontar echo 3716 - - Starting virus-scanner (amavisd-new): <3.6> 2016-12-24 00:55:01 Telcontar mdadmd 3717 - - Starting mdadmd ..done
mdadm is called 20 seconds after the array appears on boot. According to the log above, it is the kernel directly who does the assembly.
What kernel? What distro? The first line in the log printed the kernel version. But I made a mistake, that log is for 13.1.
Ah. Now I know what I'm looking for I can see it ... :-) But I've been doing a bit of searching trying to find when auto-assembly was deleted. It was deprecated in 2009, but I can't find when it was removed from the kernel. I'm sure it was, however. I'll ask on the raid mailing list, as I've just checked the kernel source and something like it appears to be there - strange ... (I notice Andrei says it was assembled by udev, which makes sense... there's been a bunch of bugs with mdadm and udev fighting over disks :-) Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-13 21:23, Wols Lists wrote:
On 13/01/17 20:09, Carlos E. R. wrote:
What kernel? What distro? The first line in the log printed the kernel version. But I made a mistake, that log is for 13.1.
Ah. Now I know what I'm looking for I can see it ... :-)
:-) And then I posted the equivalent thing for 42.2, same hardware.
But I've been doing a bit of searching trying to find when auto-assembly was deleted. It was deprecated in 2009, but I can't find when it was removed from the kernel. I'm sure it was, however. I'll ask on the raid mailing list, as I've just checked the kernel source and something like it appears to be there - strange ...
(I notice Andrei says it was assembled by udev, which makes sense... there's been a bunch of bugs with mdadm and udev fighting over disks :-)
Could be udev, dunno. The log does not say, the entries are printed by the kernel, though. Could udev be telling the kernel what to do? -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith))
On 14/01/17 02:07, Carlos E. R. wrote:
But I've been doing a bit of searching trying to find when auto-assembly
was deleted. It was deprecated in 2009, but I can't find when it was removed from the kernel. I'm sure it was, however. I'll ask on the raid mailing list, as I've just checked the kernel source and something like it appears to be there - strange ...
(I notice Andrei says it was assembled by udev, which makes sense... there's been a bunch of bugs with mdadm and udev fighting over disks :-) Could be udev, dunno. The log does not say, the entries are printed by the kernel, though. Could udev be telling the kernel what to do?
Just got the correct answer straight from the horse's mouth :-) auto-assembly is still there. It only works for 0.9 metadata, though. 0.9 is obsolete - if it breaks, it's unlikely to be fixed. And it has other problems, too. I noticed my (gentoo) kernel supports it, but it's also likely to be disabled by default at some point, if it isn't already, so that's another little push. So you could still be using it, but you can't upgrade your array to a supported raid without moving off it. Typical kernel modernisation imho :-) - leave old features in until they break but effectively abandon them. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-14 11:00, Wols Lists wrote:
On 14/01/17 02:07, Carlos E. R. wrote:
Just got the correct answer straight from the horse's mouth :-)
auto-assembly is still there. It only works for 0.9 metadata, though.
0.9 is obsolete - if it breaks, it's unlikely to be fixed. And it has other problems, too. I noticed my (gentoo) kernel supports it, but it's also likely to be disabled by default at some point, if it isn't already, so that's another little push.
So you could still be using it, but you can't upgrade your array to a supported raid without moving off it. Typical kernel modernisation imho :-) - leave old features in until they break but effectively abandon them.
Ah... How can one find which version one is using? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-01-14 11:00, Wols Lists wrote:
On 14/01/17 02:07, Carlos E. R. wrote:
Just got the correct answer straight from the horse's mouth :-)
auto-assembly is still there. It only works for 0.9 metadata, though.
0.9 is obsolete - if it breaks, it's unlikely to be fixed. And it has other problems, too. I noticed my (gentoo) kernel supports it, but it's also likely to be disabled by default at some point, if it isn't already, so that's another little push.
So you could still be using it, but you can't upgrade your array to a supported raid without moving off it. Typical kernel modernisation imho :-) - leave old features in until they break but effectively abandon them.
Ah...
How can one find which version one is using?
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md127 : active raid1 sdt[0] sdl[1] 1952982848 blocks super 1.2 [2/2] [UU] ^^^^^^^^^ md1 : active raid5 sda[0] sdd[2] sdc[4] sdb[1] 5859337728 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] md0 : active raid5 sdv[3] sdw[4] sdu[8](S) sdx[6] sdh[1] sdj[7](S) sdg[0] sdi[2] 9764912640 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU] If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly. -- Per Jessen, Zürich (0.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-14 15:11, Per Jessen wrote:
Carlos E. R. wrote:
Ah...
How can one find which version one is using?
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md127 : active raid1 sdt[0] sdl[1] 1952982848 blocks super 1.2 [2/2] [UU] ^^^^^^^^^
Ah. Telcontar:~ # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb11[4] sdd15[3] sdc10[1] 25173504 blocks super 1.0 level 5, 128k chunk, algorithm 2 [3/3] [UUU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none>
If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly.
It is 1.0. It appears to be assembled by the kernel :-? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 14/01/17 14:11, Per Jessen wrote:
If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly.
ashdown anthony # mdadm --detail /dev/md127 /dev/md127: Version : 1.2 ^^^^^^^^^^^^^ Creation Time : Sun Jul 6 13:03:35 2014 Raid Level : raid1 Array Size : 2694204672 (2569.39 GiB 2758.87 GB) Used Dev Size : 2694204672 (2569.39 GiB 2758.87 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sat Jan 14 23:13:56 2017 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : ashdown:0 (local to host ashdown) UUID : 42514e8a:2d127c98:7c2f52fe:60835b32 Events : 515496 Number Major Minor RaidDevice State 0 8 21 0 active sync /dev/sdb5 2 8 5 1 active sync /dev/sda5 Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Wols Lists wrote:
On 14/01/17 14:11, Per Jessen wrote:
If it's not shown, I think it's 0.9, but there is probably some tool that will tell you exactly.
ashdown anthony # mdadm --detail /dev/md127
Thanks Wol, that's exactly what I had in mind. /Per -- Per Jessen, Zürich (0.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, 12 Jan 2017 01:04:14 -0600, David C. Rankin wrote:
On 01/11/2017 03:35 PM, Istvan Gabor wrote:
Hi Andrei,
Was the new log file I sent this morning useful?
I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.
Thanks,
Istvan
Istvan,
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch at least, mdadm is added as a hook to the initcpio initramfs creation setup so that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)
There is something that I forgot to mention in the list. I have a SATA Seagate HD which use for backups. To decrease debug load I disconnected this drive. When this HD is not attached, the arrays are created correctly at boot. I would say there's some problem with that hard drive, but I've just run a smartctl long test on it last Saturday and it passed, no errors were reported. openSUSE 13.1 and 13.2 systems assemble the arrays correctly even when the Seagate drive is attached. Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-12 18:12, Istvan Gabor wrote:
There is something that I forgot to mention in the list.
I have a SATA Seagate HD which use for backups. To decrease debug load I disconnected this drive. When this HD is not attached, the arrays are created correctly at boot. I would say there's some problem with that hard drive, but I've just run a smartctl long test on it last Saturday and it passed, no errors were reported. openSUSE 13.1 and 13.2 systems assemble the arrays correctly even when the Seagate drive is attached.
Perhaps a duplicate UUID or label. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On Thu, 12 Jan 2017 20:19:41 +0100, Carlos E. R. wrote:
On 2017-01-12 18:12, Istvan Gabor wrote:
There is something that I forgot to mention in the list.
I have a SATA Seagate HD which use for backups. To decrease debug load I disconnected this drive. When this HD is not attached, the arrays are created correctly at boot. I would say there's some problem with that hard drive, but I've just run a smartctl long test on it last Saturday and it passed, no errors were reported. openSUSE 13.1 and 13.2 systems assemble the arrays correctly even when the Seagate drive is attached.
Perhaps a duplicate UUID or label.
I doubt it. The Seagate 1 TB HD is independent from the system disks which contain the raid arrays. I experimented a little bit. Replaced the 1 TB Seagate with a 320 GB Western Digital WD3200AVVS drive. In this case the arrays were assembled correctly at boot. The I replaced the WD drive with another 1 TB Seagate (ST1000DM003, same model but different device). With this other Seagate the arrays are not assembled. So it seems that problem it triggered by some device specific property. smartctl -i for the Seagate drives gives: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162 Serial Number: xxxxxxxx LU WWN Device Id: 5 000c50 06c9a272d Firmware Version: CC47 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Jan 12 21:05:45 2017 CET ==> WARNING: A firmware update for this drive is available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled Should I try to update the drive's firmware? I never updated a HD's firmware so far. The question, why oS 13.x systems has no problem with the arrays in the same configuration, still remains. Cheers, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-01-12 21:25, Istvan Gabor wrote:
On Thu, 12 Jan 2017 20:19:41 +0100, Carlos E. R. wrote:
Perhaps a duplicate UUID or label.
I doubt it. The Seagate 1 TB HD is independent from the system disks which contain the raid arrays.
I experimented a little bit. Replaced the 1 TB Seagate with a 320 GB Western Digital WD3200AVVS drive. In this case the arrays were assembled correctly at boot. The I replaced the WD drive with another 1 TB Seagate (ST1000DM003, same model but different device). With this other Seagate the arrays are not assembled. So it seems that problem it triggered by some device specific property.
It does look as if some identifier is duplicated. Check them all. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
smartctl -i for the Seagate drives gives:
=== START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST1000DM003-1CH162
Should I try to update the drive's firmware? I never updated a HD's firmware so far.
Probably you should try: http://ask.adaptec.com/app/answers/detail/a_id/17241/~/known-issues-with-sea... Make a backup before the firmware update!
On 2017-01-12 21:25, Istvan Gabor wrote:
==> WARNING: A firmware update for this drive is available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en
SMART support is: Available - device has SMART capability. SMART support is: Enabled
Should I try to update the drive's firmware? I never updated a HD's firmware so far.
Make sure that it is indeed available. In my case, I get this, very similar, message (is/may): ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en and there is no update. So, read those two articles; if I remember correctly, you have to compare the exact version of your disk with a table on that article or a link, and then you know what the issues are and if there is an update. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
12.01.2017 10:04, David C. Rankin пишет:
On 01/11/2017 03:35 PM, Istvan Gabor wrote:
Hi Andrei,
Was the new log file I sent this morning useful?
I have tried openSUSE 13.2 and it assembles the arrays correctly at boot. I haven't tried Leap 42.1 because I don't have it installed.
Thanks,
Istvan
Istvan,
I'll let Andrei decipher the udev messages, but I do have a thought. On Arch
I answered it off list (as I got them off list) - I do not see anything explaining this behavior. It looks like events related to disk are simply lost or udev for some reasons decides to not apply some rules. Unfortunately udev debug logging is both quite voluminous (so you are always in danger to lose some output) and rather non-informative, making it hard to associate particular log line with specific event.
at least, mdadm is added as a hook to the initcpio initramfs creation setup so
You (should) need it only if your boot/hibernate devices are on MD RAID. Otherwise there is no point to add them to initrd.
that mdadm is present in your boot image. If OpenSuSE requires something similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)
It actually works fine with three disks and stops working when the fourth disk is added. Currently I'm afraid I'm out of ideas how to debug it. At least, without being able to reproduce it or having access to this problem configuration. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/01/17 17:24, Andrei Borzenkov wrote:
that mdadm is present in your boot image. If OpenSuSE requires something
similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into linux-raid on 42.2 yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)
It actually works fine with three disks and stops working when the fourth disk is added.
Currently I'm afraid I'm out of ideas how to debug it. At least, without being able to reproduce it or having access to this problem configuration.
Raid 0.9 can have problems accurately identifying disks - I thought this was fixed in 1.0 (the problem doesn't apply to 1.1 and 1.2). You're not getting any errors or warnings about superblocks, are you? What version of mdadm are you using? mdadm 4.0 has just been released - it might well pay to upgrade. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, 12 Jan 2017 20:12:21 +0000, Wols Lists wrote:
On 12/01/17 17:24, Andrei Borzenkov wrote:
similar, then when you moved arrays to your server, perhaps mdadm is missing from your image since it wasn't present when YAST created your setup. Like I said, this is just a 'thought' since I haven't waded into
that mdadm is present in your boot image. If OpenSuSE requires something linux-raid on 42.2
yet and do not know what the underlying udev (should be Upray) involvement in activation and assembly is on this release. It just seems like you would need the kernel-level support active before udev could do its magic (maybe udev can do it all now -- I'll let Andrei fill in the details -- I don't know)
It actually works fine with three disks and stops working when the fourth disk is added.
Currently I'm afraid I'm out of ideas how to debug it. At least, without being able to reproduce it or having access to this problem configuration.
Raid 0.9 can have problems accurately identifying disks - I thought this was fixed in 1.0 (the problem doesn't apply to 1.1 and 1.2). You're not getting any errors or warnings about superblocks, are you?
What version of mdadm are you using? mdadm 4.0 has just been released - it might well pay to upgrade.
Thanks. I don't think it is an mdadm problem, for several reasons: - The arrays are correctly assembled and run well in openSUSE 13.1 and 13.2. - Running cfdisk on the disks which contains the partitions for the arrays triggers mdadm and array assembly immediately. - The problem occurs only if a specific type/model of SATA HD is attached. Thank you again, Istvan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
12.01.2017 00:35, Istvan Gabor пишет:
On Wed, 11 Jan 2017 06:58:21 +0300, Andrei Borzenkov wrote:
10.01.2017 22:32, Istvan Gabor пишет: ...
Upload output of "journalctl -b" somewhere, e.g. http://susepaste.org/
I tried to upload the output from journalctl -b to susepaste.org but the site haven't accepted it. I guess it's too much to paste in. The whole log file is ~1 MB and it contains 11678 lines. I have sent the log to your email too, I don't know if you've received it.
I did; as I answered in another mail, udev debug output was not that useful and there is nothing standing out between good and bad cases. Also some logs are definitely lost.
Hi Andrei,
Was the new log file I sent this morning useful?
Not really. It does show that udevadm trigger is run second time after switch to real root, so this can be ruled out.
I have tried openSUSE 13.2 and it assembles the arrays correctly at boot.
Well, it used completely different approach, so it not comparable. As I mentioned, one problem is that logs are lost (apparently, journal cannot cope with this amount or it is started too late). Log for working case does show traces of udev event(s) that trigger raid assembly - but only second part of them. Unfortunately kernel appears to ratelimit messages as well; fortunately Leap kernel appears to include recent patch to control it on kernel command line. Could you once more try both good and bad cases with udev debugging, but now using udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on The later parameter disables ratelimiting in kernel. We should now hopefully get full log and it is enough to just send output of "dmesg" (as everything must be now in kernel buffer). Please compress files, thet are expected to be rather large. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
14.01.2017 21:31, Andrei Borzenkov пишет:
Could you once more try both good and bad cases with udev debugging, but now using
udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on
OK, this was more successful. Now we at least see [ 23.647300] systemd-udevd[623]: IMPORT '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' /usr/lib/udev/rules.d/64-md-raid-assembly.rules:33 [ 26.224213] systemd-udevd[623]: Process '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' failed with exit code 1. for all components. This proves that startup sequence is correct. What you can try next - insert "-vvv" into mdadm invocation in udev rules; something like ACTION=="add|change", IMPORT{program}="BINDIR/mdadm -vvv --incremental --export $devnode --offroot ${DEVLINKS}" This hopefully will provide some additional output when mdadm fails and this output should be logged by udev. Run bad case with the same kernel parameters. There is no need to run good case as we know where it fails now. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
16.01.2017 20:01, Andrei Borzenkov пишет:
14.01.2017 21:31, Andrei Borzenkov пишет:
Could you once more try both good and bad cases with udev debugging, but now using
udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on
OK, this was more successful. Now we at least see
[ 23.647300] systemd-udevd[623]: IMPORT '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' /usr/lib/udev/rules.d/64-md-raid-assembly.rules:33 [ 26.224213] systemd-udevd[623]: Process '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' failed with exit code 1.
for all components.
This proves that startup sequence is correct. What you can try next - insert "-vvv" into mdadm invocation in udev rules; something like
ACTION=="add|change", IMPORT{program}="BINDIR/mdadm -vvv --incremental
s@BINDIR@/sbin@ of course; copied directly from mdadm sources.
--export $devnode --offroot ${DEVLINKS}"
This hopefully will provide some additional output when mdadm fails and this output should be logged by udev. Run bad case with the same kernel parameters. There is no need to run good case as we know where it fails now.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
16.01.2017 20:05, Andrei Borzenkov пишет:
16.01.2017 20:01, Andrei Borzenkov пишет:
14.01.2017 21:31, Andrei Borzenkov пишет:
Could you once more try both good and bad cases with udev debugging, but now using
udev.log-priority=debug systemd.log_target=kmsg log_buf_len=16M printk.devkmsg=on
OK, this was more successful. Now we at least see
[ 23.647300] systemd-udevd[623]: IMPORT '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' /usr/lib/udev/rules.d/64-md-raid-assembly.rules:33 [ 26.224213] systemd-udevd[623]: Process '/sbin/mdadm --incremental --export /dev/sdc21 --offroot ${DEVLINKS}' failed with exit code 1.
for all components.
This proves that startup sequence is correct. What you can try next - insert "-vvv" into mdadm invocation in udev rules; something like
ACTION=="add|change", IMPORT{program}="BINDIR/mdadm -vvv --incremental
s@BINDIR@/sbin@ of course; copied directly from mdadm sources.
--export $devnode --offroot ${DEVLINKS}"
This hopefully will provide some additional output when mdadm fails and this output should be logged by udev. Run bad case with the same kernel parameters. There is no need to run good case as we know where it fails now.
Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation. Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com). -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/17 03:36, Andrei Borzenkov wrote:
Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation.
Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com).
mdadm is now maintained by Jes Sorensen Jes.Sorensen@redhat.com Neil handed it over at the start of 2016. Put a post on the mailing list and both Jes and Neil will see it (Neil has adopted an "elder statesman" role :-) Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
17.01.2017 19:37, Wols Lists пишет:
On 17/01/17 03:36, Andrei Borzenkov wrote:
Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation.
Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com).
mdadm is now maintained by Jes Sorensen Jes.Sorensen@redhat.com
Sigh ... bor@bor-Latitude-E5450:~$ osc maintainer -v Base:System mdadm bugowner of Base:System/mdadm : neilbrown Neil Brown <nfbrown@suse.com> maintainer of Base:System/mdadm : michal-m Michal Marek <mmarek@suse.com> msmeissn Marcus Meissner <meissner@suse.com> neilbrown Neil Brown <nfbrown@suse.com>
Neil handed it over at the start of 2016. Put a post on the mailing list and both Jes and Neil will see it (Neil has adopted an "elder statesman" role :-)
Cheers, Wol
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/17 16:51, Andrei Borzenkov wrote:
17.01.2017 19:37, Wols Lists пишет:
On 17/01/17 03:36, Andrei Borzenkov wrote:
Unfortunately it did not show anything new. mdadm simply silently fails. We can actually see (as debug output in other rules) that device does have correct UUID and MD name as expected by mdadm.conf you shown earlier. So mdadm simply silently fails without any explanation.
Sorry, I have to give up. I suggest you open bug report and assign it to mdadm maintainer (nfbrown@suse.com).
mdadm is now maintained by Jes Sorensen Jes.Sorensen@redhat.com
Sigh ...
Are you sure? osc looks to me like a SUSE command, and as a SUSE person, Neil could still be the package maintainer ...
bor@bor-Latitude-E5450:~$ osc maintainer -v Base:System mdadm bugowner of Base:System/mdadm : neilbrown Neil Brown <nfbrown@suse.com>
maintainer of Base:System/mdadm : michal-m Michal Marek <mmarek@suse.com> msmeissn Marcus Meissner <meissner@suse.com> neilbrown Neil Brown <nfbrown@suse.com>
Especially as it doesn't say mdadm, it says "Base:System/mdadm". Jes is RedHat, why would he be maintaining a SUSE package? Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (7)
-
Andrei Borzenkov -
Carlos E. R. -
David C. Rankin -
Florian Gleixner -
Istvan Gabor -
Per Jessen -
Wols Lists