http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c2 --- Comment #2 from Tamás Németh 2009-04-15 17:32:19 MDT --- An example of trying to access a separate path to a multipath volume: carrier4:~ # multipathd -k multipathd> show topology .. xentemp103-32 (3600a0b80004808d2000009884846b2cc) dm-1 SUN,CSM200_R [size=20G][features=1 queue_if_no_path][hwhandler=1 rdac][rw] \_ round-robin 0 [prio=12][active] \_ 7:0:0:1 sdg 8:96 [active][ready] \_ 5:0:1:1 sdbq 68:64 [active][ready] \_ round-robin 0 [prio=2][enabled] \_ 5:0:0:1 sdc 8:32 [active][ghost] \_ 7:0:1:1 sdan 66:112 [active][ghost] .. carrier4:~ # mount -t ext3 /dev/sdg2 /mnt mount: /dev/sdg2 already mounted or /mnt busy carrier4:~ # mount -t ext3 /dev/sdbq2 /mnt mount: /dev/sdbq2 already mounted or /mnt busy carrier4:~ # mount -t ext3 /dev/sdc2 /mnt mount: /dev/sdc2 already mounted or /mnt busy carrier4:~ # mount -t ext3 /dev/sdan2 /mnt mount: /dev/sdan2 already mounted or /mnt busy carrier4:~ # mount -t ext3 /dev/mapper/xentemp103-32_part2 /mnt carrier4:~ # mount | grep mnt /dev/mapper/xentemp103-32_part2 on /mnt type ext3 (rw) -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 Leon Wang changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |llwang@novell.com AssignedTo|bnc-team-screening@forge.pr |kernel-maintainers@forge.pr |ovo.novell.com |ovo.novell.com -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User jreidinger@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c5 Josef Reidinger changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |nice@titanic.nyme.hu --- Comment #5 from Josef Reidinger 2009-05-19 02:14:35 MDT --- Do you have enabled user-friendly names for multipath (from topology it looks that yes) so it is duplicate of bug 470109 . Please try disable user friendly names if this helps (fix waiting for maintenance update). http://bugzilla.novell.com/show_bug.cgi?id=470109 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c7 --- Comment #7 from Tamás Németh 2009-05-19 10:27:04 MDT --- Created an attachment (id=293082) --> (http://bugzilla.novell.com/attachment.cgi?id=293082) boot.jpg Some /etc/init.d startup scripts start runing while the kernel is still collecting multipath data -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User jreidinger@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c9 Josef Reidinger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |juhliarik@novell.com AssignedTo|jreidinger@novell.com |aschnell@novell.com --- Comment #9 from Josef Reidinger 2009-05-19 10:32:25 MDT --- OK, this this is another issues. perl-bootloader response only for mkinitrd hangup and is tracked in bug 4970109 . Assign to arvin who response for partitioner...other issues I am not sure who response for it and hope arvin know. juhliarek added for yast2 bootloader hang up. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c11 Tamás Németh changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nice@titanic.nyme.hu | --- Comment #11 from Tamás Németh 2009-05-21 08:30:48 MDT --- (In reply to comment #10)

What is wrong with the screenshot from comment #8?

sda is not part of any multipath device so it's displayed with all it's partitions. sdaa is included in a multipath device and thus only the disk without partitions is shows. That is the expected way. By double-clicking on the disks and selecting the Overview you can check that sda is not used by any device and that sdaa is used by a multipath device.

OK, my /etc/multipath.conf looks like this, now: defaults { user_friendly_names "no" } blacklist { devnode "^sda$" } However, LUN WWN / friendly name mappins are still present in /etc/multipath_bindings. Yesterday I wanted to start yast2/partitinor but it was unable to start in about twelve hours, so I restarted the whole machine, and today yast2/partitioner was finally able to start. If you take a look at the attached image, you can see, that three devices, which are otherwise paths to multipath volumes, can be actually managed! The three devices are: /dev/sdce; /dev/sdcf and /dev/sdcg. They are not shown in the multipath topology, instead 'multipath -d' displays this: carrier4:~ # multipath -d reload: 3600a0b80004808d200000b5548931bee n/a SUN,CSM200_R [size=600G][features=0][hwhandler=1 rdac][n/a] \_ round-robin 0 [prio=8][undef] \_ 7:0:0:9 sdx 65:112 [active][ghost] \_ 5:0:1:9 sdbg 67:160 [active][ghost] \_ round-robin 0 [prio=6][undef] \_ 5:0:0:9 sdr 65:16 [active][ready] \_ 7:0:1:9 sdce 69:32 [undef][ready] reload: 3600a0b80004808d200000b70489bf7f8 n/a SUN,CSM200_R [size=20G][features=0][hwhandler=1 rdac][n/a] \_ round-robin 0 [prio=12][undef] \_ 7:0:0:10 sdaa 65:160 [active][ready] \_ 5:0:1:10 sdbh 67:176 [active][ready] \_ round-robin 0 [prio=2][undef] \_ 5:0:0:10 sdt 65:48 [active][ghost] \_ 7:0:1:10 sdcf 69:48 [undef][ghost] reload: 3600a0b80004807f8000008cb489c01b3 n/a SUN,CSM200_R [size=300G][features=0][hwhandler=1 rdac][n/a] \_ round-robin 0 [prio=12][undef] \_ 5:0:0:11 sdu 65:64 [active][ready] \_ 7:0:1:11 sdcg 69:64 [undef][ready] \_ round-robin 0 [prio=2][undef] \_ 7:0:0:11 sdab 65:176 [active][ghost] \_ 5:0:1:11 sdbi 67:192 [active][ghost] These three multipats seem to be degraded somehow. We also use this fibrechannel storage system with Windows, without any problem. Multipath topology also shows these affected volumes to be in an unusual state (consisting of only three devices insted of four): create: 3600a0b80004808d200000b5548931bee dm-67 SUN,CSM200_R [size=600G][features=1 queue_if_no_path][hwhandler=1 rdac][rw] \_ round-robin 0 [prio=8][active] \_ 7:0:0:9 sdx 65:112 [active][ghost] \_ 5:0:1:9 sdbg 67:160 [active][ghost] \_ round-robin 0 [prio=3][enabled] \_ 5:0:0:9 sdr 65:16 [active][ready] create: 3600a0b80004808d200000b70489bf7f8 dm-70 SUN,CSM200_R [size=20G][features=1 queue_if_no_path][hwhandler=1 rdac][rw] \_ round-robin 0 [prio=12][active] \_ 7:0:0:10 sdaa 65:160 [active][ready] \_ 5:0:1:10 sdbh 67:176 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 5:0:0:10 sdt 65:48 [active][ghost] create: 3600a0b80004807f8000008cb489c01b3 dm-79 SUN,CSM200_R [size=300G][features=1 queue_if_no_path][hwhandler=1 rdac][rw] \_ round-robin 0 [prio=6][enabled] \_ 5:0:0:11 sdu 65:64 [active][ready] \_ round-robin 0 [prio=2][enabled] \_ 7:0:0:11 sdab 65:176 [active][ghost] \_ 5:0:1:11 sdbi 67:192 [active][ghost] For comparison, here is a volume in normal state: 3600a0b80004808d200000b8948b281a6 dm-81 SUN,CSM200_R [size=20G][features=1 queue_if_no_path][hwhandler=1 rdac][rw] \_ round-robin 0 [prio=12][active] \_ 7:0:0:12 sdae 65:224 [active][ready] \_ 5:0:1:12 sdbj 67:208 [active][ready] \_ round-robin 0 [prio=2][enabled] \_ 5:0:0:12 sdv 65:80 [active][ghost] \_ 7:0:1:12 sdch 69:80 [active][ghost] -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User aschnell@novell.com added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c13 Arvin Schnell changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aschnell@novell.com AssignedTo|aschnell@novell.com |hare@novell.com --- Comment #13 from Arvin Schnell 2009-05-21 13:11:45 MDT --- So there's something wrong with multipath on the system. Assigning to maintainer or multipath-tools. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c15 --- Comment #15 from Tamás Németh 2009-05-22 01:13:48 MDT --- Created an attachment (id=293804) --> (http://bugzilla.novell.com/attachment.cgi?id=293804) Output of lsscsi -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c17 Tamás Németh changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nice@titanic.nyme.hu | --- Comment #17 from Tamás Németh 2009-05-22 01:21:17 MDT --- (In reply to comment #14)

Please attach the output of 'lsscsi' and '/var/log/messages'.

Plus you'll have to ensure that the 'host type' of the storage array is set to 'Linux'. Otherwise weird things will happen.

I've attached the files you asked. You seem to know this Sun StorageTek 6140 (rebranded LSI Engenio 3994) storage system (The system integrator who brought it to us had no real experts ;-). In fact I set the host type for every Linux host to be 'AIX failover' in 2008. My reason for doing this was, that I thought that the storage system won't let the host to change paths in order to achieve round robin functionality or failover. I thought the the host type 'Linux' is for host without multipath capability and with a single path to the storage system. What will change if I set the host types to Linux? Won't I experience performance degradation and failover incapability? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c19 --- Comment #19 from Tamás Németh 2009-05-22 03:23:32 MDT --- (In reply to comment #18)

The suggestion here is to set the host type to 'Linux' (if you are not using the 'rdac' hardware handler) or 'Solaris' (if you are using the 'rdac' hardware handler). Otherwise unexpected behaviour will occur as you've just seen.

For this reason I would suggest using 'Solaris' as host type

We do need the 'rdac' hardware handler here, though, as multipath has to send switch-over commands.

Which one of these do you suggest: Solaris (with Traffic Manager) Solaris (with Veritas DMP or other) -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c21 --- Comment #21 from Tamás Németh 2009-05-22 04:53:43 MDT --- Read this please: http://markmail.org/message/zxnzrrteon5slkla#query:storagetek%206140%20rdac%... It seems to me, that the hot type "Linux" is better for the RDAC mode. Until now, I used openSUSE 10.3 on xen.org 2.6.18 kernel, which doesn't support RDAC (AFAIK), so I chose "AIX failover" in orger to activate AVT. Should I switch to "Linux" with openSUSE 11.1 which seems to support RDAC? Anyway, I will try that NOW. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c23 --- Comment #23 from Tamás Németh 2009-05-22 15:04:03 MDT --- (In reply to comment #22)

Yes, please always use hosttype 'Linux'. This actually enables 'AVT', so it's the preferred choice here. And miles better than AIX in any configuration. AIX has a really weird SCSI behaviour which should be avoided here.

So far, I was able to figure out that the 'Linux' host type disables the AVT mode, which means it expects the host to do explicit (e.g. RDAC, which seems to be supported by openSUSE 11.1's kernel) failover instead of implicit. 'Solaris (with Veritas DMP or other)' seems to be the right choice for implicit failover on Linux (older systems). Maybe I should try the instead of 'AIX failover', because when I set the host type to 'Linux', my hosts are almost always unable to finish the boot process! -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c25 --- Comment #25 from Tamás Németh 2009-05-22 18:32:58 MDT --- Created an attachment (id=294036) --> (http://bugzilla.novell.com/attachment.cgi?id=294036) xen boot process stalled -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c27 --- Comment #27 from Tamás Németh 2009-05-22 23:26:09 MDT --- Created an attachment (id=294044) --> (http://bugzilla.novell.com/attachment.cgi?id=294044) our fibrechannel topology You can see our SAN topology here: every host has two HBA cards, and every HBA card has two ports, but only one port per HBA is connected to the fibrechannel network. There are two fibrachannel switches, which are not conected to each other, so there are two distinct fibrechannel fabrics. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c29 --- Comment #29 from Tamás Németh 2009-05-24 16:18:59 MDT --- Just one more info to help you to analyze the problem: The entire Linux system is on a local (non-multipath, of course) hardware RAID array, allt the fibrechannel multipth volumes are only for Xen domUs, which don't start automatically, yet despite this, problems with multipath volumes can keep the system from finishing the boot process. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c31 Tamás Németh changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nice@titanic.nyme.hu | --- Comment #31 from Tamás Németh 2009-05-31 16:13:51 MDT --- Created an attachment (id=295385) --> (http://bugzilla.novell.com/attachment.cgi?id=295385) After pressing ctrl-c (In reply to comment #30)

Have you tried to press 'Control-C' when the boot process stalled?

Yes, pressing ctrl-c lead to make the system to finish the boot process (unusually quickly), but it behaved like it was missing some of the filesystems, or something. I was even unable to log in, because it was complaining about some missing service component. Please take a look at the attached image. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c33 Tamás Németh changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nice@titanic.nyme.hu | --- Comment #33 from Tamás Németh 2009-07-03 05:07:39 MDT --- (In reply to comment #32)

Please look in /dev/.udev/failed. It looks as if some events couldn't be processed before the timeout.

Sorry, I'm not an expert of udev, so, I don't know what /dev/.udev/failed is for. All I can see there is a broken symbolic link: carrier4:/dev/.udev/failed # ls -l /dev/.udev/failed total 0 lrwxrwxrwx 1 root root 46 2009-07-03 13:01 \x2fdevices\x2fplatform\x2fmicrocode\x2ffirmware\x2fmicrocode -> /devices/platform/microcode/firmware/microcode -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c35 Tamás Németh changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nice@titanic.nyme.hu | --- Comment #35 from Tamás Németh 2009-07-03 05:47:53 MDT --- (In reply to comment #34)

... And which is exactly what I was looking for. So there was an microcode uevent which couldn't be handled.

This is why multipath volumes are handled improperly? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c37 --- Comment #37 from Tamás Németh 2009-07-03 06:07:51 MDT --- (In reply to comment #36)

Not, but it's the reason for

...

Finally I set the host type to 'Linux' (non-AVT, RDAC supported), but the xen kernel always(?) fails to load the system. Is stalls at a certain point (see and attached image later). Default kernel also stalls sometimes (seen another attched image), but sometimes succeeds to load the system and then:

And this doesn't have anything to do with multipathing.

Do you mean that the CPU microcode is somehow influenced by the configuration of the external (fibrechannel) storage system? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=495224 User nice@titanic.nyme.hu added comment http://bugzilla.novell.com/show_bug.cgi?id=495224#c39 --- Comment #39 from Tamás Németh 2009-07-03 07:16:38 MDT --- (In reply to comment #38)

No, I mean the issue of a system stalling during boot and any fibrechannel configuration problems are totally unrelated. Hence the former warrants a new bugzilla entry.

OK, I understand, but as you can read, the total boot stalling only happens with a certain (external) storage setting (and with the xen kernel). With a different storage setting even the xen kernel is able to boot the system. My problem is that multipath handling is unreliable, as described in this report and in others also submitted by me. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.