[opensuse] System hangs when enabling multipathd with iscsi drivers on OpenSuse 13.2
Hi. We are facing a big trouble when trying to get our new Linux box boot with the multipathd enabled. With multipathd enabled, the system gets into a loop when booting, after some time it asks for the password to get into maintenance mode. As we are using ICSSI drivers it depends on the network to be fully working. But when UDEV starts and sees that multipath is enabled it tries to get the device online. Since we don't have the network configured already it keeps trying and this way the system won't start. This is the relevant part of the log: Oct 26 09:27:10 jupiter mtp-probe[1284]: checking bus 1, device 3: "/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4" Oct 26 09:27:10 jupiter mtp-probe[1284]: bus: 1, device: 3 was not an MTP device Oct 26 09:27:10 jupiter kernel: device-mapper: multipath service-time: version 0.2.0 loaded Oct 26 09:27:10 jupiter kernel: device-mapper: table: 254:0: multipath: error getting device Oct 26 09:27:10 jupiter kernel: device-mapper: ioctl: error adding target to table Oct 26 09:27:11 jupiter lvm[1340]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it! Oct 26 09:27:11 jupiter lvm[1340]: No volume groups found Oct 26 09:27:11 jupiter lvm[1344]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it! Oct 26 09:27:11 jupiter lvm[1344]: No volume groups found Oct 26 09:28:40 jupiter systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-3485b99d\x2da022\x2d4fe1\x2da1cb\x2df35360ff6e7b.device. Oct 26 09:28:40 jupiter systemd[1]: Dependency failed for /var. If we disable the service multipathd the system gets up. Here is the log of the system with multipathd disabled [ 15.921844] bonding: bond0: link status definitely up for interface em4, 1000 Mbps full duplex [ 17.760907] Loading iSCSI transport class v2.0-870. [ 17.791444] iscsi: registered transport (tcp) [ 23.353112] systemd[1]: Cannot add dependency job for unit cyrus.service, ignoring: Unit cyrus.service failed to load: No such file or directory. [ 23.363046] systemd[1]: Cannot add dependency job for unit ldap.service, ignoring: Unit ldap.service failed to load: No such file or directory. [ 24.325477] scsi11 : iSCSI Initiator over TCP/IP [ 24.336726] scsi12 : iSCSI Initiator over TCP/IP [ 24.348961] scsi 11:0:0:0: Direct-Access DELL MD32xxi 0820 PQ: 0 ANSI: 5 [ 24.349124] scsi 12:0:0:0: Direct-Access DELL MD32xxi 0820 PQ: 0 ANSI: 5 [ 24.349992] sd 11:0:0:0: Attached scsi generic sg2 type 0 [ 24.350404] sd 12:0:0:0: Attached scsi generic sg3 type 0 [ 24.352235] sd 11:0:0:0: [sdb] 1048576000 512-byte logical blocks: (536 GB/500 GiB) [ 24.352442] scsi 11:0:0:1: Direct-Access DELL MD32xxi 0820 PQ: 0 ANSI: 5 [ 24.352472] sd 12:0:0:0: [sdc] 1048576000 512-byte logical blocks: (536 GB/500 GiB) [ 24.352606] scsi 12:0:0:1: Direct-Access DELL MD32xxi 0820 PQ: 0 ANSI: 5 [ 24.353557] sd 11:0:0:1: Attached scsi generic sg4 type 0 [ 24.353910] sd 12:0:0:0: [sdc] Write Protect is off [ 24.353914] sd 12:0:0:0: [sdc] Mode Sense: 83 00 10 08 [ 24.353934] sd 12:0:0:1: Attached scsi generic sg5 type 0 [ 24.353978] sd 11:0:0:0: [sdb] Write Protect is off [ 24.353983] sd 11:0:0:0: [sdb] Mode Sense: 83 00 10 08 [ 24.354337] sd 12:0:0:1: [sde] 209715200 512-byte logical blocks: (107 GB/100 GiB) [ 24.354584] sd 12:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 24.354823] scsi 12:0:0:31: Direct-Access DELL Universal Xport 0820 PQ: 0 ANSI: 5 [ 24.356067] sd 11:0:0:1: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB) [ 24.356181] sd 12:0:0:1: [sde] Write Protect is off [ 24.356188] sd 12:0:0:1: [sde] Mode Sense: 83 00 10 08 [ 24.356239] scsi 12:0:0:31: Attached scsi generic sg6 type 0 [ 24.356384] scsi 11:0:0:31: Direct-Access DELL Universal Xport 0820 PQ: 0 ANSI: 5 [ 24.356791] sd 12:0:0:1: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 24.358052] sd 11:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 24.359166] scsi 11:0:0:31: Attached scsi generic sg7 type 0 [ 24.359347] sdc: sdc1 [ 24.359506] sd 11:0:0:1: [sdd] Write Protect is off [ 24.359511] sd 11:0:0:1: [sdd] Mode Sense: 83 00 10 08 [ 24.360732] sd 11:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 24.362594] sd 12:0:0:0: [sdc] Attached SCSI disk [ 24.363809] sde: unknown partition table [ 24.364069] sdb: sdb1 [ 24.365518] sdd: unknown partition table [ 24.366318] sd 12:0:0:1: [sde] Attached SCSI disk [ 24.367101] sd 11:0:0:0: [sdb] Attached SCSI disk [ 24.368706] sd 11:0:0:1: [sdd] Attached SCSI disk [ 29.442051] send_arp uses obsolete (PF_INET,SOCK_PACKET) Note that the devices got up after the network being up. If then we start the multipathd later by hand, the discs shows up on the system. [ 1700.313701] device-mapper: multipath service-time: version 0.2.0 loaded [ 1700.313978] device-mapper: table: 254:0: multipath: error getting device [ 1700.313982] device-mapper: ioctl: error adding target to table [ 1700.337922] sd 11:0:0:0: rdac: LUN 0 (IOSHIP) (unowned) [ 1700.338877] sd 12:0:0:0: rdac: LUN 0 (IOSHIP) (owned) [ 1700.339829] sd 12:0:0:1: rdac: LUN 1 (IOSHIP) (unowned) [ 1700.340726] sd 11:0:0:1: rdac: LUN 1 (IOSHIP) (owned) [ 1700.340731] rdac: device handler registered This issue do not happens on the Opensuse 12.3 version. It looks like that there is a bug with the new system/multipathd packages. This issue can be easily replicated, one just have to install the multipath tools and setup a host as a iscsi initiator. Please, could you help on solving this issue. Best regards, Carlos Xavier. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos Xavier wrote:
Hi.
We are facing a big trouble when trying to get our new Linux box boot with the multipathd enabled. With multipathd enabled, the system gets into a loop when booting, after some time it asks for the password to get into maintenance mode. As we are using ICSSI drivers it depends on the network to be fully working. But when UDEV starts and sees that multipath is enabled it tries to get the device online. Since we don't have the network configured already it keeps trying and this way the system won't start.
Hi Carlos I have not tried this with 13.2, but I did see a looping multipathd at some point: http://lists.opensuse.org/opensuse-factory/2015-09/msg00503.html I'm sorry that's not of much help. Maybe I'll try and install 13.2 with iSCSI. -- Per Jessen, Zürich (13.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Thank you very much for your fast reply. Searching the Internet I have found something related to this issue on Ubuntu at this thread https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1431650 It seems they have made a path to solve this. Regards, Carlos Xavier.
-----Mensagem original----- De: Per Jessen [mailto:per@computer.org] Enviada em: segunda-feira, 26 de outubro de 2015 13:40 Para: opensuse@opensuse.org Assunto: Re: [opensuse] System hangs when enabling multipathd with iscsi drivers on OpenSuse 13.2
Carlos Xavier wrote:
Hi.
We are facing a big trouble when trying to get our new Linux box boot with the multipathd enabled. With multipathd enabled, the system gets into a loop when booting, after some time it asks for the password to get into maintenance mode. As we are using ICSSI drivers it depends on the network to be fully working. But when UDEV starts and sees that multipath is enabled it tries to get the device online. Since we don't have the network configured already it keeps trying and this way the system won't start.
Hi Carlos
I have not tried this with 13.2, but I did see a looping multipathd at some point:
http://lists.opensuse.org/opensuse-factory/2015-09/msg00503.html
I'm sorry that's not of much help. Maybe I'll try and install 13.2 with iSCSI.
-- Per Jessen, Zürich (13.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi Per. I did not gave up on OpenSus3 13.2 with multipath and continued my search for a solution for this issue. Then I found this thread about the bug on Debian https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799781 At some point we have a post from Tero Marttila <tero.marttila@paivola.fi> that says "The slow boot with systemd-udev timeouts killing deadlocked multipath workers no longer occurs with 0.5.0+git0.770e6d0d-3, since multipath is no longer invoked from udev:
multipath-tools (0.5.0+git0.770e6d0d-3) unstable; urgency=medium
* [fd43c41] Drop udev rule to invoke multipath per path. Thanks to Apollon Oikonomopoulos (Closes: #580972)
I can confirm that a similar workaround (disabling /lib/udev/rules.d/60-multipath.rules) fixes slow boot times on jessie, synchronizing on the /etc/init.d/multipath-tools service (with #776214 systemd Before=remote-fs-pre.target modifications) instead." So I decided to test this workaround. I located the file and disabled it with jupiter:~ # locate multipath.rules /usr/lib/udev/rules.d/56-multipath.rules jupiter:~ # jupiter:~ # more /usr/lib/udev/rules.d/56-multipath.rules # Set DM_MULTIPATH_DEVICE_PATH if the device should be handled by multipath SUBSYSTEM!="block", GOTO="end_mpath" ENV{MPATH_SBIN_PATH}="/sbin" TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin" SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd*|dasd*", \ ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -i -u %k", \ ENV{DM_MULTIPATH_DEVICE_PATH}="1" \ ENV{ID_FS_TYPE}="none" \ ENV{SYSTEMD_READY}="0" LABEL="end_mpath" jupiter:~ # jupiter:~ # jupiter:~ # mv /usr/lib/udev/rules.d/56-multipath.rules /usr/lib/udev/rules.d/56-multipath.rules_disabled jupiter:~ # jupiter:~ # systemctl enable multipathd jupiter:~ # jupiter:~ # shutdown -r -t10 now And Voila!!! The system boot up flawlessly, everything worked as expected and the multipath drivers were available. I don't know how or to whom address this issue/solution to have it fixed on OpenSuse 13.2 Regards, Carlos Xavier.
-----Mensagem original----- De: Per Jessen [mailto:per@computer.org] Enviada em: segunda-feira, 26 de outubro de 2015 13:40 Para: opensuse@opensuse.org Assunto: Re: [opensuse] System hangs when enabling multipathd with iscsi drivers on OpenSuse 13.2
Carlos Xavier wrote:
Hi.
We are facing a big trouble when trying to get our new Linux box boot with the multipathd enabled. With multipathd enabled, the system gets into a loop when booting, after some time it asks for the password to get into maintenance mode. As we are using ICSSI drivers it depends on the network to be fully working. But when UDEV starts and sees that multipath is enabled it tries to get the device online. Since we don't have the network configured already it keeps trying and this way the system won't start.
Hi Carlos
I have not tried this with 13.2, but I did see a looping multipathd at some point:
http://lists.opensuse.org/opensuse-factory/2015-09/msg00503.html
I'm sorry that's not of much help. Maybe I'll try and install 13.2 with iSCSI.
-- Per Jessen, Zürich (13.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
28.10.2015 00:32, Carlos Xavier пишет:
I don't know how or to whom address this issue/solution to have it fixed on OpenSuse 13.2
To openSUSE bugzilla of course. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (3)
-
Andrei Borzenkov -
Carlos Xavier -
Per Jessen