https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c2 Mike Latimer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |eruby@knowledgematters.net --- Comment #2 from Mike Latimer 2013-11-04 16:35:15 UTC --- Can you create and attach a supportconfig archive from dom0 to this bug? That will provide the details I need to verify your configuration, and help with any testing I need to do here. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c4 --- Comment #4 from Mike Latimer 2013-11-06 15:25:52 UTC --- (In reply to comment #3)

I'm pretty sure that supportconfig only ships in the supportutils package in SLES. I don't see it on my OpenSUSE 12.3 server.

Sorry about that. I didn't realize this was not included in 12.3. You can download the Factory version (1.20) at the following link: http://software.opensuse.org/package/supportutils It will install and run just fine under openSUSE.

Is there something I can run in OpenSUSE that will tell you what you want to know?

The problem is, there are quite a few things I would like to know. For example, here is the immediate list of things I would like: /etc/lvm/lvm.conf multipath -ll dmsetup ls --tree lscsi xml file for a guest with this problem I'm sure there are other questions the above output will raise as well, that is why the supportconfig output is so useful. If it is not possible to get that, we can try starting with the above. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c6 Mike Latimer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |eruby@knowledgematters.net --- Comment #6 from Mike Latimer 2013-11-18 20:04:16 UTC --- Thanks for the information. There are a few additional items I would have liked to see from a full supportconfig output, but at least we have a starting point now. There are some MPIO configuration issues in dom0 which need to be addressed. Specifically, the LVM filter (in lvm.conf) is currently at the default setting. This allows for LVM to be activated on a single raw path to a LUN, rather than being forced to only activate on an MPIO device. This can result in failures if LVM decides to do this. To avoid this problem, force LVM to only use MPIO devices with the following filter: filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r/.*/" ] The above filter may not be 100% correct for your environment, as this prevents LVM from being activated on any non-MPIO devices (such as a local disk). If you also need to activate on local disks, add another entry into the filter to also accept that specific device. The next issue is in the configuration for the vm (dir0). In the config file, you are specifying the disk using the following syntax: disk = [ 'phy:/dev/disk/by-id/scsi-2000b0804e0002142,xvda,w' ] Due to the way udev and MPIO works, the above syntax may not always be correct. When the '/dev/disk/by-id/scsi-2000b0804e0002142' device is first created it is a symlink to one raw path to the LUN. Later on, this symlink is remapped to the MPIO device node. As in the LVM case, we really want this to specifically link to a MPIO device node all the time. As such, a more correct device to use would be as follows: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b0804e0002142,xvda,w' ] After resolving the above issues, I am still confused about the original issue... According to dmsetup, you have the following LVM devices for dir0: dir0-swap1 (253:26) dir0-swap0 (253:25) dir0-home (253:22) dir0-var_log (253:31) dir0-opt (253:23) dir0-root (253:24) dir0-usr_local (253:29) dir0-tmp (253:27) dir0-usr (253:28) dir0-var (253:30) All of the above LVs are located on 2000b0804e0002142-part2. Is it safe to assume that 2000b0804e0002142-part1 is a /boot partition on the VM itself? If so, I think I understand your environment - and the problem with it. It is potentially very dangerous for dom0 to be able to activate the LVM devices belonging to a domU. If both dom0 and domU can see the same LV, it is possible for both environments to mount the LV and the risk for corruption is quite high. To avoid this, it is our recommendation to ensure dom0 cannot activate the LVM devices belonging to a guest. The best way to do this would be to blacklist the LUNs from the LVM filter on dom0, so it will not activate the domU devices. Serving the MPIO device to the guest (through phy:/...) should be sufficient for the guest to pick up the device, and activate LVM on it properly (as long as boot.lvm and the LVM filter on the guest is setup properly). Can we change your configuration to this type of model? If you can provide a full supportconfig from dom0 and one domU, I can be more specific about the changes I'd recommend. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c8 --- Comment #8 from Mike Latimer 2013-11-18 22:53:43 UTC --- (In reply to comment #7)

The domU LVs are *never* mounted in the dom0 environment and rarely migrated from one dom0 to another, so maybe that's why this has always worked OK.

Yes, if you are careful, you should be able to get away with this. It is usually safer to ensure dom0 never has access to those devices in the first place.

For dir0, right now /etc/xen/vm/dir0 is using:

/dev/disk/by-id/scsi-2000b0804e0002142

...which is symlinked to ../../dm-3. I changed /etc/xen/vm/dir0 to use:

/dev/disk/by-id/dm-uuid-mpath-2000b0804e0002142

... which is also symlinked to ../../dm-3.

Right now, your udev is synchronized with mpio, so both the scsi-XXX and dm-uuid-mpath-XXX devices should link to ../../dm-3. The problem is that the scsi-XXX devices start out being linked to ../../sdX, and are later remapped to ./../dm-3. If LVM activates before they are remapped, you can have a problem (i.e. LVM activated on a single path, rather than the MPIO device).

I still had the same issue when I tried to start dir0, so I added the domU /dev/disk/by-id/dm-uuid-mpath-* names to the filter list in dom0:

filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r|/dev/disk/by-id/dm-uuid-part2-mpath-2000b0804e0002142|", "r|/dev/disk/by-id/dm-uuid-part2-mpath-2000b0804df002142|", "r|/dev/disk/by-id/dm-uuid-part2-mpath-2000b0804e0002142|", "r/.*/" ]

The problem with the above filter is that rules are processed in order. As the first accept rule will match all the dm-uuid-.*-mpath-.* devices, they will be accepted and never rejected. Changing that accept to after the reject rules will accomplish what you need here. BTW - I usually recommend people use MPIO aliases to name their LUNs, rather than leave that up to LVM. If you do that, you can create a more simple LVM filter that will reject all LUNs that start with 'vm-*' (as an example). Either way should work though. I am still not sure if the boot failure is due to LVM being activated on dom0 or not. For the time being, please try your test again after changing the filter. When the guest still fails to boot, are you left in a working maintenance shell? If so, I'd like to get the output of lsscsi, pvscan, and lvscan from that shell...

What additional output from supportconfig do you need? What flags?

Simply running supportconfig (with no parameters) will provide everything I need. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c10 --- Comment #10 from Earl Ruby 2013-11-19 00:54:16 UTC --- I fixed the filters and rebooted again. This time there was no dir0 /dev/disk/by-id/dm-uuid-*-mpath-* device listed when I ran pvs. I started the dir0 VM and still got the error. The only way I can run supportconfig within the dir0 VM is to rebuild the dir0 VM and run supportconfig from the newly-built VM. As soon as I reboot dom0 the VM will no longer work. So, I tried rebuilding dir0, but now that the LUN is being filtered AutoYast is telling me that it can't find the disk. (Doh!) It looks like I need to: * Remove the domU filters * Reboot dom0 * Rebuild the dir0 VM using the /dev/disk/by-id/dm-uuid-*-mpath-* device * Make sure that the dir0 VM has the new lvm.conf filter * Run supportconfig within the dir0 VM before I try rebooting dom0 Does that sound correct? FWIW, the output I sent from supportconfig was generated by running it with no parameters from the dom0. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c12 Earl Ruby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|eruby@knowledgematters.net | --- Comment #12 from Earl Ruby 2013-11-19 23:45:51 UTC --- Right after rebuilding dir0 using a fresh LUN: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b0804e1002142,xvda,w' ] Logging into dir0 shows: dir0:~ # ls -al /dev/xvda* brw-rw---- 1 root disk 202, 0 Nov 19 23:05 /dev/xvda brw-rw---- 1 root disk 202, 1 Nov 19 23:04 /dev/xvda1 brw-rw---- 1 root disk 202, 2 Nov 19 23:04 /dev/xvda2 dir0:~ # pvs PV VG Fmt Attr PSize PFree /dev/xvda2 dir0 lvm2 a-- 141.00g 117.38g dir0:~ # vgdisplay --- Volume group --- VG Name dir0 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 11 VG Access read/write VG Status resizable MAX LV 0 Cur LV 10 Open LV 10 Max PV 0 Cur PV 1 Act PV 1 VG Size 141.00 GiB PE Size 128.00 MiB Total PE 1128 Alloc PE / Size 189 / 23.62 GiB Free PE / Size 939 / 117.38 GiB VG UUID gBorbt-Z4tA-8Nbf-k3xu-EVk3-tx7O-mcPwO3 dir0:~ # ls -al /dev/disk/by-id/ total 0 drwxr-xr-x 2 root root 440 Nov 19 23:03 . drwxr-xr-x 4 root root 80 Nov 19 23:03 .. lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-home -> ../../dm-0 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-opt -> ../../dm-1 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-root -> ../../dm-2 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-swap0 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-swap1 -> ../../dm-4 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-tmp -> ../../dm-5 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-usr -> ../../dm-6 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-usr_local -> ../../dm-7 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-var -> ../../dm-8 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-name-dir0-var_log -> ../../dm-9 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO32aC5o1iY2Eo8NZTuieBM1MAchKljAcsq -> ./../dm-3 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3DeOeAcN9zN1421cXk7f8bf12I2zg298Y -> ./../dm-9 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3GGo1Eo2YIKcY3xpBqQNHkerGf6dDHZj5 -> ./../dm-7 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3OfNbgeIZU1M3g7rdIsPRuvthA9rTz5lY -> ./../dm-8 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3SBWsbViRejUlu41PiBRsnC6UpNu3c9O3 -> ./../dm-4 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3U8g2QMnGksvhf4mO87qrIvCPT81VFBoD -> ./../dm-5 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3bopLcKyNr8rjI73jJ7Wv1idQFw9fBafJ -> ./../dm-0 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3cWg0aHAsjxQxjk4XlFG709zg7Ip5rzYC -> ./../dm-6 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3dYfbhPVQUnDy8HrDtr03cX2KrpHwLjOX -> ./../dm-2 lrwxrwxrwx 1 root root 10 Nov 19 23:04 dm-uuid-LVM-gBorbtZ4tA8Nbfk3xuEVk3tx7OmcPwO3tPRt3MfDA44w0X5CsAFcpjMZcOiqtIkQ -> ./../dm-1 After I reboot dom0 and I try to start the dir0 VM the console spews the error messages: Volume group "dir0" not found Reading all physical volumes. This may take a while... No volume groups found PARTIAL MODE. Incomplete logical volumes will be processed. .. over and over for about 5 minutes. Then it says: Volume group "dir0" not found Could not find /dev/dir0/root. Want me to fall back to /dev/dir0/root? (Y/n) I hit "Y", and it starts spewing the same error messages for about 5 more minutes, then I get: Volume group "dir0" not found PARTIAL MODE. Incomplete logical volumes will be processed. Volume group "dir0" not found not found -- exiting to /sbin/sulogin Give root password for login: Login incorrect. Give root password for login: Login incorrect. Give root password for login: Login incorrect. Give root password for login: The root password does not work. If I destroy the VM, create it, and answer "N" when I get the "Want me to fall back to /dev/dir0/root? (Y/n)" question I get: Could not find /dev/dir0/root. Want me to fall back to /dev/dir0/root? (Y/n) n not found -- exiting to /sbin/sulogin Give root password for login: Login incorrect. Give root password for login: The root password does not work. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c14 Mike Latimer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |eruby@knowledgematters.net --- Comment #14 from Mike Latimer 2013-11-20 21:47:34 UTC --- Thanks for those additional details, Earl. I just finished setting this up in my environment, where I was able to duplicate the problem. The root cause for this is actually on the MPIO side. By default, MPIO will create a multipath map for partitions residing on MPIO disks. These maps prevent Xen guests from fully accessing the partitions, and strange things can be seen. I should have guessed this earlier, but the original description did not match my memory of this issue, so it took a little longer to track down. This issue and resolution is documented in the following TID: https://www.novell.com/support/kb/doc.php?id=7011590 The basic overview is that you need to enable the "no_partitions" feature for the MPIO configuration matching your SAN. In my case, this was through the following setting in /etc/multipath.conf: devices { device { vendor "DGC" product "RAID 5" features "1 no_partitions" } } After adding this section to multipath.conf, make sure the setting is taking effect by checking the configuration in `multipath -t`, or using `multipath -v3` when building the maps. With this setting, when the MPIO maps are rebuilt, corresponding maps for the partitions will not be created. In other words, the following devices marked as BAD should not be created: /dev/disk/by-id/dm-uuid-mpath-2000b0804e1002142 <--- OK /dev/disk/by-id/dm-uuid-part1-mpath-2000b0804e1002142 <--- BAD /dev/disk/by-id/dm-uuid-part2-mpath-2000b0804e1002142 <--- BAD In my testing, if the above 'partX' devices existed, I can see the problem you are having. With no_partitions in effect, these devices are not created, and the domU will reboot properly. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c16 Mike Latimer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |eruby@knowledgematters.net --- Comment #16 from Mike Latimer 2013-11-21 04:49:07 UTC --- (In reply to comment #15)

I added the features flag to /etc/multipath.conf:

devices { device { vendor "Pillar" product "Axiom.*" path_grouping_policy "group_by_prio" path_checker "tur" hardware_handler "0" prio "alua" rr_weight "uniform" features "1 no_partitions" } }

The problem is likely due to the wildcard in the product field. According to the MPIO developers, wildcards are not supported in those fields. Based on my own testing, using wildcards in those fields can work, but is also inconsistent. It depends on the default entries already in the internal tables. If changing that does not help, I should have some more advice with the following (complete) details: - /etc/multipath.conf - Output of lsscsi - Output of multipath -t - Output of multipath -v3 -d - Output of multipath -ll FYI - Most of the above is in a supportconfig output. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c18 --- Comment #18 from Mike Latimer 2013-11-22 01:45:28 UTC --- (In reply to comment #17)

I got the whole "device { ... }" entry by running "multipath -t | less" and looking for "Pillar". There was only one entry, so I modified that. Many of the entries shipped with OpenSUSE include wildcards in the "product" field. Are they all wrong then?

No. The internal tables are allowed to use wildcard characters. It is only when adding an entry manually into /etc/multipath.conf that the wildcard characters may not work properly.

I figured out a series of commands to type last night and got it working. I posted a follow-up message with my notes but don't see it here. When I go home tonight I'll check my computer and see if they're still on the screen. If not I'll have to reconstruct what I did. (I'm still logged in at home, so all of the commands are on my screen there.)

After getting no_partitions to work, did the VMs start properly?

I'll also run supportconfig for you again and add a second supportconfig dump to this ticket.

Thanks. I'll watch for the rest of the details then. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c20 Earl Ruby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|eruby@knowledgematters.net | --- Comment #20 from Earl Ruby 2013-12-03 00:15:34 UTC --- With 3 VMs: dir0: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b0804e1002142,xvda,w' ] rtr1: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b0804e2002142,xvda,w' ] www4: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b080432002142,xvda,w' ] Added the following filter to /etc/lvm/lvm.conf: filter = [ "r|.*2000b0804e1002142.*|", "r|.*2000b0804e2002142.*|", "r|.*2000b080432002142.*|", "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r|.*|" ] Edited /etc/multipath.conf to contain both the product with a wildcard name and the actual name. If I *just* add the actual name the wildcard name still shows up without the 'features "1 no_partitions"' line, so I added both so it's going to match one way or another: ------------------------------------------------------ defaults { # If all paths are down just queue up ios (not much else to do). no_path_retry queue } devices { device { vendor "Pillar" product "Axiom.*" path_grouping_policy "group_by_prio" path_checker "tur" hardware_handler "0" prio "alua" rr_weight "uniform" features "1 no_partitions" } device { vendor "Pillar" product "Axiom 500" path_grouping_policy "group_by_prio" path_checker "tur" hardware_handler "0" prio "alua" rr_weight "uniform" features "1 no_partitions" } } multipaths { # dir0 multipath { wwid 2000b0804e1002142 features "1 no_partitions" } # rtr1 multipath { wwid 2000b0804e2002142 features "1 no_partitions" } # www4 multipath { wwid 2000b080432002142 features "1 no_partitions" } } ------------------------------------------------------ After rebooting I only see one VG: # vgdisplay --- Volume group --- VG Name switch System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 13 VG Access read/write VG Status resizable MAX LV 0 Cur LV 10 Open LV 10 Max PV 0 Cur PV 1 Act PV 1 VG Size 60.00 GiB PE Size 128.00 MiB Total PE 480 Alloc PE / Size 189 / 23.62 GiB Free PE / Size 291 / 36.38 GiB VG UUID lUzwRa-dXRO-dTRk-zukb-J4WM-2cAQ-yVq0jy After rebooting "multipath -ll" shows paths to all 4 PVs: # multipath -ll 2000b0804e1002142 dm-3 Pillar ,Axiom 500 size=142G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:2:3 sdd 8:48 active ready running | `- 2:0:3:3 sdp 8:240 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:3:3 sdh 8:112 active ready running `- 2:0:2:3 sdl 8:176 active ready running 2000b0804e2002142 dm-2 Pillar ,Axiom 500 size=121G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:2:2 sdc 8:32 active ready running | `- 2:0:3:2 sdo 8:224 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:3:2 sdg 8:96 active ready running `- 2:0:2:2 sdk 8:160 active ready running 2000b080432002142 dm-1 Pillar ,Axiom 500 size=81G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:3:1 sdf 8:80 active ready running | `- 2:0:2:1 sdj 8:144 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:2:1 sdb 8:16 active ready running `- 2:0:3:1 sdn 8:208 active ready running 2000b080452002142 dm-0 Pillar ,Axiom 500 size=60G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:3:0 sde 8:64 active ready running | `- 2:0:2:0 sdi 8:128 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:2:0 sda 8:0 active ready running `- 2:0:3:0 sdm 8:192 active ready running *** At this point, if I try to create a VM using "xm create dir0 -c" I get the same error and the VM will not start. *** However if I run "multipath -F" then "multipath -ll" I see: # multipath -F Dec 02 23:30:59 | 2000b080452002142-part2: map in use Dec 02 23:30:59 | failed to remove multipath map 2000b080452002142 # multipath -ll 2000b080452002142 dm-0 Pillar ,Axiom 500 size=60G features='1 no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:3:0 sde 8:64 active ready running | `- 2:0:2:0 sdi 8:128 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:2:0 sda 8:0 active ready running `- 2:0:3:0 sdm 8:192 active ready running *** At this point, if I try to create a VM using "xm create dir0 -c" I get a different error:*** # xm create dir0 -c Using config file "/etc/xen/vm/dir0". Error: Disk '/dev/disk/by-id/dm-uuid-mpath-2000b0804e1002142' isn't accessible .. which makes sense, since there's no path to the PV. At this point, If I type "multipath -r" I get some errors and the paths come back. If I type "multipath -ll" it once again shows the same paths as it showed after rebooting: # multipath -r Dec 02 23:51:55 | cciss/c0d0: HDIO_GETGEO failed with 6 Dec 02 23:51:55 | nbd0: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd1: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd10: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd11: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd12: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd13: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd14: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd15: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd2: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd3: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd4: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd5: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd6: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd7: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd8: HDIO_GETGEO failed with 25 Dec 02 23:51:55 | nbd9: HDIO_GETGEO failed with 25 reload: 2000b080452002142 undef Pillar ,Axiom 500 size=60G features='2 no_partitions queue_if_no_path' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=130 status=undef | |- 1:0:2:0 sda 8:0 active ready running | `- 2:0:3:0 sdm 8:192 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 1:0:3:0 sde 8:64 active ready running `- 2:0:2:0 sdi 8:128 active ready running reload: 2000b080432002142 undef Pillar ,Axiom 500 size=81G features='2 no_partitions queue_if_no_path' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=130 status=undef | |- 1:0:3:1 sdf 8:80 active ready running | `- 2:0:2:1 sdj 8:144 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 1:0:2:1 sdb 8:16 active ready running `- 2:0:3:1 sdn 8:208 active ready running reload: 2000b0804e2002142 undef Pillar ,Axiom 500 size=121G features='2 no_partitions queue_if_no_path' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=130 status=undef | |- 1:0:2:2 sdc 8:32 active ready running | `- 2:0:3:2 sdo 8:224 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 1:0:3:2 sdg 8:96 active ready running `- 2:0:2:2 sdk 8:160 active ready running reload: 2000b0804e1002142 undef Pillar ,Axiom 500 size=142G features='2 no_partitions queue_if_no_path' hwhandler='0' wp=undef |-+- policy='service-time 0' prio=130 status=undef | |- 1:0:3:3 sdh 8:112 active ready running | `- 2:0:2:3 sdl 8:176 active ready running `-+- policy='service-time 0' prio=10 status=undef |- 1:0:2:3 sdd 8:48 active ready running `- 2:0:3:3 sdp 8:240 active ready running # multipath -ll 2000b0804e1002142 dm-3 Pillar ,Axiom 500 size=142G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:3:3 sdh 8:112 active ready running | `- 2:0:2:3 sdl 8:176 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:2:3 sdd 8:48 active ready running `- 2:0:3:3 sdp 8:240 active ready running 2000b0804e2002142 dm-2 Pillar ,Axiom 500 size=121G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:2:2 sdc 8:32 active ready running | `- 2:0:3:2 sdo 8:224 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:3:2 sdg 8:96 active ready running `- 2:0:2:2 sdk 8:160 active ready running 2000b080432002142 dm-1 Pillar ,Axiom 500 size=81G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:3:1 sdf 8:80 active ready running | `- 2:0:2:1 sdj 8:144 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:2:1 sdb 8:16 active ready running `- 2:0:3:1 sdn 8:208 active ready running 2000b080452002142 dm-0 Pillar ,Axiom 500 size=60G features='2 queue_if_no_path no_partitions' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=130 status=active | |- 1:0:2:0 sda 8:0 active ready running | `- 2:0:3:0 sdm 8:192 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 1:0:3:0 sde 8:64 active ready running `- 2:0:2:0 sdi 8:128 active ready running *** However, now when I run "xm create dir0 -c" it works! *** I can also now create the other two VMs, rtr1 and www4, and they also boot up. I'm not yet sure exactly which of the lvm.conf and multipath.conf restrictions are necessary, but I can tell you that the following does work: * Restrict the volumes from being activated by adding them to lvm.conf's filters. * Add 'features "1 no_partitions"' to the devices{} and multipaths{} sections of /etc/multipath.conf. * After rebooting, run "multipath -F", then "multipath -r" * Start the VMs I've used "multipath -r" before to update path PV information after enlarging a PV on the SAN, so I understand that, and -f will "flush all unused multipath device maps", but between the two of them they're resetting or fixing something that is broken after reboot. So the upshot is that I now have steps I can use to get my VMs working, but I'm not sure why they work. (I'd like to know why they work.) It also seems like I'm doing a lot of extra work to get Xen to perform basic functions, like booting a VM. I looked through the config options to see if I could find a "Don't break just because volumes are visible to dom0" setting, but it doesn't look like this is configurable. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c22 Mike Latimer changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |eruby@knowledgematters.net --- Comment #22 from Mike Latimer 2013-12-03 16:09:27 UTC --- (In reply to comment #21)

Why is it behaving differently at boot time?

The likely explanation is that you have multipath included in your initrd, so your changes in /etc/multipath.conf have not yet been added to the multipath.conf that is used from the initrd. You can add your updated multipath.conf to your initrd using `mkinitrd -f multipath`. (If your system partitions are not on a multipath device, the other option is to remove mpio from your initrd.) Regarding the multiple entries in /etc/multipath.conf, I would recommend that you determine which one is actually being used. You can do this by commenting one out, and looking at the output of `multipath -v3 -d`. If the 'no_partitions' feature is seen during the map assembly, you will know it read the entry correctly. Another way to investigate this is through `multipath -t`. If you look at the full table, you really only want one device entry to match your SAN vendor/product. When I have tested wilcards in my multipath.conf device entries, I usually see two such entries end up in the full table, then it is uncertain which one will be used.

I've used "multipath -r" before to update path PV information after enlarging a PV on the SAN, so I understand that, and -f will "flush all unused multipath device maps", but between the two of them they're resetting or fixing something that is broken after reboot.

I think the main thing that is happening here is that you are using the initrd multipath.conf during startup, which does not have the no_partitions feature, so the partition maps are being created. Later, after the server is up, you are flushing and rebuilding the maps using the /etc/multipath.conf that does have no_partitions, so the partition maps are being removed, and not recreated. Can you rebuild the initrd so the same multipath.conf is in both places and let me know if that helps? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c24 --- Comment #24 from Mike Latimer 2013-12-04 03:56:37 UTC --- (In reply to comment #23)

Oh, of course. I wasn't thinking about initrd.

So I rebuilt initrd. Then rebooted.

Of course now the "no_partitions" feature applies to all Pillar LUNs at boot time, including the dom0 boot LUN, so now it won't boot.

Sorry for the hassle. Can I please get that full supportconfig output? That would help me come up with a complete working configuration... However, I've made a few more guesses, and the rest of this comment should help.

At this point it booted but wouldn't recognize the "switch" VG. Probably something to do with the changes to the lvm.conf filter.

Yes, I'm sure the filter caused that. However, in an MPIO environment, it is incorrect (and potentially damaging) to use non-MPIO paths. (Those duplicate PV messages are a clear indication of the problem.) So, I would recommend sticking to the original filter I recommended in comment #6. I appreciate all the efforts, and I think we're close to a working solution. The key points are that we need a solution that does the following: multipath - no_partitions for everything except the 'switch' VG LUN LVM - An LVM filter which only activates on MPIO partitions Xen guest - A phy disk connection that utilizes the MPIO path The following settings should allow this: /etc/lvm/lvm.conf: filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r/.*/" ] /etc/multipath.conf: defaults { # If all paths are down just queue up ios (not much else to do). no_path_retry queue } devices { device { vendor "Pillar" product "Axiom 500" path_grouping_policy "group_by_prio" path_checker "tur" hardware_handler "0" prio "alua" rr_weight "uniform" features "1 no_partitions" } } multipaths { # switch multipath { wwid 2000b080452002142 features "0" } } Sample xen guest configuration: disk = [ 'phy:/dev/disk/by-id/dm-uuid-mpath-2000b0804e1002142,xvda,w' ] In the above multipath.conf, you could also use your method from the previous comment, but as long as no_partitions takes effect from the device setting, it can be disabled on an individual LUN using my syntax above. With only a handful of LUNs either approach would work. As we already encountered, after making changes to lvm.conf or multipath.conf, rebuild the initrd so we have matching configurations in both places. The above configuration really should work. If you continue to run into problems, I will need either a supportconfig or remote access to track down the root cause. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c25 Earl Ruby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|eruby@knowledgematters.net | --- Comment #25 from Earl Ruby 2013-12-04 20:47:58 UTC --- I like your last configuration setup much better than mine. Much simpler to implement, and I don't have to change it every time I add a VM. I changed the files the way you show them, ran mkinitrd, and rebooted. Everything works fine now. No duplicate PV messages when I run vgdisplay, only the "switch" volume appears in dom0, and all VMs boot just fine. Now I just have to modify our install scripts so that all of our new Xen servers are built this way. Thanks for all of your help! Any idea what prompted the Xen team to make this change? It certainly makes working with Xen a whole lot more difficult. If you hadn't helped me I probably would have switched over to KVM. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=848754 https://bugzilla.novell.com/show_bug.cgi?id=848754#c27 --- Comment #27 from Earl Ruby 2013-12-11 01:38:56 UTC --- Ticket https://www.novell.com/support/kb/doc.php?id=7011590 makes a pretty clear case that this problem was introduced in between SLES 11 SP1 and SLES 11 SP2. I don't know what the equivalent version numbers are in OpenSUSE, but we've been using Xen with MPIO and LVM-per-VM since OpenSUSE 10.2 and it was working fine until we started upgrading our Xen dom0 servers from OpenSUSE 11.4 to 12.3. The idea that someone would need to make this many arcane configuration changes to get Xen working seems more like a bug than a configuration issue to me. If you shipped a version of YaST that built a GRUB menu that pointed to the wrong kernel, you could edit the menu configuration and get your server to boot, but most people would still call that a bug. It would be nice if the installer either handled this for you or if someone rolled back the changes in SLES 11 SP2 that make these configuration changes necessary. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.