LVM inactive after upgrading from 15.1 to 15.2
Hello everyone, This is my first message to this list (I sometimes post in Factory list). Today I dist-upgraded my home server from 15.1 to 15.2. After reboot the server failed to boot due to an LVM volume not mounted (it is an extra volume, root is on standalone disk). From the maintenance console, I can enable the volume with lvchange -ay /dev/lvm0/storage and mount it correctly. I removed the mount point from /etc/fstab and the server was booted. Then I can again activate the volume and mounted but after reboot it is again inactive. romantsa:~ # lvscan inactive '/dev/lvm0/storage' [2,17 TiB] inherit I have spotted this on journal logs. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sda. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sda: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdb. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdb: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdc. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdc: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdd. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdd: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde1. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde1: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde2. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde2: Failed to get external handle [udev]. Don't know if it has to do with problem but after searching for many hours I haven't managed to find a solution so the volume can be activated on boot and mounted by fstab. I have opened a thread on forums.opensuse.org and will post any solution also there. Any hints or ideas on where to look? Thanks in advance.
W dniu 30.01.2021 o 17:45, Stratos Zolotas pisze:
Hello everyone,
This is my first message to this list (I sometimes post in Factory list). Today I dist-upgraded my home server from 15.1 to 15.2. After reboot the server failed to boot due to an LVM volume not mounted (it is an extra volume, root is on standalone disk).
From the maintenance console, I can enable the volume with
lvchange -ay /dev/lvm0/storage
and mount it correctly.
I removed the mount point from /etc/fstab and the server was booted. Then I can again activate the volume and mounted but after reboot it is again inactive.
romantsa:~ # lvscan inactive '/dev/lvm0/storage' [2,17 TiB] inherit
I have spotted this on journal logs.
Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sda. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sda: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdb. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdb: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdc. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdc: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sdd. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sdd: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde1. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde1: Failed to get external handle [udev]. Ιαν 30 17:31:15 romantsa lvm[480]: Udev database has incomplete information about device /dev/sde2. Ιαν 30 17:31:15 romantsa lvm[480]: /dev/sde2: Failed to get external handle [udev].
Don't know if it has to do with problem but after searching for many hours I haven't managed to find a solution so the volume can be activated on boot and mounted by fstab.
I have opened a thread on forums.opensuse.org and will post any solution also there.
Any hints or ideas on where to look?
Thanks in advance.
Could it be this? https://bugzilla.opensuse.org/show_bug.cgi?id=1181432
On Sat, Jan 30, 2021 at 7:03 PM Adam Mizerski <adam@mizerski.pl> wrote:
Could it be this? https://bugzilla.opensuse.org/show_bug.cgi?id=1181432
I'm not experiencing this in chroot but by reading it maybe the root cause is the same... cannot say although...
Don't know if it helps to debug but the lvm volume has 4 block devices and I can see three systemd services for activation... maybe something is missing there. romantsa:~ # pvscan PV /dev/sdb VG lvm0 lvm2 [465,76 GiB / 0 free] PV /dev/sdd VG lvm0 lvm2 [596,17 GiB / 0 free] PV /dev/sda VG lvm0 lvm2 [465,76 GiB / 0 free] PV /dev/sdc VG lvm0 lvm2 [698,63 GiB / 0 free] Total: 4 [2,17 TiB] / in use: 4 [2,17 TiB] / in no VG: 0 [0 ] romantsa:~ # systemctl | grep lvm lvm2-monitor.service loaded activ e exited Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling lvm2-pvscan@8:0.service loaded activ e exited LVM event activation on device 8:0 lvm2-pvscan@8:16.service loaded activ e exited LVM event activation on device 8:16 lvm2-pvscan@8:32.service loaded activ e exited LVM event activation on device 8:32 system-lvm2\x2dpvscan.slice loaded activ e active system-lvm2\x2dpvscan.slice lvm2-lvmpolld.socket loaded activ e listening LVM2 poll daemon socket I suspect the issue has to do with something on systemd but can't say why it broke witht the dist-upgrade.
I think the issue is on systemd pvscan services. My /dev/sdd is not activated on boot due to systemd pvscan service missing. That happened with the upgrade to 15.2. There are three physical volume activation services on boot but the logical volume had 4 devices. How can I add a systemd service for the missing physical volume?
31.01.2021 09:53, Stratos Zolotas пишет:
I think the issue is on systemd pvscan services. My /dev/sdd is not activated on boot due to systemd pvscan service missing. That happened with the upgrade to 15.2. There are three physical volume activation services on boot but the logical volume had 4 devices.
How can I add a systemd service for the missing physical volume?
You shoot the messenger. This service is started from /usr/lib/udev/rules.d/69-dm-lvm-metad.rules which is part of LVM. You need to debug this rule to find out why it decides to skip pvscan for this particular device.
On Sun, Jan 31, 2021 at 9:08 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
You shoot the messenger. This service is started from /usr/lib/udev/rules.d/69-dm-lvm-metad.rules which is part of LVM. You need to debug this rule to find out why it decides to skip pvscan for this particular device.
I think debugging this udev rule is beyond my knowledge.... The expected behavior is to activate all PVs, like on 15.1 and it shouldn't break with the upgrade to 15.2. I opened a bug here: https://bugzilla.opensuse.org/show_bug.cgi?id=1181615 Thanks for the comments.
Ok found the solution. It seems an old raid signature was lying on this disk, although Leap 15.1 didn't had any issue with that. baskinsy:~ # dmraid -s ERROR: pdc: wrong # of devices in RAID set "pdc_bgdejjghgi" [1/2] on /dev/sde ERROR: pdc: wrong # of devices in RAID set "pdc_bgdejjghgi" [1/2] on /dev/sde *** *Inconsistent* Set name : pdc_bgdejjghgi size : 1250000000 stride : 128 type : mirror status : inconsistent subsets: 0 devs : 1 spares : 0 So I used the following to clear the signature baskinsy:~ # dmraid -r -E /dev/sde Do you really want to erase "pdc" ondisk metadata on /dev/sde ? [y/n] :y baskinsy:~ # dmraid -s no raid disks After that /dev/sde was assing a UUID and recognised by udev a LVM2 member so started to be mounted on boot. Sadly, i didn't found that before formatting the upgraded so I have to redo my services configuration. I closed also the bug report as invalid. Will post on forum also for anyone that may face a similar problem. Thanks for the help and patience. On Sun, Jan 31, 2021 at 10:05 AM Stratos Zolotas <strzol@gmail.com> wrote:
On Sun, Jan 31, 2021 at 9:08 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
You shoot the messenger. This service is started from /usr/lib/udev/rules.d/69-dm-lvm-metad.rules which is part of LVM. You need to debug this rule to find out why it decides to skip pvscan for this particular device.
I think debugging this udev rule is beyond my knowledge.... The expected behavior is to activate all PVs, like on 15.1 and it shouldn't break with the upgrade to 15.2.
I opened a bug here: https://bugzilla.opensuse.org/show_bug.cgi?id=1181615
Thanks for the comments.
participants (4)
-
Adam Mizerski -
Andrei Borzenkov -
Carlos E. R. -
Stratos Zolotas