Local OBS doesn't build anymore (waiting for 4 compliant workers (4 of them bad))
Hi, my local OBS refuses to build any packages anymore with some strange errors: On the old monitor, it displays two builds on the same host: Kernel:GIT lime-kmp openSUSE_Tumbleweed - x86_64 6 minutes obsserver:2 x86_64 Kernel:GIT linux-tools openSUSE_Tumbleweed - x86_64 24 minutes obsserver:2 x86_64 One displays this on the build log: [ 0s] Using BUILD_ROOT=/var/cache/obs/worker/root_2/.mount [ 0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386 [ 0s] Doing kvm build in /var/cache/obs/worker/root_2/root [ 0s] [ 0s] [ 0s] obsserver started "build lime-kmp.spec" at Mon Sep 27 20:29:50 UTC 2021. [ 0s] [ 0s] Building lime-kmp for project 'LISA:Kernel:GIT' repository 'openSUSE_Tumbleweed' arch 'x86_64' srcmd5 '79907b8fb5bea8539ac4822376cdf7e2' [ 0s] [ 0s] processing recipe /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec ... [ 0s] running changelog2spec --target rpm --file /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec [ 0s] init_buildsystem --configdir /srv/obs/run/worker/2/build/configs --cachedir /var/cache/build --prepare --clean --rpmlist /var/cache/obs/worker/root_2/.build.rpmlist /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec build ... [ 1s] cycle: rpm-config-SUSE -> rpm [ 1s] breaking dependency rpm-config-SUSE -> rpm [ 1s] [1/33] preinstalling filesystem... [ 2s] [2/33] preinstalling permissions... [ 2s] [3/33] preinstalling glibc... [ 2s] [4/33] preinstalling diffutils... [ 2s] [5/33] preinstalling fillup... [ 2s] [6/33] preinstalling libacl1... [ 2s] [7/33] preinstalling libattr1... [ 2s] [8/33] preinstalling libbz2-1... [ 2s] [9/33] preinstalling libcap2... [ 2s] [10/33] preinstalling libgcc_s1... [ 2s] [11/33] preinstalling libgpg-error0... [ 2s] [12/33] preinstalling liblua5_4-5... [ 2s] [13/33] preinstalling liblzma5... [ 2s] [14/33] preinstalling libpcre1... [ 2s] [15/33] preinstalling libpcre2-8-0... [ 2s] [16/33] preinstalling libpopt0... [ 3s] [17/33] preinstalling libz1... [ 3s] [18/33] preinstalling libzstd1... [ 3s] [19/33] preinstalling attr... [ 3s] [20/33] preinstalling libelf1... [ 3s] [21/33] preinstalling libgcrypt20... [ 3s] [22/33] preinstalling libncurses6... [ 3s] [23/33] preinstalling libselinux1... [ 3s] [24/33] preinstalling libreadline8... [ 3s] [25/33] preinstalling tar... [ 3s] [26/33] preinstalling bash... [ 3s] [27/33] preinstalling grep... [ 3s] [28/33] preinstalling pam... [ 3s] [29/33] preinstalling sed... [ 3s] [30/33] preinstalling coreutils... [ 3s] [31/33] preinstalling aaa_base... [ 3s] [32/33] preinstalling rpm-config-SUSE... [ 4s] [33/33] preinstalling rpm... [ 4s] [ 4s] [1/11] preinstalling kernel-obs-build... [ 4s] [2/11] preinstalling libblkid1... [ 4s] [3/11] preinstalling libcrypt1... [ 4s] [4/11] preinstalling libdb-4_8... [ 4s] [5/11] preinstalling libmnl0... [ 4s] [6/11] preinstalling libsmartcols1... [ 4s] [7/11] preinstalling libuuid1... [ 4s] [8/11] preinstalling iproute2... [ 4s] [9/11] preinstalling libmount1... [ 4s] [10/11] preinstalling perl-base... [ 4s] [11/11] preinstalling util-linux... [ 5s] copying packages... [ 6s] reordering...cycle: libncurses6 -> terminfo-base [ 6s] breaking dependency terminfo-base -> libncurses6 [ 6s] cycle: binutils -> libctf0 [ 6s] breaking dependency binutils -> libctf0 [ 6s] cycle: gio-branding-upstream -> libgio-2_0-0 [ 6s] breaking dependency libgio-2_0-0 -> gio-branding-upstream [ 6s] cycle: libgio-2_0-0 -> glib2-tools [ 6s] breaking dependency libgio-2_0-0 -> glib2-tools [ 6s] cycle: libfipscheck1 -> fipscheck [ 6s] breaking dependency libfipscheck1 -> fipscheck [ 6s] cycle: libpython3_8-1_0 -> python38-base [ 6s] breaking dependency python38-base -> libpython3_8-1_0 [ 6s] cycle: rpm-config-SUSE -> rpm [ 6s] breaking dependency rpm -> rpm-config-SUSE [ 6s] cycle: mozilla-nss -> libsoftokn3 [ 6s] breaking dependency mozilla-nss -> libsoftokn3 [ 6s] cycle: pam -> pam_unix [ 6s] breaking dependency pam -> pam_unix [ 6s] cycle: polkit -> libpolkit0 [ 6s] breaking dependency polkit -> libpolkit0 [ 6s] done [ 7s] booting kvm... [ 7s] ### VM INTERACTION START ### [ 7s] Using UART console [ 7s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime ext4.allow_unsupported=1 mitigations=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 24576 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 4 [ 9s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org) [ 15s] Booting from ROM..c[?7l[2J[0m. [ 5.571871] dracut-mount[415]: Warning: Can't mount root filesystem [ 15s] [ 15s] Generating "/run/initramfs/rdsosreport.txt" [ 15s] [ 15s] [ 15s] Entering emergency mode. Exit the shell to coPress Enter for maintenance [ 313s] (or press Control-D to continue): [ 313s] ### WATCHDOG TRIGGERED, KILLING VM ### and the other: No live log available: connect to 127.0.0.1:37055: Connection refused And indeed, root=/dev/disk/by-id/virtio-0 isn't available: $ l /dev/disk/by-id total 0 drwxr-xr-x 2 root root 480 Sep 27 20:28 ./ drwxr-xr-x 7 root root 140 Sep 27 20:28 ../ lrwxrwxrwx 1 root root 9 Sep 27 20:28 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-cache -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-server -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_1 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_root_2 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_3 -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_4 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_1 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_swap_2 -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_3 -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_4 -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeay1ZsXU9spfsLsjW61XOEfnsqHXXtKzMAr -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaydlN9v9BSGYid5UG5s2vKuziJPDnPR077 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayelJVStiXfjDUSSsmAohuy2tDdaWQLQNy -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayhvUIyJg3fw0iWlT0hvcOfaNMyp6x4Hpm -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayrLlJ3DCnAoX0v4fdgm0dHmnP5RzUUePS -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaySHCyHyhCn7HNcVqHuh9rVGwLBMBYLg3t -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytjzvdkkL3LxKDPMorzAmtDuWgNPFfmhj -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytqe2f0ZLa8zfrujQB372s7WQtqskhLiB -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayU01AQattR6ji7QacX6o3IZlgrMFVjicE -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaywGF9PHv3TXZPfoYAIsabBBPLNwvOXaVn -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 lvm-pv-uuid-aCosqc-dRQ5-kunk-IahE-JUue-5FYl-GYFQAy -> ../../sdb1 $ l /srv/obs/workers/*/* -rw-r--r-- 1 obsrun obsrun 2219 Sep 27 20:42 /srv/obs/workers/building/x86_64:obsserver:2 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:1 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:3 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:4 $ file -sL /dev/OBS/worker_root_* /dev/OBS/worker_root_1: data /dev/OBS/worker_root_2: Linux rev 1.0 ext2 filesystem data, UUID=ee460b18-9a7d-4bba-b539-646345952fcb (extents) (64bit) (large files) /dev/OBS/worker_root_3: data /dev/OBS/worker_root_4: data $ file -sL /dev/OBS/worker_swap_* /dev/OBS/worker_swap_1: data /dev/OBS/worker_swap_2: data /dev/OBS/worker_swap_3: data /dev/OBS/worker_swap_4: data The whole issue happened out of the blue. It operated correctly for month. $ systemctl --failed 0 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. Monitor shows all servers green. Rebooting didn't help. Updating didn't help. How do I get to the ominous /run/initramfs/rdsosreport.txt or red herring? Cheers, Pete
On Montag, 27. September 2021, 22:55:30 CEST Hans-Peter Jansen wrote:
Hi,
my local OBS refuses to build any packages anymore with some strange errors:
On the old monitor, it displays two builds on the same host: Kernel:GIT lime-kmp openSUSE_Tumbleweed - x86_64 6 minutes obsserver:2 x86_64 Kernel:GIT linux-tools openSUSE_Tumbleweed - x86_64 24 minutes obsserver:2 x86_64
One displays this on the build log:
[ 0s] Using BUILD_ROOT=/var/cache/obs/worker/root_2/.mount [ 0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386 [ 0s] Doing kvm build in /var/cache/obs/worker/root_2/root [ 0s] [ 0s] [ 0s] obsserver started "build lime-kmp.spec" at Mon Sep 27 20:29:50 UTC 2021. [ 0s] [ 0s] Building lime-kmp for project 'LISA:Kernel:GIT' repository 'openSUSE_Tumbleweed' arch 'x86_64' srcmd5 '79907b8fb5bea8539ac4822376cdf7e2' [ 0s] [ 0s] processing recipe /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec ... [ 0s] running changelog2spec --target rpm --file /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec [ 0s] init_buildsystem --configdir /srv/obs/run/worker/2/build/configs --cachedir /var/cache/build --prepare --clean --rpmlist /var/cache/obs/worker/root_2/.build.rpmlist /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec build ... [ 1s] cycle: rpm-config-SUSE -> rpm [ 1s] breaking dependency rpm-config-SUSE -> rpm [ 1s] [1/33] preinstalling filesystem... [ 2s] [2/33] preinstalling permissions... [ 2s] [3/33] preinstalling glibc... [ 2s] [4/33] preinstalling diffutils... [ 2s] [5/33] preinstalling fillup... [ 2s] [6/33] preinstalling libacl1... [ 2s] [7/33] preinstalling libattr1... [ 2s] [8/33] preinstalling libbz2-1... [ 2s] [9/33] preinstalling libcap2... [ 2s] [10/33] preinstalling libgcc_s1... [ 2s] [11/33] preinstalling libgpg-error0... [ 2s] [12/33] preinstalling liblua5_4-5... [ 2s] [13/33] preinstalling liblzma5... [ 2s] [14/33] preinstalling libpcre1... [ 2s] [15/33] preinstalling libpcre2-8-0... [ 2s] [16/33] preinstalling libpopt0... [ 3s] [17/33] preinstalling libz1... [ 3s] [18/33] preinstalling libzstd1... [ 3s] [19/33] preinstalling attr... [ 3s] [20/33] preinstalling libelf1... [ 3s] [21/33] preinstalling libgcrypt20... [ 3s] [22/33] preinstalling libncurses6... [ 3s] [23/33] preinstalling libselinux1... [ 3s] [24/33] preinstalling libreadline8... [ 3s] [25/33] preinstalling tar... [ 3s] [26/33] preinstalling bash... [ 3s] [27/33] preinstalling grep... [ 3s] [28/33] preinstalling pam... [ 3s] [29/33] preinstalling sed... [ 3s] [30/33] preinstalling coreutils... [ 3s] [31/33] preinstalling aaa_base... [ 3s] [32/33] preinstalling rpm-config-SUSE... [ 4s] [33/33] preinstalling rpm... [ 4s] [ 4s] [1/11] preinstalling kernel-obs-build... [ 4s] [2/11] preinstalling libblkid1... [ 4s] [3/11] preinstalling libcrypt1... [ 4s] [4/11] preinstalling libdb-4_8... [ 4s] [5/11] preinstalling libmnl0... [ 4s] [6/11] preinstalling libsmartcols1... [ 4s] [7/11] preinstalling libuuid1... [ 4s] [8/11] preinstalling iproute2... [ 4s] [9/11] preinstalling libmount1... [ 4s] [10/11] preinstalling perl-base... [ 4s] [11/11] preinstalling util-linux... [ 5s] copying packages... [ 6s] reordering...cycle: libncurses6 -> terminfo-base [ 6s] breaking dependency terminfo-base -> libncurses6 [ 6s] cycle: binutils -> libctf0 [ 6s] breaking dependency binutils -> libctf0 [ 6s] cycle: gio-branding-upstream -> libgio-2_0-0 [ 6s] breaking dependency libgio-2_0-0 -> gio-branding-upstream [ 6s] cycle: libgio-2_0-0 -> glib2-tools [ 6s] breaking dependency libgio-2_0-0 -> glib2-tools [ 6s] cycle: libfipscheck1 -> fipscheck [ 6s] breaking dependency libfipscheck1 -> fipscheck [ 6s] cycle: libpython3_8-1_0 -> python38-base [ 6s] breaking dependency python38-base -> libpython3_8-1_0 [ 6s] cycle: rpm-config-SUSE -> rpm [ 6s] breaking dependency rpm -> rpm-config-SUSE [ 6s] cycle: mozilla-nss -> libsoftokn3 [ 6s] breaking dependency mozilla-nss -> libsoftokn3 [ 6s] cycle: pam -> pam_unix [ 6s] breaking dependency pam -> pam_unix [ 6s] cycle: polkit -> libpolkit0 [ 6s] breaking dependency polkit -> libpolkit0 [ 6s] done [ 7s] booting kvm... [ 7s] ### VM INTERACTION START ### [ 7s] Using UART console [ 7s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime ext4.allow_unsupported=1 mitigations=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 24576 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 4 [ 9s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org) [ 15s] Booting from ROM..c[?7l[2J[0m. [ 5.571871] dracut-mount[415]: Warning: Can't mount root filesystem [ 15s] [ 15s] Generating "/run/initramfs/rdsosreport.txt" [ 15s] [ 15s] [ 15s] Entering emergency mode. Exit the shell to coPress Enter for maintenance [ 313s] (or press Control-D to continue): [ 313s] ### WATCHDOG TRIGGERED, KILLING VM ###
and the other:
No live log available: connect to 127.0.0.1:37055: Connection refused
And indeed, root=/dev/disk/by-id/virtio-0 isn't available:
$ l /dev/disk/by-id total 0 drwxr-xr-x 2 root root 480 Sep 27 20:28 ./ drwxr-xr-x 7 root root 140 Sep 27 20:28 ../ lrwxrwxrwx 1 root root 9 Sep 27 20:28 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-cache -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-server -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_1 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_root_2 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_3 -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_4 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_1 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_swap_2 -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_3 -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_4 -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeay1ZsXU9spfsLsjW61XOEfnsqHXXtKzMAr -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaydlN9v9BSGYid5UG5s2vKuziJPDnPR077 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayelJVStiXfjDUSSsmAohuy2tDdaWQLQNy -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayhvUIyJg3fw0iWlT0hvcOfaNMyp6x4Hpm -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayrLlJ3DCnAoX0v4fdgm0dHmnP5RzUUePS -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaySHCyHyhCn7HNcVqHuh9rVGwLBMBYLg3t -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytjzvdkkL3LxKDPMorzAmtDuWgNPFfmhj -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytqe2f0ZLa8zfrujQB372s7WQtqskhLiB -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayU01AQattR6ji7QacX6o3IZlgrMFVjicE -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaywGF9PHv3TXZPfoYAIsabBBPLNwvOXaVn -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 lvm-pv-uuid-aCosqc-dRQ5-kunk-IahE-JUue-5FYl-GYFQAy -> ../../sdb1
$ l /srv/obs/workers/*/* -rw-r--r-- 1 obsrun obsrun 2219 Sep 27 20:42 /srv/obs/workers/building/x86_64:obsserver:2 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:1 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:3 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:4
$ file -sL /dev/OBS/worker_root_* /dev/OBS/worker_root_1: data /dev/OBS/worker_root_2: Linux rev 1.0 ext2 filesystem data, UUID=ee460b18-9a7d-4bba-b539-646345952fcb (extents) (64bit) (large files) /dev/OBS/worker_root_3: data /dev/OBS/worker_root_4: data
$ file -sL /dev/OBS/worker_swap_* /dev/OBS/worker_swap_1: data /dev/OBS/worker_swap_2: data /dev/OBS/worker_swap_3: data /dev/OBS/worker_swap_4: data
The whole issue happened out of the blue. It operated correctly for month.
I suppose you got some newer kernel-obs-build package from somewhere. And the new kernel/initrd is not able to mount/attach to the storage anymore. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On Dienstag, 28. September 2021, 08:46:55 CEST Adrian Schröter wrote:
On Montag, 27. September 2021, 22:55:30 CEST Hans-Peter Jansen wrote:
Hi,
my local OBS refuses to build any packages anymore with some strange errors:
On the old monitor, it displays two builds on the same host: Kernel:GIT lime-kmp openSUSE_Tumbleweed - x86_64 6 minutes obsserver:2 x86_64 Kernel:GIT linux-tools openSUSE_Tumbleweed - x86_64 24 minutes obsserver:2 x86_64
One displays this on the build log:
[ 0s] Using BUILD_ROOT=/var/cache/obs/worker/root_2/.mount [ 0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386 [ 0s] Doing kvm build in /var/cache/obs/worker/root_2/root [ 0s] [ 0s] [ 0s] obsserver started "build lime-kmp.spec" at Mon Sep 27 20:29:50 UTC 2021. [ 0s] [ 0s] Building lime-kmp for project 'LISA:Kernel:GIT' repository 'openSUSE_Tumbleweed' arch 'x86_64' srcmd5 '79907b8fb5bea8539ac4822376cdf7e2' [ 0s] [ 0s] processing recipe /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec ... [ 0s] running changelog2spec --target rpm --file /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec [ 0s] init_buildsystem --configdir /srv/obs/run/worker/2/build/configs --cachedir /var/cache/build --prepare --clean --rpmlist /var/cache/obs/worker/root_2/.build.rpmlist /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec build ... [ 1s] cycle: rpm-config-SUSE -> rpm [ 1s] breaking dependency rpm-config-SUSE -> rpm [ 1s] [1/33] preinstalling filesystem... [ 2s] [2/33] preinstalling permissions... [ 2s] [3/33] preinstalling glibc... [ 2s] [4/33] preinstalling diffutils... [ 2s] [5/33] preinstalling fillup... [ 2s] [6/33] preinstalling libacl1... [ 2s] [7/33] preinstalling libattr1... [ 2s] [8/33] preinstalling libbz2-1... [ 2s] [9/33] preinstalling libcap2... [ 2s] [10/33] preinstalling libgcc_s1... [ 2s] [11/33] preinstalling libgpg-error0... [ 2s] [12/33] preinstalling liblua5_4-5... [ 2s] [13/33] preinstalling liblzma5... [ 2s] [14/33] preinstalling libpcre1... [ 2s] [15/33] preinstalling libpcre2-8-0... [ 2s] [16/33] preinstalling libpopt0... [ 3s] [17/33] preinstalling libz1... [ 3s] [18/33] preinstalling libzstd1... [ 3s] [19/33] preinstalling attr... [ 3s] [20/33] preinstalling libelf1... [ 3s] [21/33] preinstalling libgcrypt20... [ 3s] [22/33] preinstalling libncurses6... [ 3s] [23/33] preinstalling libselinux1... [ 3s] [24/33] preinstalling libreadline8... [ 3s] [25/33] preinstalling tar... [ 3s] [26/33] preinstalling bash... [ 3s] [27/33] preinstalling grep... [ 3s] [28/33] preinstalling pam... [ 3s] [29/33] preinstalling sed... [ 3s] [30/33] preinstalling coreutils... [ 3s] [31/33] preinstalling aaa_base... [ 3s] [32/33] preinstalling rpm-config-SUSE... [ 4s] [33/33] preinstalling rpm... [ 4s] [ 4s] [1/11] preinstalling kernel-obs-build... [ 4s] [2/11] preinstalling libblkid1... [ 4s] [3/11] preinstalling libcrypt1... [ 4s] [4/11] preinstalling libdb-4_8... [ 4s] [5/11] preinstalling libmnl0... [ 4s] [6/11] preinstalling libsmartcols1... [ 4s] [7/11] preinstalling libuuid1... [ 4s] [8/11] preinstalling iproute2... [ 4s] [9/11] preinstalling libmount1... [ 4s] [10/11] preinstalling perl-base... [ 4s] [11/11] preinstalling util-linux... [ 5s] copying packages... [ 6s] reordering...cycle: libncurses6 -> terminfo-base [ 6s] breaking dependency terminfo-base -> libncurses6 [ 6s] cycle: binutils -> libctf0 [ 6s] breaking dependency binutils -> libctf0 [ 6s] cycle: gio-branding-upstream -> libgio-2_0-0 [ 6s] breaking dependency libgio-2_0-0 -> gio-branding-upstream [ 6s] cycle: libgio-2_0-0 -> glib2-tools [ 6s] breaking dependency libgio-2_0-0 -> glib2-tools [ 6s] cycle: libfipscheck1 -> fipscheck [ 6s] breaking dependency libfipscheck1 -> fipscheck [ 6s] cycle: libpython3_8-1_0 -> python38-base [ 6s] breaking dependency python38-base -> libpython3_8-1_0 [ 6s] cycle: rpm-config-SUSE -> rpm [ 6s] breaking dependency rpm -> rpm-config-SUSE [ 6s] cycle: mozilla-nss -> libsoftokn3 [ 6s] breaking dependency mozilla-nss -> libsoftokn3 [ 6s] cycle: pam -> pam_unix [ 6s] breaking dependency pam -> pam_unix [ 6s] cycle: polkit -> libpolkit0 [ 6s] breaking dependency polkit -> libpolkit0 [ 6s] done [ 7s] booting kvm... [ 7s] ### VM INTERACTION START ### [ 7s] Using UART console [ 7s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime ext4.allow_unsupported=1 mitigations=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 24576 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 4 [ 9s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org) [ 15s] Booting from ROM..c[?7l[2J[0m. [ 5.571871] dracut-mount[415]: Warning: Can't mount root filesystem [ 15s] [ 15s] Generating "/run/initramfs/rdsosreport.txt" [ 15s] [ 15s] [ 15s] Entering emergency mode. Exit the shell to coPress Enter for maintenance [ 313s] (or press Control-D to continue): [ 313s] ### WATCHDOG TRIGGERED, KILLING VM ###
and the other:
No live log available: connect to 127.0.0.1:37055: Connection refused
And indeed, root=/dev/disk/by-id/virtio-0 isn't available:
$ l /dev/disk/by-id total 0 drwxr-xr-x 2 root root 480 Sep 27 20:28 ./ drwxr-xr-x 7 root root 140 Sep 27 20:28 ../ lrwxrwxrwx 1 root root 9 Sep 27 20:28 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-cache -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-server -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_1 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_root_2 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_3 -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_4 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_1 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_swap_2 -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_3 -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_4 -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeay1ZsXU9spfsLsjW61XOEfnsqHXXtKzMAr -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaydlN9v9BSGYid5UG5s2vKuziJPDnPR077 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayelJVStiXfjDUSSsmAohuy2tDdaWQLQNy -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayhvUIyJg3fw0iWlT0hvcOfaNMyp6x4Hpm -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayrLlJ3DCnAoX0v4fdgm0dHmnP5RzUUePS -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaySHCyHyhCn7HNcVqHuh9rVGwLBMBYLg3t -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytjzvdkkL3LxKDPMorzAmtDuWgNPFfmhj -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytqe2f0ZLa8zfrujQB372s7WQtqskhLiB -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayU01AQattR6ji7QacX6o3IZlgrMFVjicE -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaywGF9PHv3TXZPfoYAIsabBBPLNwvOXaVn -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 lvm-pv-uuid-aCosqc-dRQ5-kunk-IahE-JUue-5FYl-GYFQAy -> ../../sdb1
$ l /srv/obs/workers/*/* -rw-r--r-- 1 obsrun obsrun 2219 Sep 27 20:42 /srv/obs/workers/building/x86_64:obsserver:2 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:1 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:3 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:4
$ file -sL /dev/OBS/worker_root_* /dev/OBS/worker_root_1: data /dev/OBS/worker_root_2: Linux rev 1.0 ext2 filesystem data, UUID=ee460b18-9a7d-4bba-b539-646345952fcb (extents) (64bit) (large files) /dev/OBS/worker_root_3: data /dev/OBS/worker_root_4: data
$ file -sL /dev/OBS/worker_swap_* /dev/OBS/worker_swap_1: data /dev/OBS/worker_swap_2: data /dev/OBS/worker_swap_3: data /dev/OBS/worker_swap_4: data
The whole issue happened out of the blue. It operated correctly for month.
I suppose you got some newer kernel-obs-build package from somewhere. And the new kernel/initrd is not able to mount/attach to the storage anymore.
the obvious next step would be to find out from where the kernel is coming ;) Eg. via "osc buildinfo" -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
Hi, Am Dienstag, den 28.09.2021, 08:47 +0200 schrieb Adrian Schröter:
On Dienstag, 28. September 2021, 08:46:55 CEST Adrian Schröter wrote:
On Montag, 27. September 2021, 22:55:30 CEST Hans-Peter Jansen wrote:
Hi,
my local OBS refuses to build any packages anymore with some strange errors: : I suppose you got some newer kernel-obs-build package from somewhere. And the new kernel/initrd is not able to mount/attach to the storage anymore.
the obvious next step would be to find out from where the kernel is coming ;) Eg. via "osc buildinfo"
The same thing happened to PMBS Saturday, Sep 18th, roughly around 2 a.m., without any package updates. Probably this is triggered by changes in OBS, not by changes in the local installation. Please see https://lists.links2linux.de/pipermail/packman/2021-September/016827.html , seife found the solution of the missing /srv directory in the created buildroot. His suggestion was to use a newer version of build from openSUSE:Tools, but IMHO this should be fixed in the build packages of the respective released obs versions below https://download.opensuse.org/repositories/OBS:/Server:/ Greetings, Stefan PS: no, I did not file a bug anywhere, no time... -- Stefan Botter zu Hause Bremen
On Dienstag, 28. September 2021, 13:58:06 CEST Stefan Botter wrote:
Hi,
Am Dienstag, den 28.09.2021, 08:47 +0200 schrieb Adrian Schröter:
On Dienstag, 28. September 2021, 08:46:55 CEST Adrian Schröter wrote:
On Montag, 27. September 2021, 22:55:30 CEST Hans-Peter Jansen wrote:
Hi,
my local OBS refuses to build any packages anymore with some strange errors: : I suppose you got some newer kernel-obs-build package from somewhere. And the new kernel/initrd is not able to mount/attach to the storage anymore.
the obvious next step would be to find out from where the kernel is coming ;) Eg. via "osc buildinfo"
The same thing happened to PMBS Saturday, Sep 18th, roughly around 2 a.m., without any package updates. Probably this is triggered by changes in OBS, not by changes in the local installation.
it might be that OBS delivered the package, but you need to find out which one we talk about at all.
Please see https://lists.links2linux.de/pipermail/packman/2021-September/016827.html , seife found the solution of the missing /srv directory in the created buildroot.
okay, the new dracut caused it....
His suggestion was to use a newer version of build from openSUSE:Tools, but IMHO this should be fixed in the build packages of the respective released obs versions below https://download.opensuse.org/repositories/OBS:/Server:/
indeed, 2.10 got a new release meanwhile. bye adrian -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
Am Mittwoch, 29. September 2021, 10:06:14 CEST schrieb Adrian Schröter:
On Dienstag, 28. September 2021, 13:58:06 CEST Stefan Botter wrote:
Hi,
Am Dienstag, den 28.09.2021, 08:47 +0200 schrieb Adrian Schröter:
On Dienstag, 28. September 2021, 08:46:55 CEST Adrian Schröter wrote:
On Montag, 27. September 2021, 22:55:30 CEST Hans-Peter Jansen
wrote:
Hi,
my local OBS refuses to build any packages anymore with some
strange errors: I suppose you got some newer kernel-obs-build package from somewhere. And the new kernel/initrd is not able to mount/attach to the storage anymore.
the obvious next step would be to find out from where the kernel is coming ;) Eg. via "osc buildinfo"
The same thing happened to PMBS Saturday, Sep 18th, roughly around 2 a.m., without any package updates. Probably this is triggered by changes in OBS, not by changes in the local installation.
it might be that OBS delivered the package, but you need to find out which one we talk about at all.
Please see https://lists.links2linux.de/pipermail/packman/2021-September/016827.html , seife found the solution of the missing /srv directory in the created buildroot.
okay, the new dracut caused it....
His suggestion was to use a newer version of build from openSUSE:Tools, but IMHO this should be fixed in the build packages of the respective released obs versions below https://download.opensuse.org/repositories/OBS:/Server:/
indeed, 2.10 got a new release meanwhile.
After updating OBS, containing build-20210902, it is building packages again. Yay. Thanks everybody involved! In the kernel build project, all packages were sitting in "scheduled" state with popup stating: "waiting for 4 compliant workers (4 of them bad)" $ obs_admin --drop-badhosts fixed that issue for me. Cheers, Pete
participants (3)
-
Adrian Schröter -
Hans-Peter Jansen -
Stefan Botter