Hi, my local OBS refuses to build any packages anymore with some strange errors: On the old monitor, it displays two builds on the same host: Kernel:GIT lime-kmp openSUSE_Tumbleweed - x86_64 6 minutes obsserver:2 x86_64 Kernel:GIT linux-tools openSUSE_Tumbleweed - x86_64 24 minutes obsserver:2 x86_64 One displays this on the build log: [ 0s] Using BUILD_ROOT=/var/cache/obs/worker/root_2/.mount [ 0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386 [ 0s] Doing kvm build in /var/cache/obs/worker/root_2/root [ 0s] [ 0s] [ 0s] obsserver started "build lime-kmp.spec" at Mon Sep 27 20:29:50 UTC 2021. [ 0s] [ 0s] Building lime-kmp for project 'LISA:Kernel:GIT' repository 'openSUSE_Tumbleweed' arch 'x86_64' srcmd5 '79907b8fb5bea8539ac4822376cdf7e2' [ 0s] [ 0s] processing recipe /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec ... [ 0s] running changelog2spec --target rpm --file /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec [ 0s] init_buildsystem --configdir /srv/obs/run/worker/2/build/configs --cachedir /var/cache/build --prepare --clean --rpmlist /var/cache/obs/worker/root_2/.build.rpmlist /var/cache/obs/worker/root_2/.build-srcdir/lime-kmp.spec build ... [ 1s] cycle: rpm-config-SUSE -> rpm [ 1s] breaking dependency rpm-config-SUSE -> rpm [ 1s] [1/33] preinstalling filesystem... [ 2s] [2/33] preinstalling permissions... [ 2s] [3/33] preinstalling glibc... [ 2s] [4/33] preinstalling diffutils... [ 2s] [5/33] preinstalling fillup... [ 2s] [6/33] preinstalling libacl1... [ 2s] [7/33] preinstalling libattr1... [ 2s] [8/33] preinstalling libbz2-1... [ 2s] [9/33] preinstalling libcap2... [ 2s] [10/33] preinstalling libgcc_s1... [ 2s] [11/33] preinstalling libgpg-error0... [ 2s] [12/33] preinstalling liblua5_4-5... [ 2s] [13/33] preinstalling liblzma5... [ 2s] [14/33] preinstalling libpcre1... [ 2s] [15/33] preinstalling libpcre2-8-0... [ 2s] [16/33] preinstalling libpopt0... [ 3s] [17/33] preinstalling libz1... [ 3s] [18/33] preinstalling libzstd1... [ 3s] [19/33] preinstalling attr... [ 3s] [20/33] preinstalling libelf1... [ 3s] [21/33] preinstalling libgcrypt20... [ 3s] [22/33] preinstalling libncurses6... [ 3s] [23/33] preinstalling libselinux1... [ 3s] [24/33] preinstalling libreadline8... [ 3s] [25/33] preinstalling tar... [ 3s] [26/33] preinstalling bash... [ 3s] [27/33] preinstalling grep... [ 3s] [28/33] preinstalling pam... [ 3s] [29/33] preinstalling sed... [ 3s] [30/33] preinstalling coreutils... [ 3s] [31/33] preinstalling aaa_base... [ 3s] [32/33] preinstalling rpm-config-SUSE... [ 4s] [33/33] preinstalling rpm... [ 4s] [ 4s] [1/11] preinstalling kernel-obs-build... [ 4s] [2/11] preinstalling libblkid1... [ 4s] [3/11] preinstalling libcrypt1... [ 4s] [4/11] preinstalling libdb-4_8... [ 4s] [5/11] preinstalling libmnl0... [ 4s] [6/11] preinstalling libsmartcols1... [ 4s] [7/11] preinstalling libuuid1... [ 4s] [8/11] preinstalling iproute2... [ 4s] [9/11] preinstalling libmount1... [ 4s] [10/11] preinstalling perl-base... [ 4s] [11/11] preinstalling util-linux... [ 5s] copying packages... [ 6s] reordering...cycle: libncurses6 -> terminfo-base [ 6s] breaking dependency terminfo-base -> libncurses6 [ 6s] cycle: binutils -> libctf0 [ 6s] breaking dependency binutils -> libctf0 [ 6s] cycle: gio-branding-upstream -> libgio-2_0-0 [ 6s] breaking dependency libgio-2_0-0 -> gio-branding-upstream [ 6s] cycle: libgio-2_0-0 -> glib2-tools [ 6s] breaking dependency libgio-2_0-0 -> glib2-tools [ 6s] cycle: libfipscheck1 -> fipscheck [ 6s] breaking dependency libfipscheck1 -> fipscheck [ 6s] cycle: libpython3_8-1_0 -> python38-base [ 6s] breaking dependency python38-base -> libpython3_8-1_0 [ 6s] cycle: rpm-config-SUSE -> rpm [ 6s] breaking dependency rpm -> rpm-config-SUSE [ 6s] cycle: mozilla-nss -> libsoftokn3 [ 6s] breaking dependency mozilla-nss -> libsoftokn3 [ 6s] cycle: pam -> pam_unix [ 6s] breaking dependency pam -> pam_unix [ 6s] cycle: polkit -> libpolkit0 [ 6s] breaking dependency polkit -> libpolkit0 [ 6s] done [ 7s] booting kvm... [ 7s] ### VM INTERACTION START ### [ 7s] Using UART console [ 7s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime ext4.allow_unsupported=1 mitigations=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 24576 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 4 [ 9s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org) [ 15s] Booting from ROM..c[?7l[2J[0m. [ 5.571871] dracut-mount[415]: Warning: Can't mount root filesystem [ 15s] [ 15s] Generating "/run/initramfs/rdsosreport.txt" [ 15s] [ 15s] [ 15s] Entering emergency mode. Exit the shell to coPress Enter for maintenance [ 313s] (or press Control-D to continue): [ 313s] ### WATCHDOG TRIGGERED, KILLING VM ### and the other: No live log available: connect to 127.0.0.1:37055: Connection refused And indeed, root=/dev/disk/by-id/virtio-0 isn't available: $ l /dev/disk/by-id total 0 drwxr-xr-x 2 root root 480 Sep 27 20:28 ./ drwxr-xr-x 7 root root 140 Sep 27 20:28 ../ lrwxrwxrwx 1 root root 9 Sep 27 20:28 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-cache -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-server -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_1 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_root_2 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_3 -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_root_4 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_1 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-name-OBS-worker_swap_2 -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_3 -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-name-OBS-worker_swap_4 -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeay1ZsXU9spfsLsjW61XOEfnsqHXXtKzMAr -> ../../dm-1 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaydlN9v9BSGYid5UG5s2vKuziJPDnPR077 -> ../../dm-7 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayelJVStiXfjDUSSsmAohuy2tDdaWQLQNy -> ../../dm-2 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayhvUIyJg3fw0iWlT0hvcOfaNMyp6x4Hpm -> ../../dm-4 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayrLlJ3DCnAoX0v4fdgm0dHmnP5RzUUePS -> ../../dm-5 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaySHCyHyhCn7HNcVqHuh9rVGwLBMBYLg3t -> ../../dm-6 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytjzvdkkL3LxKDPMorzAmtDuWgNPFfmhj -> ../../dm-0 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaytqe2f0ZLa8zfrujQB372s7WQtqskhLiB -> ../../dm-8 lrwxrwxrwx 1 root root 10 Sep 27 20:28 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeayU01AQattR6ji7QacX6o3IZlgrMFVjicE -> ../../dm-9 lrwxrwxrwx 1 root root 10 Sep 27 20:48 dm-uuid-LVM-vJFMz2yAnHezxPnDLUWBMsO21jdGxeaywGF9PHv3TXZPfoYAIsabBBPLNwvOXaVn -> ../../dm-3 lrwxrwxrwx 1 root root 10 Sep 27 20:28 lvm-pv-uuid-aCosqc-dRQ5-kunk-IahE-JUue-5FYl-GYFQAy -> ../../sdb1 $ l /srv/obs/workers/*/* -rw-r--r-- 1 obsrun obsrun 2219 Sep 27 20:42 /srv/obs/workers/building/x86_64:obsserver:2 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:1 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:3 -rw-r--r-- 1 obsrun obsrun 2051 Sep 27 20:43 /srv/obs/workers/idle/x86_64:obsserver:4 $ file -sL /dev/OBS/worker_root_* /dev/OBS/worker_root_1: data /dev/OBS/worker_root_2: Linux rev 1.0 ext2 filesystem data, UUID=ee460b18-9a7d-4bba-b539-646345952fcb (extents) (64bit) (large files) /dev/OBS/worker_root_3: data /dev/OBS/worker_root_4: data $ file -sL /dev/OBS/worker_swap_* /dev/OBS/worker_swap_1: data /dev/OBS/worker_swap_2: data /dev/OBS/worker_swap_3: data /dev/OBS/worker_swap_4: data The whole issue happened out of the blue. It operated correctly for month. $ systemctl --failed 0 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. Monitor shows all servers green. Rebooting didn't help. Updating didn't help. How do I get to the ominous /run/initramfs/rdsosreport.txt or red herring? Cheers, Pete