[opensuse-buildservice] KVM-based OBS workers
Are there more complete instructions for creating a KVM-based OBS worker? I am aiming for OBS workers based off of a minimal openSUSE base install (either the openSUSE Leap cloud image or NET .iso). I can get this to work with a VM install of the OBS appliance (I have both of the .iso and the .qcow2 installs working, as VMs. I do not have the .iso working as a bare metal install, which is confounding…). I would prefer my OBS workers are not full appliance installs. The best instructions I can find are via the code (dist/obsworker and dist/sysconfig.obs-server), namely: to ensure the qemu-kvm package is installed, and that virtualization is available (this requires the correct BIOS settings, and nested virtualization, in the case of a VM). elif [ -e /dev/kvm -a -x /usr/bin/qemu-kvm ] ; then Further, dist/sysconfig.obs-server offers this: # for KVM, you have to create with (example for openSUSE 11.2): # # export rootfstype="ext4" # mkinitrd -d /dev/null -m "ext4 binfmt_misc virtio_pci virtio_blk" -k #vmlinuz-2.6.31.12-0.2-default -i initrd-2.6.31.12-0.2-default-obs_worker # # a working initrd file which includes virtio and binfmt_misc for OBS in order #to work fine This is not proving sufficient. On the base openSUSE cloud images, qemu-kvm fires up, and then just hangs there, and eventually times out. There is very little in the way of error messaging or logs here. Digging further, on an existing OBS appliance install, I find a more complete list of kernel modules here (/etc/dracut.conf.d/03-my-obs.conf), but "zypper wp ..." ("what provides") on that file does not reveal anything. Also, this file does not seem to be in the open-build-service git repo. force_drivers+="loop dm-mod dm-snapshot binfmt-misc fuse kqemu squashfs ext2 ext3 ext4 reiserfs nf_conntrack_ipv6 binfmt_misc virtio_pci virtio_mmio virtio_blk virtio-rng fat vfat nls_cp437 nls_iso8859-1 ibmvscsi ibmvscsic" This longer list indicates that there might be more complete instructions floating around that I can not find. Can someone clarify what additional packages are needed on top of an openSUSE base install in order to get KVM-based OBS workers? (If documentation for this does not exist, if someone could guide me here, I would be happy to write a draft of that documentation. --- Phil King phil@cpanel.net -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Is anyone able to assist with this? Is there more information that I can provide? Here is how I am invoking mkinitrd: export rootfstype="ext4" VMLINUZ=/boot/vmlinuz-4.12.14-lp151.28.59-default INITRD=/boot/initrd-4.12.14-lp151.28.59-default mkinitrd -d /dev/null -m "loop dm-mod dm-snapshot binfmt-misc fuse kqemu squashfs ext2 ext3 ext4 reiserfs nf_conntrack_ipv6 binfmt_misc virtio_pci virtio_mmio virtio_blk virtio_rng fat vfat nls_cp437 nls_iso8859-1 ibmvscsi ibmvscsic" -k $VMLINUZ -i $INITRD And here is the relevant excerpt from the build log: 3s] ### VM INTERACTION START ### [ 3s] Using UART console [ 3s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/hwrng,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_1/.mount/boot/kernel -initrd /boot/initrd-4.12.14-lp151.28.59-default_obs -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nobarrier,commit=150,noatime ext4.allow_unsupported=1 kpti=off pti=off spectre_v2=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 512 -drive file=/var/cache/obs/worker/root_1/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_1/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_1/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 1 [ 4s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-0-ga698c89-rebuilt.opensuse.org) [ 5s] Booting from ROM..c[?7l[2J[[0;1;31mFAILED[0m] Failed to start Load Kernel Modules. [ 6s] See 'systemctl status systemd-modules-load.service' for details. [ 6s] [[0;1;31mFAILED[0m] Failed to start Setup Virtual Console. [ 6s] See 'systemctl status systemd-vconsole-setup.service' for details. [ 6s] [[0;1;33mDEPEND[0m] Dependency failed for dracut ask for additional cmdline parameters. [ 6s] [[0;32m OK [0m] Started Create Static Device Nodes in /dev. [ 6s] Starting dracut cmdline hook… This is a useful, advanced feature that is, as far as I can tell, completely undocumented. I need assistance please. --- Phil King phil@cpanel.net
On Aug 19, 2020, at 10:44 AM, Phil King <phil@cpanel.net> wrote:
Are there more complete instructions for creating a KVM-based OBS worker?
I am aiming for OBS workers based off of a minimal openSUSE base install (either the openSUSE Leap cloud image or NET .iso).
I can get this to work with a VM install of the OBS appliance (I have both of the .iso and the .qcow2 installs working, as VMs. I do not have the .iso working as a bare metal install, which is confounding…). I would prefer my OBS workers are not full appliance installs.
The best instructions I can find are via the code (dist/obsworker and dist/sysconfig.obs-server), namely: to ensure the qemu-kvm package is installed, and that virtualization is available (this requires the correct BIOS settings, and nested virtualization, in the case of a VM).
elif [ -e /dev/kvm -a -x /usr/bin/qemu-kvm ] ; then
Further, dist/sysconfig.obs-server offers this:
# for KVM, you have to create with (example for openSUSE 11.2): # # export rootfstype="ext4" # mkinitrd -d /dev/null -m "ext4 binfmt_misc virtio_pci virtio_blk" -k #vmlinuz-2.6.31.12-0.2-default -i initrd-2.6.31.12-0.2-default-obs_worker # # a working initrd file which includes virtio and binfmt_misc for OBS in order #to work fine
This is not proving sufficient. On the base openSUSE cloud images, qemu-kvm fires up, and then just hangs there, and eventually times out. There is very little in the way of error messaging or logs here.
Digging further, on an existing OBS appliance install, I find a more complete list of kernel modules here (/etc/dracut.conf.d/03-my-obs.conf), but "zypper wp ..." ("what provides") on that file does not reveal anything. Also, this file does not seem to be in the open-build-service git repo.
force_drivers+="loop dm-mod dm-snapshot binfmt-misc fuse kqemu squashfs ext2 ext3 ext4 reiserfs nf_conntrack_ipv6 binfmt_misc virtio_pci virtio_mmio virtio_blk virtio-rng fat vfat nls_cp437 nls_iso8859-1 ibmvscsi ibmvscsic"
This longer list indicates that there might be more complete instructions floating around that I can not find.
Can someone clarify what additional packages are needed on top of an openSUSE base install in order to get KVM-based OBS workers? (If documentation for this does not exist, if someone could guide me here, I would be happy to write a draft of that documentation.
--- Phil King phil@cpanel.net -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 26. August 2020, 22:45:28 CEST Phil King wrote:
Is anyone able to assist with this? Is there more information that I can provide?
Here is how I am invoking mkinitrd: export rootfstype="ext4" VMLINUZ=/boot/vmlinuz-4.12.14-lp151.28.59-default INITRD=/boot/initrd-4.12.14-lp151.28.59-default mkinitrd -d /dev/null -m "loop dm-mod dm-snapshot binfmt-misc fuse kqemu squashfs ext2 ext3 ext4 reiserfs nf_conntrack_ipv6 binfmt_misc virtio_pci virtio_mmio virtio_blk virtio_rng fat vfat nls_cp437 nls_iso8859-1 ibmvscsi ibmvscsic" -k $VMLINUZ -i $INITRD
And here is the relevant excerpt from the build log: 3s] ### VM INTERACTION START ### [ 3s] Using UART console [ 3s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none -cpu host -object rng-random,filename=/dev/hwrng,id=rng0 -device virtio-rng-pci,rng=rng0 -runas qemu -net none -kernel /var/cache/obs/worker/root_1/.mount/boot/kernel -initrd /boot/initrd-4.12.14-lp151.28.59-default_obs -append root=/dev/disk/by-id/virtio-0 rootfstype=ext3 rootflags=data=writeback,nobarrier,commit=150,noatime ext4.allow_unsupported=1 kpti=off pti=off spectre_v2=off panic=1 quiet no-kvmclock elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc console=ttyS0 init=/.build/build -m 512 -drive file=/var/cache/obs/worker/root_1/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-pci,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_1/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-pci,drive=swap,serial=1 -serial stdio -chardev socket,id=monitor,server,nowait,path=/var/cache/obs/worker/root_1/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 1 [ 4s] c[?7l[2J[0mSeaBIOS (version rel-1.12.0-0-ga698c89-rebuilt.opensuse.org) [ 5s] Booting from ROM..c[?7l[2J[[0;1;31mFAILED[0m] Failed to start Load Kernel Modules. [ 6s] See 'systemctl status systemd-modules-load.service' for details. [ 6s] [[0;1;31mFAILED[0m] Failed to start Setup Virtual Console. [ 6s] See 'systemctl status systemd-vconsole-setup.service' for details. [ 6s] [[0;1;33mDEPEND[0m] Dependency failed for dracut ask for additional cmdline parameters. [ 6s] [[0;32m OK [0m] Started Create Static Device Nodes in /dev. [ 6s] Starting dracut cmdline hook…
This is a useful, advanced feature that is, as far as I can tell, completely undocumented. I need assistance please.
I saw that your issue is solved meanwhile, but just for the record: A failing initrd is not something what the OBS/build code can influence. It is in the end content of the distribution. Or a not working fallback initrd from your worker. From OBS POV this is no different then any other build failure... -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 2020-08-19 10:44:26 -0500, Phil King wrote:
Are there more complete instructions for creating a KVM-based OBS worker?
I am aiming for OBS workers based off of a minimal openSUSE base install (either the openSUSE Leap cloud image or NET .iso).
I can get this to work with a VM install of the OBS appliance (I have both of the .iso and the .qcow2 installs working, as VMs. I do not have the .iso working as a bare metal install, which is confounding…). I would prefer my OBS workers are not full appliance installs.
That is, you want to run the obs worker in a VM, right? You are _not_ talking _building_ in a VM, right? If it is about the former, the following might be a starting point for manually building a VM image that can be used with qemu. Disclaimer: I use the following VM setup only for debugging purposes... so some "important" configs may be missing... (as I said, it probably just serves a starting point) # create a fixed size image with an ext4 fs $> dd if=/dev/zero of=./vm.img iflag=fullblock bs=1MiB count=2048 $> /sbin/mkfs.ext4 vm.img # mount the image + proc $> mount vm.img /mnt/ $> mkdir /mnt/proc $> mount -t proc none /mnt/proc/ # add the tumbleweed repo and install some packages (you may want to install # the obs-worker package/other packages, too) $> zypper --root /mnt/ ar http://download.opensuse.org/tumbleweed/repo/oss/ tw $> zypper --root /mnt/ in --no-recommends aaa_base coreutils util-linux vim \ iproute2 openssh vim vim-data rpm zypper systemd kernel-default dracut \ strace glibc-locale-base less (also, you may want to omit the "--no-recommends" option) # umount proc, create an initrd, setup root pw + user account $> umount /mnt/proc $> chroot /mnt $> dracut --kver 5.8.0-1-default # you might need to adjust the version $> passwd # set root password $> useradd -m marcus $> passwd marcus # set user password $> exit # leave chroot # copy kernel + initrd to the host and umount the image $> cp /mnt/boot/vmlinuz . $> cp /mnt/boot/initrd . $> umount /mnt # start the vm (assumption: port 2222 is not used) and login as root $> qemu-kvm -smp 1 -m 1024 -nographic -no-reboot \ -append "root=/dev/vda rw console=ttyS0 panic=-1" \ -kernel ./vmlinuz -initrd ./initrd \ -drive file=./vm.img,media=disk,if=virtio,format=raw \ -device e1000e,netdev=net1 \ -netdev user,id=net1,ipv4=on,ipv6=off,hostfwd=tcp::2222-10.0.2.7:22 # setup network (you can also use your favorite dhcp client...) and start sshd # (this happens in the guest, of course) $> ip link set dev ens3 up $> ip addr add dev ens3 10.0.2.7/24 # a somewhat arbitrary address $> ip route add default via 10.0.2.2 dev ens3 $> echo nameserver 10.0.2.3 > /etc/resolv.conf $> systemctl start sshd # now, you can login from the host via ssh $> ssh -p 2222 marcus@localhost At this point, you can start installing/configuring the obs-worker. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Interesting notes that I’ll work through later. It’s ultimately not what I am looking for. I want an openSUSE cloud image based VM, turn that in to an OBS worker (via the .repo and obsworker package as you describe), specifically as a KVM-based obsworker (so that I can have the memory constraints). I think I finally figured out my roadblock. As I was converting existing “chroot” obsworkers to KVM, I believe the invisible speed bump was a permissions issue in /var/cache/obs/worker. When I obliterate that directory, everything works as expected even with default values for initrd and the kernel in /etc/sysconfig/obs-server. (The lack of error messaging from the ‘build’ package proved really problematic here.) Thank you for the reply. --- Phil King phil@cpanel.net
On Aug 26, 2020, at 9:22 PM, Marcus Hüwe <suse-tux@gmx.de> wrote:
On 2020-08-19 10:44:26 -0500, Phil King wrote:
Are there more complete instructions for creating a KVM-based OBS worker?
I am aiming for OBS workers based off of a minimal openSUSE base install (either the openSUSE Leap cloud image or NET .iso).
I can get this to work with a VM install of the OBS appliance (I have both of the .iso and the .qcow2 installs working, as VMs. I do not have the .iso working as a bare metal install, which is confounding…). I would prefer my OBS workers are not full appliance installs.
That is, you want to run the obs worker in a VM, right? You are _not_ talking _building_ in a VM, right? If it is about the former, the following might be a starting point for manually building a VM image that can be used with qemu.
Disclaimer: I use the following VM setup only for debugging purposes... so some "important" configs may be missing... (as I said, it probably just serves a starting point)
# create a fixed size image with an ext4 fs $> dd if=/dev/zero of=./vm.img iflag=fullblock bs=1MiB count=2048 $> /sbin/mkfs.ext4 vm.img
# mount the image + proc $> mount vm.img /mnt/ $> mkdir /mnt/proc $> mount -t proc none /mnt/proc/
# add the tumbleweed repo and install some packages (you may want to install # the obs-worker package/other packages, too) $> zypper --root /mnt/ ar http://download.opensuse.org/tumbleweed/repo/oss/ tw $> zypper --root /mnt/ in --no-recommends aaa_base coreutils util-linux vim \ iproute2 openssh vim vim-data rpm zypper systemd kernel-default dracut \ strace glibc-locale-base less
(also, you may want to omit the "--no-recommends" option)
# umount proc, create an initrd, setup root pw + user account $> umount /mnt/proc $> chroot /mnt $> dracut --kver 5.8.0-1-default # you might need to adjust the version $> passwd # set root password $> useradd -m marcus $> passwd marcus # set user password $> exit # leave chroot
# copy kernel + initrd to the host and umount the image $> cp /mnt/boot/vmlinuz . $> cp /mnt/boot/initrd . $> umount /mnt
# start the vm (assumption: port 2222 is not used) and login as root $> qemu-kvm -smp 1 -m 1024 -nographic -no-reboot \ -append "root=/dev/vda rw console=ttyS0 panic=-1" \ -kernel ./vmlinuz -initrd ./initrd \ -drive file=./vm.img,media=disk,if=virtio,format=raw \ -device e1000e,netdev=net1 \ -netdev user,id=net1,ipv4=on,ipv6=off,hostfwd=tcp::2222-10.0.2.7:22
# setup network (you can also use your favorite dhcp client...) and start sshd # (this happens in the guest, of course) $> ip link set dev ens3 up $> ip addr add dev ens3 10.0.2.7/24 # a somewhat arbitrary address $> ip route add default via 10.0.2.2 dev ens3 $> echo nameserver 10.0.2.3 > /etc/resolv.conf $> systemctl start sshd
# now, you can login from the host via ssh $> ssh -p 2222 marcus@localhost
At this point, you can start installing/configuring the obs-worker.
Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter -
Marcus Hüwe -
Phil King