Mailinglist Archive: opensuse-buildservice (46 mails)

< Previous Next >
Re: [opensuse-buildservice] Random delays during worker start
  • From: Adrian Schröter <adrian@xxxxxxx>
  • Date: Thu, 06 Dec 2018 09:10:30 +0100
  • Message-id: <5167222.uD76HebRUb@linux-ywca>
On Mittwoch, 5. Dezember 2018, 19:54:54 CET Keil, Karsten wrote:
Hello,

In our private OBS instance we use KVM workers on SLE12S-SP3.
The build servers are HP DL380 Gen9 with 40 cores with 256GB RAM for big
workers, 128 GB Ram for smaller ones. On such HW we running for example
10 instances with 3 Jobs.
The Problem is that after the initial install stage here are very often (I
would say ~ 1/3 of the builds) huge delays in the start of the VM:

A good one:
....
[ 10s] booting kvm...
[ 10s] ### VM INTERACTION START ###
[ 11s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none
-object rng-random,filename=/dev/random,id=rng0 -device
virtio-rng-pci,rng=rng0 -runas qemu -cpu host -net n [ 11s] [
1.063999] dracut-pre-udev[161]: modprobe: FATAL: Module kqemu not found. [
12s] [ 2.047790] dracut-pre-udev[161]: modprobe: FATAL: Module ibmvscsi
not found. [ 12s] [ 2.052451] dracut-pre-udev[161]: modprobe: FATAL:
Module ibmveth not found. [ 12s] ### VM INTERACTION END ###
[ 12s] 2nd stage started in virtual machine
[ 12s] machine type: x86_64
...
Bad looks like:
...
[ 11s] booting kvm...
[ 12s] ### VM INTERACTION START ###
[ 12s] /usr/bin/qemu-kvm -nodefaults -no-reboot -nographic -vga none
-object rng-random,filename=/dev/random,id=rng0 -device
virtio-rng-pci,rng=rng0 -runas qemu -cpu host -net n [ 12s] [
1.437692] dracut-pre-udev[161]: modprobe: FATAL: Module kqemu not found. [
111s] [ 98.122901] dracut-pre-udev[161]: modprobe: FATAL: Module ibmvscsi
not found. [ 112s] [ 99.982451] dracut-pre-udev[161]: modprobe: FATAL:
Module ibmveth not found. [ 112s] ### VM INTERACTION END ###
[ 112s] 2nd stage started in virtual machine
[ 112s] machine type: x86_64
...
This could also up to 300 seconds, I do not really found a relationship to
the size of the build jobs or amount of packages to install. Also the build
itself seems not to be much slower after the delay. The host itself does
not have any high load at these moments, most of CPUs is idle. So my
questions are:
What exact happens in that stage of the VM start, are there copies of the
initial packages ongoing ? Does somebody see similar things ?
OBS version is 2.92.

It seems to be the initrd booting...

OBS version should not really matter here, it is more a question which kernel/
initrd is used IMHO. Do you use a kernel-obs-build here or is it the one from
the worker system?



--

Adrian Schroeter
SUSE Linux Products GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
email: adrian@xxxxxxx




--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups
References