On Wednesday 19 August 2015, 12:16:02 wrote Andrew Davidoff:
Hi,
In our private OBS instance often workers become unavailable and their state is reported as "discarded". I'm not sure, but it seems like one cause of this might be when OBS kills a job (like if sources change for the package being built). Just a moment ago a worker became discarded while it was doing a build for a package that we know for a fact had its sources changed during the build. This is what was on the worker's screen (some details obfuscated with %TOKENS%):
[ 144s] qemu: terminating on signal 15 from pid 81153 [ 144s] No buildstatus set, either the base system is broken (glibc/bash/perl) [ 144s] or the build host has a kernel or hardware problem...
2015-08-19 09:59:26: build finished '%PACKAGENAME%-%PACKAGEVER%' for project 'home:%USERNAME%:%PROJECTNAME%' repository 'common_xUbuntu_14.04' arch 'x86_64'
build discarded... umount: /var/cache/obs/worker/root_1: not mounted umount tmpfs failed: No such file or directory could not kill build in /var/cache/obs/worker/root_1/root
The last line is repeated dozens of times, and the worker is hung. The only way I know to fix this is to restart all the workers with /etc/init.d/obsworker.
Do you know why workers might hang in discarded states like this, and/or do you know how I can restart a single worker instead of all of them?
It appears usually when a worker gets either explicit killed, or when the worker detects a known bug inside of the VM handling of the host. So when anything of these are part of the build log: $kill_job = "kvm memory page bug" if $tail =~ /BUG: unable to handle kernel NULL pointer dereference at/; $kill_job = "kvm spinnlock bug" if $tail =~ /INFO: rcu_sched self-detected stall on CPU/; $kill_job = "xen soft lockup" if $tail =~ /BUG: soft lockup - CPU#\d+ stuck for/; -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org