Mailinglist Archive: opensuse-buildservice (136 mails)

< Previous Next >
Re: [opensuse-buildservice] "discarded" workers
On Wednesday 19 August 2015, 12:16:02 wrote Andrew Davidoff:
Hi,

In our private OBS instance often workers become unavailable and their
state is reported as "discarded". I'm not sure, but it seems like one
cause of this might be when OBS kills a job (like if sources change
for the package being built). Just a moment ago a worker became
discarded while it was doing a build for a package that we know for a
fact had its sources changed during the build. This is what was on the
worker's screen (some details obfuscated with %TOKENS%):

[ 144s] qemu: terminating on signal 15 from pid 81153
[ 144s] No buildstatus set, either the base system is broken
(glibc/bash/perl)
[ 144s] or the build host has a kernel or hardware problem...

2015-08-19 09:59:26: build finished '%PACKAGENAME%-%PACKAGEVER%' for
project 'home:%USERNAME%:%PROJECTNAME%' repository
'common_xUbuntu_14.04' arch 'x86_64'

build discarded...
umount: /var/cache/obs/worker/root_1: not mounted
umount tmpfs failed: No such file or directory
could not kill build in /var/cache/obs/worker/root_1/root

The last line is repeated dozens of times, and the worker is hung. The
only way I know to fix this is to restart all the workers with
/etc/init.d/obsworker.

Do you know why workers might hang in discarded states like this,
and/or do you know how I can restart a single worker instead of all of
them?

It appears usually when a worker gets either explicit killed, or when
the worker detects a known bug inside of the VM handling of the
host. So when anything of these are part of the build log:

$kill_job = "kvm memory page bug" if $tail =~ /BUG: unable to handle
kernel NULL pointer dereference at/;
$kill_job = "kvm spinnlock bug" if $tail =~ /INFO: rcu_sched
self-detected stall on CPU/;
$kill_job = "xen soft lockup" if $tail =~ /BUG: soft lockup - CPU#\d+
stuck for/;



--

Adrian Schroeter
email: adrian@xxxxxxx

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284
(AG Nürnberg)

Maxfeldstraße 5
90409 Nürnberg
Germany


--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
References