AW: Builds taking forever and not finishing
Hi, me again.
So what I saw in the rep_server.log was
2024-03-07 10:14:13: [19782] maxchild limit reached
2024-03-07 10:14:13: maxchild limit reached
and on a worker itself
rpc failed: socket write: Connection timed out
sleeping one minute just in case...
send_statehttp://obs.uib.gmbh:5252 http://obs.uib.gmbh:5252 : rpc timeout
Any suggestion what I might do ?
Regards
Mathias
Mathias Radtke
Software Developer
uib GmbH
Bonifaziusplatz 1B
55118 Mainz
E-Mail: m.radtke@uib.de mailto:m.radtke@uib.de
Telefon: +49 6131 27561 0
https://uib.de
https://opsi.org
Sitz der Gesellschaft: Mainz
Handelsregister: Amtsgericht Mainz HRB 6942
Geschäftsführer: Erol Ülükmen
USt-IdNr.: DE 203 394 450
Save the Date: opsiConf in Mainz am 09. & 10. Juli 2024!
Die Voranmeldung ist bereits eröffnet. opsi.org/opsiconf/
-----Ursprüngliche Nachricht-----
Von: Mathias Radtke
On Donnerstag, 7. März 2024, 12:13:42 CET Mathias Radtke wrote:
Hi, me again.
So what I saw in the rep_server.log was
2024-03-07 10:14:13: [19782] maxchild limit reached
2024-03-07 10:14:13: maxchild limit reached
and on a worker itself
that sounds like hanging connections, as usually you should not run into that limit. bs_serverstatus /srv/obs/run/$each.status may give a hint. maybe strace also.
rpc failed: socket write: Connection timed out
sleeping one minute just in case...
send_statehttp://obs.uib.gmbh:5252 http://obs.uib.gmbh:5252 : rpc timeout
Any suggestion what I might do ?
Regards
Mathias
Mathias Radtke Software Developer
uib GmbH Bonifaziusplatz 1B 55118 Mainz
E-Mail: m.radtke@uib.de mailto:m.radtke@uib.de Telefon: +49 6131 27561 0
https://uib.de https://opsi.org
Sitz der Gesellschaft: Mainz Handelsregister: Amtsgericht Mainz HRB 6942 Geschäftsführer: Erol Ülükmen USt-IdNr.: DE 203 394 450
Save the Date: opsiConf in Mainz am 09. & 10. Juli 2024! Die Voranmeldung ist bereits eröffnet. opsi.org/opsiconf/
-----Ursprüngliche Nachricht----- Von: Mathias Radtke
Gesendet: Donnerstag 7. März 2024 8:04 An: buildservice@lists.opensuse.org Betreff: Builds taking forever and not finishing Hi
since yesterday we have a strange behaviour on our ptivate OBS instance. This instance links our packages published on the official OBS into our internal instance.
The new updated packages get downloaded into our local instance. They are "finished" according to the log but then still having the "building" state. The time of the build continues also and it never reaches a success.
Here's a Log from one of the workers
[ 166s] obsworker1 finished "build opsi-linux-bootimage.spec" at Thu Mar 7 06:43:26 UTC 2024.
[ 166s]
2024-03-07 07:43:26: finished 'opsi-linux-bootimage' for project 'home:uibmz:opsi:4.3:development' repository 'openSUSE_Leap_15.4' arch 'x86_64'
build succeeded, send everything back...
rpc failed: socket write: Connection timed out
In addition in the UI monitoring the Repserver seems to die and the workers dissappear. Systemctl says the obsrespserver is running. Restarting it makes the workers appear again in the UI Montoring page, and they start building until the builds never finish again.
Any suggestion what might cause the issue? Which log should I look into?
Regards
Mathias
Mathias Radtke Software Developer
uib GmbH Bonifaziusplatz 1B 55118 Mainz
E-Mail: m.radtke@uib.de mailto:m.radtke@uib.de Telefon: +49 6131 27561 0
https://uib.de https://opsi.org
Sitz der Gesellschaft: Mainz Handelsregister: Amtsgericht Mainz HRB 6942 Geschäftsführer: Erol Ülükmen USt-IdNr.: DE 203 394 450
Save the Date: opsiConf in Mainz am 09. & 10. Juli 2024! Die Voranmeldung ist bereits eröffnet. opsi.org/opsiconf/
--
Adrian Schroeter
participants (2)
-
Adrian Schröter
-
Mathias Radtke