Mailinglist Archive: opensuse-buildservice (266 mails)

< Previous Next >
Re: [opensuse-buildservice] Build service is borked - again
  • From: Adrian Schröter <adrian@xxxxxxx>
  • Date: Fri, 28 Nov 2008 15:17:12 +0100
  • Message-id: <200811281517.13162.adrian@xxxxxxx>
On Freitag 28 November 2008 14:51:50 Lars Marowsky-Bree wrote:
On 2008-11-28T13:43:20, Adrian Schröter <adrian@xxxxxxx> wrote:
The i586 scheduler has processed it meanwhile, I saw that it had more
than 2000 medium events some hours ago (means that it has to recalculate
2000 project repositories due to changed packages). If you can provide a
patch which makes this faster, I am happy to apply. (small sidenote, a
compiler which compiles the entire distribution instantly would be also
appricated :).

It is below 100 meanwhile, that means that should currently faster catch
up.

There's also a weird issue when jobs have just finished (possibly
failed).

The logs shown then - for a couple of minutes - will refer to the N-1
build. (They essentially revert from the "live log" to the stored log, I
guess, and the copy is not immediate, while the status got updated
already.)

This can be very confusing while the sync is catching up, and noone can
tell me that copying a logfile takes a long time ;-)

This is a bug, please create a report. (The file does not need even to be
copied on the server, it just get renamed).


Maybe the scheduler could be partitioned, so that it doesn't have to
process everything in one go? Maybe the sync could be partitioned too,
so that smaller increments are synced but the status is always correct?

The data export is partitioned in a seperate part (called publisher).

The status can not be always complete, except we block accepting the build
result until it has been recalculated.

Just to clarify, I am happy about any constructive suggesstion how to speed up
the process. But we have by intention a batch job system and not a life
interactive system in this regard to be handle the load at all.

Why does it take so long ? When ever a build job finishes, the build result
can provide and require anything. So it needs recalculate all related
packages. This are all packages inside of the project repo and dependending on
the build repo (while there is a prio on the packages inside of the repo).
It can not start any other build in this project before this has been done.
Recalculating of one openSUSE:Factory repo does take about 90 seconds.
(small project repos are much faster and usually in the 0-2 seconds area).

In short, the scheduler has always a complete view of all packages in all
projects including all dependencies at any time. It updates the view on every
change (source or binarie). These dependencies take currently about 750MB in
memory.

--

Adrian Schroeter
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
email: adrian@xxxxxxx


--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse-buildservice+help@xxxxxxxxxxxx

< Previous Next >
Follow Ups