Mailinglist Archive: opensuse-buildservice (182 mails)

< Previous Next >
Re: [opensuse-buildservice] Publisher takes AGES
  • From: "Hans-Peter Jansen" <hpj@xxxxxxxxx>
  • Date: Mon, 15 Feb 2010 13:09:44 +0100
  • Message-id: <201002151309.45289.hpj@xxxxxxxxx>
On Monday 15 February 2010, 10:06:59 Adrian Schröter wrote:
Am Montag, 15. Februar 2010 09:54:16 schrieb Hans-Peter Jansen:
On Monday 15 February 2010, 08:37:29 Adrian Schröter wrote:
Am Samstag, 13. Februar 2010 12:01:11 schrieb Hans-Peter Jansen:
Hi,

it appears to me, that the publisher is taking ages to get its job
done. It is not the first time, I suffer from this, therefore..

See here:
https://build.opensuse.org/project/show?project=home%3Afrispete%3Aa
ufs2

kernel-default build for i586 was done at:

build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44
UTC 2010.

Now, approx. 11 hours later, the project status is still in
finished state.

Is that to be expected?

yes and no. The publisher was hanging for 24 hours, that god fixed
yesterday morning.

Thanks god.

it was before coffee ...

I thought, that god must be loving this build service, but okay, some kind
soul did released us then ;-)

However, it had a backlog of almost 1800 repositories to process.
That takes a while, at absolute minimum 12 hours I would say.

Hmm, does it process them serialized? If yes, the publisher should be
able to throw a configurable amount of threads on it, doesn't it?

yes, serialized. No, running multiple processes will only slow down in
total, because this is pure IO load. Multiple processes would kill the IO
due to seeking.

If I remember correctly, createrepo processes _also_ do use a considerable
amount of CPU, but I might be wrong. It's been a few years ago, but you
considered this and investigated on this already, I guess. If yes, then
consolidating the repocache lookup and read into one file might reduce the
amount of necessary IO.

Is there something, that could be done from my side?

Fix/improve createrepo. I have not looked inside, but my feelings
tell me that there should be significant speedups possible.

It's quite a while since I hacked on createrepo.. There's the repocache
thing available, that it able to improve things quite considerable, by
the price of stupidly creating its checksum files for each rpm in a
repocache directory (and never remove them again..).

That is already used.

The first question is, is it possible to hide that stuff before fan
out? I guess, it would be possible (and desirable) to reduce this
directory based repocache scheme to one file (say .repocache), but
again, this should be "interned" somehow. Does BS provide
infrastructure for such file handling?

Sorry, but I don't understand what "such file handling" is in detail

You implicitely answered that question. If BS is using the repocache option,
then it obviously is hiding them already...

Pete
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse-buildservice+help@xxxxxxxxxxxx

< Previous Next >