[opensuse-buildservice] Publisher takes AGES
Hi, it appears to me, that the publisher is taking ages to get its job done. It is not the first time, I suffer from this, therefore.. See here: https://build.opensuse.org/project/show?project=home%3Afrispete%3Aaufs2 kernel-default build for i586 was done at: build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44 UTC 2010. Now, approx. 11 hours later, the project status is still in finished state. Is that to be expected? Is there something, that could be done from my side? Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Samstag, 13. Februar 2010 12:01:11 schrieb Hans-Peter Jansen:
Hi,
it appears to me, that the publisher is taking ages to get its job done. It is not the first time, I suffer from this, therefore..
See here: https://build.opensuse.org/project/show?project=home%3Afrispete%3Aaufs2
kernel-default build for i586 was done at:
build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44 UTC 2010.
Now, approx. 11 hours later, the project status is still in finished state.
Is that to be expected?
yes and no. The publisher was hanging for 24 hours, that god fixed yesterday morning. However, it had a backlog of almost 1800 repositories to process. That takes a while, at absolute minimum 12 hours I would say.
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Monday 2010-02-15 08:37, Adrian Schröter wrote:
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible.
Yes and no. There is a lot of metadata in rpmmd that openSUSE does not even read. IIRC, someone (mls?) mentioned that using .solv-like files instead would be sufficient, and I can imagine that is faster as well, both in processing and download. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Montag 15 Februar 2010 schrieb Jan Engelhardt:
On Monday 2010-02-15 08:37, Adrian Schröter wrote:
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible.
Yes and no. There is a lot of metadata in rpmmd that openSUSE does not even read. IIRC, someone (mls?) mentioned that using .solv-like files instead would be sufficient, and I can imagine that is faster as well, both in processing and download.
While this would be ok for factory or possibly 11.2, the build service repos support way more than that. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Monday 15 February 2010, 08:37:29 Adrian Schröter wrote:
Am Samstag, 13. Februar 2010 12:01:11 schrieb Hans-Peter Jansen:
Hi,
it appears to me, that the publisher is taking ages to get its job done. It is not the first time, I suffer from this, therefore..
See here: https://build.opensuse.org/project/show?project=home%3Afrispete%3Aaufs2
kernel-default build for i586 was done at:
build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44 UTC 2010.
Now, approx. 11 hours later, the project status is still in finished state.
Is that to be expected?
yes and no. The publisher was hanging for 24 hours, that god fixed yesterday morning.
Thanks god.
However, it had a backlog of almost 1800 repositories to process. That takes a while, at absolute minimum 12 hours I would say.
Hmm, does it process them serialized? If yes, the publisher should be able to throw a configurable amount of threads on it, doesn't it?
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible.
It's quite a while since I hacked on createrepo.. There's the repocache thing available, that it able to improve things quite considerable, by the price of stupidly creating its checksum files for each rpm in a repocache directory (and never remove them again..). The first question is, is it possible to hide that stuff before fan out? I guess, it would be possible (and desirable) to reduce this directory based repocache scheme to one file (say .repocache), but again, this should be "interned" somehow. Does BS provide infrastructure for such file handling? Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Montag, 15. Februar 2010 09:54:16 schrieb Hans-Peter Jansen:
On Monday 15 February 2010, 08:37:29 Adrian Schröter wrote:
Am Samstag, 13. Februar 2010 12:01:11 schrieb Hans-Peter Jansen:
Hi,
it appears to me, that the publisher is taking ages to get its job done. It is not the first time, I suffer from this, therefore..
See here: https://build.opensuse.org/project/show?project=home%3Afrispete%3Aaufs2
kernel-default build for i586 was done at:
build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44 UTC 2010.
Now, approx. 11 hours later, the project status is still in finished state.
Is that to be expected?
yes and no. The publisher was hanging for 24 hours, that god fixed yesterday morning.
Thanks god.
it was before coffee ...
However, it had a backlog of almost 1800 repositories to process. That takes a while, at absolute minimum 12 hours I would say.
Hmm, does it process them serialized? If yes, the publisher should be able to throw a configurable amount of threads on it, doesn't it?
yes, serialized. No, running multiple processes will only slow down in total, because this is pure IO load. Multiple processes would kill the IO due to seeking.
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible.
It's quite a while since I hacked on createrepo.. There's the repocache thing available, that it able to improve things quite considerable, by the price of stupidly creating its checksum files for each rpm in a repocache directory (and never remove them again..).
That is already used.
The first question is, is it possible to hide that stuff before fan out? I guess, it would be possible (and desirable) to reduce this directory based repocache scheme to one file (say .repocache), but again, this should be "interned" somehow. Does BS provide infrastructure for such file handling?
Sorry, but I don't understand what "such file handling" is in detail -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Monday 15 February 2010, 10:06:59 Adrian Schröter wrote:
Am Montag, 15. Februar 2010 09:54:16 schrieb Hans-Peter Jansen:
On Monday 15 February 2010, 08:37:29 Adrian Schröter wrote:
Am Samstag, 13. Februar 2010 12:01:11 schrieb Hans-Peter Jansen:
Hi,
it appears to me, that the publisher is taking ages to get its job done. It is not the first time, I suffer from this, therefore..
See here: https://build.opensuse.org/project/show?project=home%3Afrispete%3Aa ufs2
kernel-default build for i586 was done at:
build18 finished "build kernel-default.spec" at Sat Feb 13 00:55:44 UTC 2010.
Now, approx. 11 hours later, the project status is still in finished state.
Is that to be expected?
yes and no. The publisher was hanging for 24 hours, that god fixed yesterday morning.
Thanks god.
it was before coffee ...
I thought, that god must be loving this build service, but okay, some kind soul did released us then ;-)
However, it had a backlog of almost 1800 repositories to process. That takes a while, at absolute minimum 12 hours I would say.
Hmm, does it process them serialized? If yes, the publisher should be able to throw a configurable amount of threads on it, doesn't it?
yes, serialized. No, running multiple processes will only slow down in total, because this is pure IO load. Multiple processes would kill the IO due to seeking.
If I remember correctly, createrepo processes _also_ do use a considerable amount of CPU, but I might be wrong. It's been a few years ago, but you considered this and investigated on this already, I guess. If yes, then consolidating the repocache lookup and read into one file might reduce the amount of necessary IO.
Is there something, that could be done from my side?
Fix/improve createrepo. I have not looked inside, but my feelings tell me that there should be significant speedups possible.
It's quite a while since I hacked on createrepo.. There's the repocache thing available, that it able to improve things quite considerable, by the price of stupidly creating its checksum files for each rpm in a repocache directory (and never remove them again..).
That is already used.
The first question is, is it possible to hide that stuff before fan out? I guess, it would be possible (and desirable) to reduce this directory based repocache scheme to one file (say .repocache), but again, this should be "interned" somehow. Does BS provide infrastructure for such file handling?
Sorry, but I don't understand what "such file handling" is in detail
You implicitely answered that question. If BS is using the repocache option, then it obviously is hiding them already... Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (4)
-
Adrian Schröter -
Hans-Peter Jansen -
Jan Engelhardt -
Stephan Kulow