On Mittwoch, 2. März 2016, 08:33:53 wrote David Haller:
Hello,
I'm used to wait for packages to build, but waiting _long_ for signing and publishing used to be rather unusual. In the last days/couple weeks, the latter has become very slow, at least, weirdly, only for x86_64... Most recent example the update to openssl-1.0.2g in my home-repo:
We are currently at the IO limits in the storage system of the main backend servers. For that reason the schedulers can get into a hughe backlog. Atm. there are > 300 repositories with finished build results which needs recalculation. But I have also seen a queue of > 2000 this week already. The load on the main backends increase lately for a number of reasons, esp. also by the increased number of capable build hosts. Fetching the job, assigning for a sign job and esp. creating the repositories are local operations on the backend. We are about to seperate the storage system of the main projects and the home projects in the hope that this gives us enough additional IO bandwidth to keep up again with the build results. However, it is unlikely that we will finish that this week.
i586 build finished 2016-03-01 16:05:30 +0100 [1] and was signed and published soonish little (10, 20min?) later [2], normal soonish later.
x86_64 build finished 2016-03-01 16:20:49 +0100 [3], signing took _hours_ (at _least_ 4, IIRC 8 or so) and it has not been published yet (at 2016-03-02 08:32:00 +0100).
What's the delay? Esp. just for one arch when builds finish almost simultanously? How come signing and publishing could be arch-dependent? Yes, I've seen this before these days, always with x86_64, IIRC.
x86_64 scheduler has to manage way more repositories than the ones of the more obscure architectures.
-dnh, doing a local build now, but that won't help others
[2] http://download.opensuse.org/repositories/home:/dnh/openSUSE_12.1_Update_sta... http://download.opensuse.org/repositories/home:/dnh/openSUSE_12.1_Update_sta...
[4] http://download.opensuse.org/repositories/home:/dnh/openSUSE_12.1_Update_sta...
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org