Builds for Factory/Tumbleweed blocked: downloading 4 dod packages
Good morning, builds for Factory/Tumbleweed seem to be blocked since yesterday in the afternoon, I still get: blocked: downloading 4 dod packages Known issue? Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Freitag, 2. Juni 2023, 06:54:17 CEST Johannes Kastl wrote:
Good morning,
builds for Factory/Tumbleweed seem to be blocked since yesterday in the afternoon, I still get:
blocked: downloading 4 dod packages
Known issue?
yes, yesterday it was caused by a bottleneck in OBS interconnect reporting. Today our slow download server is the bottleneck as many workers, in our instance and also all remote instances are fetching the tumbleweed rpm updates atm. We will have a look in removing some of the bottlenecks later today. But it should also settle down when the workers received their updated packages in their caches. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
Hi Adrian, On 02.06.23 at 08:10 Adrian Schröter wrote:
yes, yesterday it was caused by a bottleneck in OBS interconnect reporting.
Today our slow download server is the bottleneck as many workers, in our instance and also all remote instances are fetching the tumbleweed rpm updates atm.
We will have a look in removing some of the bottlenecks later today.
But it should also settle down when the workers received their updated packages in their caches.
Thanks for the answer. Then I will just wait and see. Just wanted to try out some updates, but I'll find something else to do in the meantime... Have a nice weekend! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 2023/06/02 08:10:58 +0200, Adrian Schröter wrote:
On Freitag, 2. Juni 2023, 06:54:17 CEST Johannes Kastl wrote:
Good morning,
builds for Factory/Tumbleweed seem to be blocked since yesterday in the afternoon, I still get:
blocked: downloading 4 dod packages
Known issue?
yes, yesterday it was caused by a bottleneck in OBS interconnect reporting.
Today our slow download server is the bottleneck as many workers, in our instance and also all remote instances are fetching the tumbleweed rpm updates atm.
We will have a look in removing some of the bottlenecks later today.
But it should also settle down when the workers received their updated packages in their caches.
Still not working: werner/urlscan> osc build --download-api-only Building urlscan.spec for openSUSE_Factory/x86_64 Run source service: /usr/lib/obs/service/format_spec_file --outdir /usr/src/werner/urlscan/tmpj6tygefd.format_spec_file.service Run source service: /usr/lib/obs/service/source_validator --outdir /usr/src/werner/urlscan/tmp3082l0p7.source_validator.service Getting buildconfig from server and store to /usr/src/werner/urlscan/.osc/_buildconfig-openSUSE_Factory-x86_64 Getting buildinfo from server and store to /usr/src/werner/urlscan/.osc/_buildinfo-openSUSE_Factory-x86_64.xml Updating cache of required packages 2.8% cache miss. 171/176 dependencies cached. openSUSE:Tumbleweed/glibc: attempting download from api, since not found openSUSE:Tumbleweed/glibc-devel: attempting download from api, since not found openSUSE:Tumbleweed/glibc-locale-base: attempting download from api, since not found openSUSE:Tumbleweed/dwz: attempting download from api, since not found openSUSE:Tumbleweed/rpmlint-mini: attempting download from api, since not found Server returned an error: HTTP Error 500: remote error 17964https //downloadcontent.opensuse.org/tumbleweed/repo/oss/x86_64/glibc-2.37-4.1.x86_64.rpm SSL_connect error Resource temporarily unavailable remote error: 17964https //downloadcontent.opensuse.org/tumbleweed/repo/oss/x86_64/glibc-2.37-4.1.x86_64.rpm SSL_connect error Resource temporarily unavailable Request: https://api.opensuse.org/build/openSUSE:Tumbleweed/dod/x86_64/_repository?bi... Headers: Date: Fri, 02 Jun 2023 07:21:03 GMT Server: Apache/2.4.51 (Linux/SUSE) Access-Control-Allow-Origin: https://build.opensuse.org Access-Control-Allow-Credentials: true Strict-Transport-Security: max-age=31536000 cache-control: private, no-transform cache-control: no-cache referrer-policy: strict-origin-when-cross-origin x-permitted-cross-domain-policies: none x-xss-protection: 1; mode=block x-opensuse-runtimes: {"view":null,"db":3.544218063354492,"backend":0} x-request-id: e87be6f2-b3b9-4fe7-bf88-d551cf4d930e x-opensuse-apiversion: 2.11~alpha.20230531T182734.e9d1b0358 x-download-options: noopen x-runtime: 0.011670 x-frame-options: SAMEORIGIN x-content-type-options: nosniff x-powered-by: Phusion Passenger(R) content-type: text/xml content-length: 404 Connection: close -- "Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
On Freitag, 2. Juni 2023, 09:22:39 CEST Dr. Werner Fink wrote:
On 2023/06/02 08:10:58 +0200, Adrian Schröter wrote:
On Freitag, 2. Juni 2023, 06:54:17 CEST Johannes Kastl wrote:
Good morning,
builds for Factory/Tumbleweed seem to be blocked since yesterday in the afternoon, I still get:
blocked: downloading 4 dod packages
Known issue?
yes, yesterday it was caused by a bottleneck in OBS interconnect reporting.
Today our slow download server is the bottleneck as many workers, in our instance and also all remote instances are fetching the tumbleweed rpm updates atm.
We will have a look in removing some of the bottlenecks later today.
But it should also settle down when the workers received their updated packages in their caches.
Still not working:
I know.
werner/urlscan> osc build --download-api-only
But you are actually also creating the problem here. Drop that parameter to allow osc to download it directly from the mirror server. Flooding the api server to forward the content is contra productive ...
Building urlscan.spec for openSUSE_Factory/x86_64 Run source service: /usr/lib/obs/service/format_spec_file --outdir /usr/src/werner/urlscan/tmpj6tygefd.format_spec_file.service Run source service: /usr/lib/obs/service/source_validator --outdir /usr/src/werner/urlscan/tmp3082l0p7.source_validator.service Getting buildconfig from server and store to /usr/src/werner/urlscan/.osc/_buildconfig-openSUSE_Factory-x86_64 Getting buildinfo from server and store to /usr/src/werner/urlscan/.osc/_buildinfo-openSUSE_Factory-x86_64.xml Updating cache of required packages 2.8% cache miss. 171/176 dependencies cached.
openSUSE:Tumbleweed/glibc: attempting download from api, since not found openSUSE:Tumbleweed/glibc-devel: attempting download from api, since not found openSUSE:Tumbleweed/glibc-locale-base: attempting download from api, since not found openSUSE:Tumbleweed/dwz: attempting download from api, since not found openSUSE:Tumbleweed/rpmlint-mini: attempting download from api, since not found Server returned an error: HTTP Error 500: remote error 17964https //downloadcontent.opensuse.org/tumbleweed/repo/oss/x86_64/glibc-2.37-4.1.x86_64.rpm SSL_connect error Resource temporarily unavailable remote error: 17964https //downloadcontent.opensuse.org/tumbleweed/repo/oss/x86_64/glibc-2.37-4.1.x86_64.rpm SSL_connect error Resource temporarily unavailable
Request: https://api.opensuse.org/build/openSUSE:Tumbleweed/dod/x86_64/_repository?bi... Headers: Date: Fri, 02 Jun 2023 07:21:03 GMT Server: Apache/2.4.51 (Linux/SUSE) Access-Control-Allow-Origin: https://build.opensuse.org Access-Control-Allow-Credentials: true Strict-Transport-Security: max-age=31536000 cache-control: private, no-transform cache-control: no-cache referrer-policy: strict-origin-when-cross-origin x-permitted-cross-domain-policies: none x-xss-protection: 1; mode=block x-opensuse-runtimes: {"view":null,"db":3.544218063354492,"backend":0} x-request-id: e87be6f2-b3b9-4fe7-bf88-d551cf4d930e x-opensuse-apiversion: 2.11~alpha.20230531T182734.e9d1b0358 x-download-options: noopen x-runtime: 0.011670 x-frame-options: SAMEORIGIN x-content-type-options: nosniff x-powered-by: Phusion Passenger(R) content-type: text/xml content-length: 404 Connection: close
-- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
On Freitag, 2. Juni 2023, 09:37:40 CEST Adrian Schröter wrote:
On Freitag, 2. Juni 2023, 09:22:39 CEST Dr. Werner Fink wrote:
On 2023/06/02 08:10:58 +0200, Adrian Schröter wrote:
On Freitag, 2. Juni 2023, 06:54:17 CEST Johannes Kastl wrote:
Good morning,
builds for Factory/Tumbleweed seem to be blocked since yesterday in the afternoon, I still get:
blocked: downloading 4 dod packages
Known issue?
yes, yesterday it was caused by a bottleneck in OBS interconnect reporting.
Today our slow download server is the bottleneck as many workers, in our instance and also all remote instances are fetching the tumbleweed rpm updates atm.
We will have a look in removing some of the bottlenecks later today.
But it should also settle down when the workers received their updated packages in their caches.
k, the request storm seems to be a self-produced issue ... an invalid SSL verification leaded not only to a download failure, but also to an invalid notification of all of our remote instances ... (at least it seems on first glance). The issue is workarounded for now, so OBS and remote instances should be usable again. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
On Friday 2023-06-02 09:37, Adrian Schröter wrote:
werner/urlscan> osc build --download-api-only
But you are actually also creating the problem here. Drop that parameter to allow osc to download it directly from the mirror server.
The mirror situation is unsatisfactory. * no parallel downloads by osc * can't(?) set a fixed mirror to use * multiply by the TCP startup latency for d.o.o and the particular mirror * `osc build` for openSUSE:Factory/libHX, observed with iptraf, the numbers hovered around 17 Mbit, with a few bursts to ~40 (probably glibc, which is one of the larger packages) * Not a fault of the mirror (ftp.gwdg.de), it certainly does gigabit when pulling an ISO API request (--do) has something going for it: * Packages are streamed as a continuous cpio stream in just one TCP connection * At least 60 Mbit/s at all times (iptraf again); the cpio stream for libHX build deps was delivered in just 24 s. [/var/tmp/osbuild-packagecache was removed each step to get comparable numbers.]
participants (4)
-
Adrian Schröter -
Dr. Werner Fink -
Jan Engelhardt -
Johannes Kastl