build.opensuse.org slow
Hi, I just wonder if it is normal, that I currently have extremly slow bandwith to build.o.o when trying to do a openSUSE:Factory:zSystems testbuild with type qemu. The Package that I am interested in is python-onnx, and it is now downloading for 58 Minutes without any end in sight. Yes, I see some progres at /tmp/osc_build* with data rates at about typical ISDN data rates, however this is not really workable. When pinging build.opensuse.org, I get a considerable number of package drops, and tracepath tells me, that half of the latencies are added from within the twelve99 network. My own network is connected with 100/40 MBit to the internet (downstream/upstream) and works as desired (actual rates is 106/42). Some more data: # mtr -r -c 50 -w build.opensuse.org Start: 2023-09-08T14:41:25+0200 HOST: build.host Loss% Snt Last Avg Best Wrst StDev 1.|-- fritz.box 2.0% 50 2.5 3.1 2.0 5.2 0.7 2.|-- 2003:0:8902:8000::1 0.0% 50 6.6 7.2 5.6 18.8 1.8 3.|-- 2003:0:1307:4000::1 98.0% 50 11.2 11.2 11.2 11.2 0.0 4.|-- ffm-b5-link.ip.twelve99.net 84.0% 50 12.4 11.1 9.9 12.4 0.8 5.|-- ffm-bb2-v6.ip.twelve99.net 0.0% 50 10.6 11.9 10.1 24.6 2.1 6.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7.|-- ceskatele-ic318142-prag-b4.ip.twelve99-cust.net 4.0% 50 19.3 18.5 17.1 22.7 1.1 8.|-- 2a07:1f40:2:7::1 32.0% 50 26.0 26.5 25.1 28.6 0.9 9.|-- 2a07:1f40:2:7::2 12.0% 50 26.2 26.1 24.7 30.2 1.0 10.|-- 2a07:de40:b260:1::5 10.0% 50 25.5 48.3 24.8 1030. 149.7 11.|-- 2a07:de40:b250:131:10:151:131:20 16.0% 50 24.9 24.8 23.6 26.5 0.7 The current data rate makes it quite hard to contribute to opensuse, therefore I would really appreciate if this was just an exception, and not a new normal... Berthold azouhr@opensuse.org
On Freitag, 8. September 2023, 15:15:38 CEST Berthold Gunreben wrote:
Hi,
I just wonder if it is normal, that I currently have extremly slow bandwith to build.o.o when trying to do a openSUSE:Factory:zSystems testbuild with type qemu.
We are definitive not in the best shape atm performance wise, but not so bad as you are describing it atm.
The Package that I am interested in is python-onnx, and it is now downloading for 58 Minutes without any end in sight.
# time eosc getbinaries openSUSE:Factory:zSystems python-onnx standard s390x ... real 1m0,795s user 0m5,258s sys 0m0,495s from my home DSL line. Yes, that is slow for these small rpms, but it is not 1hour ...
Yes, I see some progres at /tmp/osc_build* with data rates at about typical ISDN data rates, however this is not really workable. When pinging build.opensuse.org, I get a considerable number of package drops, and tracepath tells me, that half of the latencies are added from within the twelve99 network. My own network is connected with 100/40 MBit to the internet (downstream/upstream) and works as desired (actual rates is 106/42).
okay, I leave this to our network admins from SUSE-IT to look into it. Is someone else experience these problems? thank you for reporting adrian
Some more data:
# mtr -r -c 50 -w build.opensuse.org Start: 2023-09-08T14:41:25+0200 HOST: build.host Loss% Snt Last Avg Best Wrst StDev 1.|-- fritz.box 2.0% 50 2.5 3.1 2.0 5.2 0.7 2.|-- 2003:0:8902:8000::1 0.0% 50 6.6 7.2 5.6 18.8 1.8 3.|-- 2003:0:1307:4000::1 98.0% 50 11.2 11.2 11.2 11.2 0.0 4.|-- ffm-b5-link.ip.twelve99.net 84.0% 50 12.4 11.1 9.9 12.4 0.8 5.|-- ffm-bb2-v6.ip.twelve99.net 0.0% 50 10.6 11.9 10.1 24.6 2.1 6.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7.|-- ceskatele-ic318142-prag-b4.ip.twelve99-cust.net 4.0% 50 19.3 18.5 17.1 22.7 1.1 8.|-- 2a07:1f40:2:7::1 32.0% 50 26.0 26.5 25.1 28.6 0.9 9.|-- 2a07:1f40:2:7::2 12.0% 50 26.2 26.1 24.7 30.2 1.0 10.|-- 2a07:de40:b260:1::5 10.0% 50 25.5 48.3 24.8 1030. 149.7 11.|-- 2a07:de40:b250:131:10:151:131:20 16.0% 50 24.9 24.8 23.6 26.5 0.7
The current data rate makes it quite hard to contribute to opensuse, therefore I would really appreciate if this was just an exception, and not a new normal...
Berthold azouhr@opensuse.org
-- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
On Fri, 08 Sep 2023 15:29:37 +0200 Adrian Schröter <adrian@suse.de> wrote:
On Freitag, 8. September 2023, 15:15:38 CEST Berthold Gunreben wrote:
Hi,
I just wonder if it is normal, that I currently have extremly slow bandwith to build.o.o when trying to do a openSUSE:Factory:zSystems testbuild with type qemu.
We are definitive not in the best shape atm performance wise, but not so bad as you are describing it atm.
The Package that I am interested in is python-onnx, and it is now downloading for 58 Minutes without any end in sight.
# time eosc getbinaries openSUSE:Factory:zSystems python-onnx standard s390x ... real 1m0,795s user 0m5,258s sys 0m0,495s
from my home DSL line. Yes, that is slow for these small rpms, but it is not 1hour ...
this is about # osc co openSUSE:Factory:zSystems/python-onnx && cd $_ which is kind of ok, but then: # osc build --vm-type qemu standard s390x and that is now doing fetching packages for 'openSUSE:Factory:zSystems': | Elapsed Time: 1:19:17 (ok ... just finished now, so that is how long it took) Typically, qemu builds are slow, but I guess building will be way faster than getting the packages for the build system. Berthold
Hello Berthold, and in general osc and interconnect users. We found an issue in haproxy running in front of OBS which caused hanging connections. Could you confirm that any connection issues have improved now? thanks adrian On Freitag, 8. September 2023, 15:36:29 CEST Berthold Gunreben wrote:
On Fri, 08 Sep 2023 15:29:37 +0200 Adrian Schröter <adrian@suse.de> wrote:
On Freitag, 8. September 2023, 15:15:38 CEST Berthold Gunreben wrote:
Hi,
I just wonder if it is normal, that I currently have extremly slow bandwith to build.o.o when trying to do a openSUSE:Factory:zSystems testbuild with type qemu.
We are definitive not in the best shape atm performance wise, but not so bad as you are describing it atm.
The Package that I am interested in is python-onnx, and it is now downloading for 58 Minutes without any end in sight.
# time eosc getbinaries openSUSE:Factory:zSystems python-onnx standard s390x ... real 1m0,795s user 0m5,258s sys 0m0,495s
from my home DSL line. Yes, that is slow for these small rpms, but it is not 1hour ...
this is about # osc co openSUSE:Factory:zSystems/python-onnx && cd $_
which is kind of ok, but then:
# osc build --vm-type qemu standard s390x
and that is now doing
fetching packages for 'openSUSE:Factory:zSystems': | Elapsed Time: 1:19:17
(ok ... just finished now, so that is how long it took)
Typically, qemu builds are slow, but I guess building will be way faster than getting the packages for the build system.
Berthold
-- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
Hi Adrian, after deleting the cached packages, the same package now needed 23:26 Minutes for downloading. Looks like the situation improved a lot, considering that previously, I probably still had packages in the cache. Using mtr also shows much improved repsonse times. I did not measure before the move to prague, but it now looks like it can be used the way it is. Berthold On Wed, 04 Oct 2023 11:18:31 +0200 Adrian Schröter <adrian@suse.de> wrote:
Hello Berthold,
and in general osc and interconnect users.
We found an issue in haproxy running in front of OBS which caused hanging connections.
Could you confirm that any connection issues have improved now?
thanks adrian
On Freitag, 8. September 2023, 15:36:29 CEST Berthold Gunreben wrote:
On Fri, 08 Sep 2023 15:29:37 +0200 Adrian Schröter <adrian@suse.de> wrote:
On Freitag, 8. September 2023, 15:15:38 CEST Berthold Gunreben wrote:
Hi,
I just wonder if it is normal, that I currently have extremly slow bandwith to build.o.o when trying to do a openSUSE:Factory:zSystems testbuild with type qemu.
We are definitive not in the best shape atm performance wise, but not so bad as you are describing it atm.
The Package that I am interested in is python-onnx, and it is now downloading for 58 Minutes without any end in sight.
# time eosc getbinaries openSUSE:Factory:zSystems python-onnx standard s390x ... real 1m0,795s user 0m5,258s sys 0m0,495s
from my home DSL line. Yes, that is slow for these small rpms, but it is not 1hour ...
this is about # osc co openSUSE:Factory:zSystems/python-onnx && cd $_
which is kind of ok, but then:
# osc build --vm-type qemu standard s390x
and that is now doing
fetching packages for 'openSUSE:Factory:zSystems': | Elapsed Time: 1:19:17
(ok ... just finished now, so that is how long it took)
Typically, qemu builds are slow, but I guess building will be way faster than getting the packages for the build system.
Berthold
participants (2)
-
Adrian Schröter -
Berthold Gunreben