Trial to use a CDN for download.opensuse.org

newer
[opensuse-factory] reproducible...

older
Problem with build of VirtualBox...

Dirk Müller

14 Jul 2023 14 Jul '23

14:44

Hi all, For a while I've been working in the background to get a sponsored CDN subscription. I am happy to report that Fastly.com has agreed to sponsor the openSUSE project with bandwidth. We primarily intend to use it to improve the reachability (latency) of download.opensuse.org in various areas of the world, but of course can expand it to other usecases as well. Andrii, Bernhard and myself have set up an initial proof of concept under https://cdn.opensuse.org/ which you can now use and report issues with. just use it in place of download.opensuse.org in your zypper repositories or elsewhere. Ideally we would be providing the openSUSE Leap maintenance update repositories directly via the CDN, but that is currently not possible in how the setup is working. We intend to improve that over time but that will require fixes in various places including the Open Build Service itself (it can currently not handle more than one download site). Please don't spread the usage too widely, we're still in a "BETA" stage and need to implement some measures to keep bandwidth usage under control. Happy to hear about any feedback that you might have. Greetings, Dirk

Show replies by date

llyyr

14 Jul 14 Jul

16:32

On 7/14/23 20:14, Dirk Müller wrote:

...

Hi all,

For a while I've been working in the background to get a sponsored CDN subscription. I am happy to report that Fastly.com has agreed to sponsor the openSUSE project with bandwidth. We primarily intend to use it to improve the reachability (latency) of download.opensuse.org in various areas of the world, but of course can expand it to other usecases as well. Andrii, Bernhard and myself have set up an initial proof of concept under https://cdn.opensuse.org/ which you can now use and report issues with. just use it in place of download.opensuse.org in your zypper repositories or elsewhere.

Ideally we would be providing the openSUSE Leap maintenance update repositories directly via the CDN, but that is currently not possible in how the setup is working. We intend to improve that over time but that will require fixes in various places including the Open Build Service itself (it can currently not handle more than one download site).

Please don't spread the usage too widely, we're still in a "BETA" stage and need to implement some measures to keep bandwidth usage under control.

Happy to hear about any feedback that you might have.

Greetings, Dirk I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

Other than that everything is good

Luciano Santos

15 Jul 15 Jul

01:06

On Fri, 07/14/2023 at 22:02 +0530, llyyr wrote:

...

I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

Other than that everything is good

I don't mean to be rude, but let's focus on the discussion at hand, please. There are better places to report that. You probably is using an out of date mirrorsorcerer where it changed the baseurl in your .repo files to use cache.opensuse.net.br/. If not, please drop a message in the openSUSE Admins chat room. -- Kind regards, Luciano (luc14n0), an openSUSE contributor.

llyyr

04:43

On 7/15/23 06:36, Luciano Santos wrote:

...

On Fri, 07/14/2023 at 22:02 +0530, llyyr wrote:

...
I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

Other than that everything is good

I don't mean to be rude, but let's focus on the discussion at hand, please. There are better places to report that. You probably is using an out of date mirrorsorcerer where it changed the baseurl in your .repo files to use cache.opensuse.net.br/. If not, please drop a message in the openSUSE Admins chat room.

How is this not the discussion at hand? The original email said to leave any feedback.

...

Mirrorsorcerer I didn't even know this was a thing, and definitely never used it or any other mirrorcache because they all perform horribly in India.

https://cdn.opensuse.org simply redirects me to http://mirrorcache-jp.opensuse.org. If feedback means "OMG great feature!" only, then sure my message is offtopic. But I think it's directly related to the topic at hand. Downloading the filelist with wget to see the redirects here: $ wget https://cdn.opensuse.org/tumbleweed/repo/oss/repodata/24fe09b94abd82a90a6771... 2>&1 | grep Location: Location: http://mirrorcache-jp.opensuse.org/tumbleweed/repo/oss/repodata/24fe09b94abd... [following] Location: http://ftp.riken.jp/Linux/opensuse/tumbleweed/repo/oss/repodata/24fe09b94abd... [following]

...

changed the baseurl in your .repo files to use cache.opensuse.net.br/. Where is this assumption coming from? If you're going to be a douche at least be correct.

Andrei Borzenkov

05:00

On 15.07.2023 07:43, llyyr wrote:

...

On 7/15/23 06:36, Luciano Santos wrote:

...
On Fri, 07/14/2023 at 22:02 +0530, llyyr wrote:

...
I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

Other than that everything is good

I don't mean to be rude, but let's focus on the discussion at hand, please. There are better places to report that. You probably is using an out of date mirrorsorcerer where it changed the baseurl in your .repo files to use cache.opensuse.net.br/. If not, please drop a message in the openSUSE Admins chat room.

How is this not the discussion at hand? The original email said to leave any feedback.

...
Mirrorsorcerer I didn't even know this was a thing, and definitely never used it or any other mirrorcache because they all perform horribly in India.

https://cdn.opensuse.org simply redirects me to http://mirrorcache-jp.opensuse.org.

Yes, this is long standing issue with openSUSE infrastructure. zypper disables https -> http downgrade and mirrorbrain used such redirection. It became better with mirrorcache, and those cases I have seen are due to downstream mirrors. But here it is plain bug in cdn.opensuse.org - such redirection cannot work with zypper. bor@bor-Latitude-E5450:~$ curl -sI https://cdn.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-DVD-a... | grep location: location: http://mirror.de.leaseweb.net/opensuse/distribution/leap/15.5/iso/openSUSE-L... bor@bor-Latitude-E5450:~$

Claudio Marcial Peon

10:40

In my case the mirror that cdn gives me is much faster than download.. root@Datacor-ZRH:~# curl -sI https://download.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-... | grep location: && wget -O /dev/null -q --show-progress https://download.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-... location: https://pkg.adfinis.com/opensuse/distribution/leap/15.5/iso/openSUSE-Leap-15... /dev/null 100%[=======================================================>] 4.02G 72.6MB/s in 62s root@Datacor-ZRH:~# curl -sI https://cdn.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-DVD-a... | grep location: && wget -O /dev/null -q --show-progress https://cdn.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-DVD-a... location: http://mirror.easyname.at/opensuse/distribution/leap/15.5/iso/openSUSE-Leap-... /dev/null 100%[=======================================================>] 4.02G 323MB/s in 13s root@Datacor-ZRH:~# On Sat, Jul 15, 2023 at 7:01 AM Andrei Borzenkov wrote:

...

...
On 7/15/23 06:36, Luciano Santos wrote:

...
On Fri, 07/14/2023 at 22:02 +0530, llyyr wrote:

...
I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

Other than that everything is good

I don't mean to be rude, but let's focus on the discussion at hand,

On 15.07.2023 07:43, llyyr wrote: please.

...
...
There are better places to report that. You probably is using an out of date mirrorsorcerer where it changed the baseurl in your .repo files to use cache.opensuse.net.br/. If not, please drop a message in the openSUSE Admins chat room.

How is this not the discussion at hand? The original email said to leave any feedback.

...
Mirrorsorcerer I didn't even know this was a thing, and definitely never used it or any other mirrorcache because they all perform horribly in India.

https://cdn.opensuse.org simply redirects me to http://mirrorcache-jp.opensuse.org.

Yes, this is long standing issue with openSUSE infrastructure. zypper disables https -> http downgrade and mirrorbrain used such redirection. It became better with mirrorcache, and those cases I have seen are due to downstream mirrors. But here it is plain bug in cdn.opensuse.org - such redirection cannot work with zypper.

bor@bor-Latitude-E5450:~$ curl -sI

https://cdn.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-DVD-a... | grep location: location:

http://mirror.de.leaseweb.net/opensuse/distribution/leap/15.5/iso/openSUSE-L... bor@bor-Latitude-E5450:~$

Luciano Santos

16 Jul 16 Jul

03:07

On Sat, 07/15/2023 at 10:13 +0530, llyyr wrote:

...

How is this not the discussion at hand? The original email said to leave any feedback. ... I didn't even know this was a thing, and definitely never used it or any other mirrorcache because they all perform horribly in India.

https://cdn.opensuse.org simply redirects me to http://mirrorcache-jp.opensuse.org. If feedback means "OMG great feature!" only, then sure my message is offtopic. But I think it's directly related to the topic at hand. ... Where is this assumption coming from? If you're going to be a douche at least be correct.

Alright, let's clear the air so we can move on with the thread. First, I'm sorry I came out on you like that. When I said I didn't mean to be rude, I really meant it. My assumptions were based on cases I have seen recently, all boiling down to that server I pointed. And I assure you I wasn't, even remotely, trying to be a douche and only had good intentions at heart. I'm just getting frustrated with threads in this mailing list getting excessive long with people's digressions. Gladly it wasn't the case here and we can go on. Again, I'm sorry llyyr for the wrongful assumptions. -- Kind regards, Luciano (luc14n0), an openSUSE contributor.

Dirk Müller

17 Jul 17 Jul

10:21

Hi llyyr,

...

I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

This should now be fixed. I also fixed the redirects to the local mirrorcache instances for US, BR and AU.

Lubos Kocman

19 Jul 19 Jul

13:00

Would anybody mind if I'd make it a default on https://github.com/openSUSE/openSUSE-repos ? I'm aware of this weeks non-refresh of TW, but that does not seem to an issue as Ana mentioned that we won't have new snapshot for next 2-3 days. Cheers On Mon, 2023-07-17 at 12:21 +0200, Dirk Müller wrote:

...

Hi llyyr,

...
I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

This should now be fixed. I also fixed the redirects to the local mirrorcache instances for US, BR and AU.

Andrei Dziahel

13:19

No objections from me, quite the contrary. On Wed, Jul 19, 2023, 15:00 Lubos Kocman via openSUSE Factory < factory@lists.opensuse.org> wrote:

...

Would anybody mind if I'd make it a default on https://github.com/openSUSE/openSUSE-repos ?

I'm aware of this weeks non-refresh of TW, but that does not seem to an issue as Ana mentioned that we won't have new snapshot for next 2-3 days.

Cheers

On Mon, 2023-07-17 at 12:21 +0200, Dirk Müller wrote:

...
Hi llyyr,

...
I had to change https to http because of the following error from zypper: "Error message: Redirect to protocol "http" not supported or disabled in libcurl"

This should now be fixed. I also fixed the redirects to the local mirrorcache instances for US, BR and AU.

Dirk Müller

25 Jul 25 Jul

11:43

Hi all, so I received a bit of feedback on https://cdn.opensuse.org/ and would like to confirm that I think I have covered everything. the current configuration in place (as of 30 mins ago) should be addressing existing reports. I would like to have more testers and feedback. As of 30 minutes ago, /update/leap/15* and temporarily /tumbleweed/repo/oss is being served directly, bypassing the download.o.org redirector. The reason for the latter is that we still have some spare traffic allocation to burn before the end of the month, so I am redirecting more traffic to be served directly to see "how much it improves the situation" and "how much traffic it actually generates". As such, I would like to reinforce the use of http(s)://cdn.opensuse.org for your repos that are previously pointing to download.opensuse.org and please be critical in whether it actually improves latency, reliability and bandwidth for you or not. Please do not switch anything pointing to download.opensuse.org/repositories/, you will have a worse experience. Happy to hear your feedback (Probably off list to not flood this here). Greetings, Dirk

Luciano Santos

26 Jul 26 Jul

01:52

On Tue, 07/25/2023 at 13:43 +0200, Dirk Müller wrote:

...

Hi all,

so I received a bit of feedback on https://cdn.opensuse.org/ and would like to confirm that I think I have covered everything. the current configuration in place (as of 30 mins ago) should be addressing existing reports. I would like to have more testers and feedback. As of 30 minutes ago, /update/leap/15* and temporarily /tumbleweed/repo/oss is being served directly, bypassing the download.o.org redirector. The reason for the latter is that we still have some spare traffic allocation to burn before the end of the month, so I am redirecting more traffic to be served directly to see "how much it improves the situation" and "how much traffic it actually generates".

As such, I would like to reinforce the use of http(s)://cdn.opensuse.org for your repos that are previously pointing to download.opensuse.org and please be critical in whether it actually improves latency, reliability and bandwidth for you or not. Please do not switch anything pointing to download.opensuse.org/repositories/, you will have a worse experience.

Happy to hear your feedback (Probably off list to not flood this here).

Greetings, Dirk

OK, I missed thing email, before sending the previous one. So I have to drop download-o-o/repositories, then. In the meantime, I switched back to download-o-o, and for my surprise I still got timeouts. This time it was mirrorcache-br-2.opensuse.org, instead. But probably Zypper misreported cdn-o-o timing out when, in fact, it was something else - mirrorcache-br-2-o-o in this case. -- Kind regards, Luciano (luc14n0), an openSUSE contributor.

Luciano Santos

23:32

On Tue, 07/25/2023 at 22:52 -0300, Luciano Santos wrote:

...

In the meantime, I switched back to download-o-o, and for my surprise I still got timeouts. This time it was mirrorcache-br-2.opensuse.org, instead. But probably Zypper misreported cdn-o-o timing out when, in fact, it was something else - mirrorcache-br-2-o-o in this case.

Hmm, OK. I'm taking that back. Zypper were right before. Now I have switched my official openSUSE Tumbleweed repos to cdn-o-o, and it started getting timeouts for some packages in the OSS repo. I can remember at least kernel-default-6.4.4-1.1.x86_64 and libqt5-qtwebengine-5.15.14-1.6.x86_64 (both are beefier packages) were among those. Which means that it times out sometimes during repo metadata refresh and sometimes during package download. I see that here in Brazil [1]there are more than 4 POPs (points of presence), so that looks good, I'd supposed. And latency does look really good to me:

...

$ ping -c4 cdn.opensuse.org PING dualstack.n.sni.global.fastly.net (199.232.113.91) 56(84) bytes of data. 64 bytes from 199.232.113.91 (199.232.113.91): icmp_seq=1 ttl=56 time=21.3 ms 64 bytes from 199.232.113.91 (199.232.113.91): icmp_seq=2 ttl=56 time=21.3 ms 64 bytes from 199.232.113.91 (199.232.113.91): icmp_seq=3 ttl=56 time=20.6 ms 64 bytes from 199.232.113.91 (199.232.113.91): icmp_seq=4 ttl=56 time=22.8 ms

--- dualstack.n.sni.global.fastly.net ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 20.558/21.469/22.773/0.807 ms

...

$ ping -c4 mirrorcache-br.opensuse.org PING mirrorcache-br.opensuse.org (168.196.94.194) 56(84) bytes of data. 64 bytes from 168.196.94.194 (168.196.94.194): icmp_seq=1 ttl=51 time=40.5 ms 64 bytes from 168.196.94.194 (168.196.94.194): icmp_seq=2 ttl=51 time=41.9 ms 64 bytes from 168.196.94.194 (168.196.94.194): icmp_seq=3 ttl=51 time=39.1 ms 64 bytes from 168.196.94.194 (168.196.94.194): icmp_seq=4 ttl=51 time=40.4 ms

--- mirrorcache-br.opensuse.org ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 39.120/40.468/41.864/0.970 ms

...

$ ping -c4 mirrorcache-br-2.opensuse.org PING mirrorcache-br-2.opensuse.org (200.9.155.223) 56(84) bytes of data. 64 bytes from 200-9-155-223.tynahost.com (200.9.155.223): icmp_seq=1 ttl=56 time=35.9 ms 64 bytes from 200-9-155-223.tynahost.com (200.9.155.223): icmp_seq=2 ttl=56 time=35.9 ms 64 bytes from 200-9-155-223.tynahost.com (200.9.155.223): icmp_seq=3 ttl=56 time=36.0 ms 64 bytes from 200-9-155-223.tynahost.com (200.9.155.223): icmp_seq=4 ttl=56 time=35.8 ms

--- mirrorcache-br-2.opensuse.org ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 35.797/35.871/35.981/0.067 ms

...

$ ping -c4 mirrorcache-br-3.opensuse.org PING mirrorcache-br-3.opensuse.org (200.218.224.113) 56(84) bytes of data. 64 bytes from 200.218.224.113 (200.218.224.113): icmp_seq=1 ttl=55 time=37.4 ms 64 bytes from 200.218.224.113 (200.218.224.113): icmp_seq=2 ttl=55 time=32.2 ms 64 bytes from 200.218.224.113 (200.218.224.113): icmp_seq=3 ttl=55 time=38.2 ms 64 bytes from 200.218.224.113 (200.218.224.113): icmp_seq=4 ttl=55 time=33.2 ms

--- mirrorcache-br-3.opensuse.org ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 32.179/35.263/38.230/2.602 ms

1. https://www.fastly.com/network-map/ -- Kind regards, Luciano, openSUSE contributor (luc14n0)

Dirk Müller

27 Jul 27 Jul

08:55

Hi Luciano, Am Do., 27. Juli 2023 um 01:33 Uhr schrieb Luciano Santos :

...

Now I have switched my official openSUSE Tumbleweed repos to cdn-o-o, and it started getting timeouts for some packages in the OSS repo. I can remember at least kernel-default-6.4.4-1.1.x86_64 and libqt5-qtwebengine-5.15.14-1.6.x86_64 (both are beefier packages) were among those. Which means that it times out sometimes during repo metadata refresh and sometimes during package download.

ok, so the pattern is basically large packages are getting timeotus because they are not fetched yet and the fetch from origin takes too much time. can you please give me the output of ``` curl -sI https://cdn.opensuse.org/ | grep '^x-' ``` from the host/machine where you experience those issues? that'll help me pinning it down in the logfiles. Greetings, Dirk

Dirk Müller

16:15

Hi Luciano, Am Do., 27. Juli 2023 um 10:55 Uhr schrieb Dirk Müller :

...

ok, so the pattern is basically large packages are getting timeotus because they are not fetched yet and the fetch from origin takes too much time.

I might have found a solution for this problem, testing feedback appreciated. (had to purge the CDN, so its all empty now again unfortunately) Greetings, Dirk

Joe Salmeri

17:35

On 7/27/23 12:15, Dirk Müller wrote:

...

Hi Luciano,

Am Do., 27. Juli 2023 um 10:55 Uhr schrieb Dirk Müller:

...
ok, so the pattern is basically large packages are getting timeotus because they are not fetched yet and the fetch from origin takes too much time. I might have found a solution for this problem, testing feedback appreciated. (had to purge the CDN, so its all empty now again unfortunately)

Greetings, Dirk

FWIW, I just tried wget http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope... 100%[=================================================>] 543.54M 13.0MB/s in 42s wget http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE... 100%[=================================================>] 543.54M 6.89MB/s in 1m 51s Earlier when I tested I got very similar results. The download.opensuse.org seems really slow though compared to what I do zypper dup. Even when the download is 2 GB for the dup, it is usually done in like 15-20 min. -- Regards, Joe

Dirk Müller

28 Jul 28 Jul

23:06

Hi Joe, Am Do., 27. Juli 2023 um 19:36 Uhr schrieb Joe Salmeri :

...

FWIW, I just tried

wget http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope...

skelcd-installer-openSUSE-17.8 100%[=================================================>] 543.54M 13.0MB/s in 42s

wget http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE...

skelcd-installer-openSUSE-17.8 100%[=================================================>] 543.54M 6.89MB/s in 1m 51s

Earlier when I tested I got very similar results.

The behavior between both should be now the same or better, as it is no longer redirecting to downloadcontentcdn (which does not have that file because noone ever requests skelcd - that is an internal-only package), but it will redirect you to the nearest mirror. I also fixed cache invalidation and region detection, so things should be better now.

...

Even when the download is 2 GB for the dup, it is usually done in like 15-20 min.

Well, I find that rather slow ;) I have a relatively slow internet connection, and the connection overhead to request every file separately adds up a lot. Simpy setting ZYPP_MULTICURL=0 will make things faster, as well as switching to cdn.o.org. In my "time zypper -n in -d -t patterns games" testcase, which is a good mixture of small and larger downloads, switching to cdn improves wall clock time by 30-40%, which is noticeable. Greetings, Dirk

Felix Niederwanger

06:55

Hey, Is it expected that the directory listing for e.g. http://cdn.opensuse.org/update/leap/15.5/ throws a 403 error, while it is possible for http://download.opensuse.org/update/leap/15.5/ ? Best, phoenix

Dirk Müller

18:18

Hi Felix, Am Fr., 28. Juli 2023 um 08:55 Uhr schrieb Felix Niederwanger :

...

Is it expected that the directory listing for e.g. http://cdn.opensuse.org/update/leap/15.5/ throws a 403 error, while it is possible for http://download.opensuse.org/update/leap/15.5/ ?

Fixed. Thanks, Dirk

Luciano Santos

27 Jul 27 Jul

20:30

On Thu, 07/27/2023 at 10:55 +0200, Dirk Müller wrote:

...

ok, so the pattern is basically large packages are getting timeotus because they are not fetched yet and the fetch from origin takes too much time.

Oh, that makes sense. So, the first fetch is on demand. A little bit like YouTube not buffering a whole an hour video, where people might not really watch until the end.

...

can you please give me the output of

``` curl -sI https://cdn.opensuse.org/ | grep '^x-' ```

from the host/machine where you experience those issues? that'll help me pinning it down in the logfiles.

Sure thing:

...

$ curl -sI https://cdn.opensuse.org/ | grep '^x-' x-served-by: cache-cgh11149-CGH x-cache: MISS x-cache-hits: 0 x-timer: S1690488722.456590,VS0,VE63

...

I might have found a solution for this problem, testing feedback appreciated. (had to purge the CDN, so its all empty now again unfortunately)

As long as progress is made, it's worth it. Thanks for your work. -- Kind regards, Luciano, openSUSE contributor (luc14n0)

Bernhard M. Wiedemann

28 Jul 28 Jul

02:49

On 27/07/2023 10.55, Dirk Müller wrote:

...

ok, so the pattern is basically large packages are getting timeotus because they are not fetched yet and the fetch from origin takes too much time.

That reminds me that we had this same problem when we ran downloadcontent2 and friends with varnish. Which is why we let mirrorcache redirect large files to downloadcontent until mirrors became available.

Jiri Slaby

26 Jul 26 Jul

06:00

Hi, On 25. 07. 23, 13:43, Dirk Müller wrote:

...

As such, I would like to reinforce the use of http(s)://cdn.opensuse.org for your repos that are previously pointing to download.opensuse.org and please be critical in whether it actually improves latency, reliability and bandwidth for you or not.

It's slow as hell ;). 14 MB/s vs 56 MB/s of my close mirror.

...

$ wget http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope... --2023-07-26 07:48:46-- http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope... Resolving download.opensuse.org (download.opensuse.org)... 2001:67c:2178:8::13, 195.135.221.134 Connecting to download.opensuse.org (download.opensuse.org)|2001:67c:2178:8::13|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/skel... [following] --2023-07-26 07:48:46-- http://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/skel... Resolving mirror.karneval.cz (mirror.karneval.cz)... 2a02:8301:0:2::150, 89.102.0.150 Connecting to mirror.karneval.cz (mirror.karneval.cz)|2a02:8301:0:2::150|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/ske... [following] --2023-07-26 07:48:46-- https://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/ske... Connecting to mirror.karneval.cz (mirror.karneval.cz)|2a02:8301:0:2::150|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 569946669 (544M) [application/x-redhat-package-manager] Saving to: 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.2'

skelcd-installer-openSUSE-17.89-1.60.x8 100%[============================================================================>] 543.54M 57.1MB/s in 9.6s

2023-07-26 07:48:56 (56.8 MB/s) - 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.2' saved [569946669/569946669]

$ wget http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE... --2023-07-26 07:49:03-- http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE... Resolving cdn.opensuse.org (cdn.opensuse.org)... 2a04:4e42:41::347, 199.232.17.91 Connecting to cdn.opensuse.org (cdn.opensuse.org)|2a04:4e42:41::347|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 569946669 (544M) [application/x-redhat-package-manager] Saving to: 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.3'

skelcd-installer-openSUSE-17.89-1.60.x8 100%[============================================================================>] 543.54M 10.2MB/s in 38s

2023-07-26 07:49:41 (14.2 MB/s) - 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.3' saved [569946669/569946669]

regards, -- js suse labs

Fabian Vogt

06:56

Hi, Am Mittwoch, 26. Juli 2023, 08:00:17 CEST schrieb Jiri Slaby:

...

Hi,

On 25. 07. 23, 13:43, Dirk Müller wrote:

...
As such, I would like to reinforce the use of http(s)://cdn.opensuse.org for your repos that are previously pointing to download.opensuse.org and please be critical in whether it actually improves latency, reliability and bandwidth for you or not.

It's slow as hell ;). 14 MB/s vs 56 MB/s of my close mirror.

...
$ wget http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope...

Tested from my VPS it's similar, but a bit weird. I tried downloading this a few times from download.o.o as well as cdn.o.o. Depending on the chosen mirror, with d.o.o I got between 75MB/s-133MB/s. With cdn.o.o, the first four tries were with <15MB/s but starting from the 5th download I get ~226MB/s (!) now. I guess the CDN has several nodes and if the chosen node doesn't have the requested file, it's only transferred through a ~100MBit/s link. Cheers, Fabian

...

...
--2023-07-26 07:48:46-- http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope... Resolving download.opensuse.org (download.opensuse.org)... 2001:67c:2178:8::13, 195.135.221.134 Connecting to download.opensuse.org (download.opensuse.org)|2001:67c:2178:8::13|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/skel... [following] --2023-07-26 07:48:46-- http://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/skel... Resolving mirror.karneval.cz (mirror.karneval.cz)... 2a02:8301:0:2::150, 89.102.0.150 Connecting to mirror.karneval.cz (mirror.karneval.cz)|2a02:8301:0:2::150|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/ske... [following] --2023-07-26 07:48:46-- https://mirror.karneval.cz/pub/linux/opensuse/tumbleweed/repo/oss/x86_64/ske... Connecting to mirror.karneval.cz (mirror.karneval.cz)|2a02:8301:0:2::150|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 569946669 (544M) [application/x-redhat-package-manager] Saving to: 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.2'

skelcd-installer-openSUSE-17.89-1.60.x8 100%[============================================================================>] 543.54M 57.1MB/s in 9.6s

2023-07-26 07:48:56 (56.8 MB/s) - 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.2' saved [569946669/569946669]

$ wget http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE... --2023-07-26 07:49:03-- http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-openSUSE... Resolving cdn.opensuse.org (cdn.opensuse.org)... 2a04:4e42:41::347, 199.232.17.91 Connecting to cdn.opensuse.org (cdn.opensuse.org)|2a04:4e42:41::347|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 569946669 (544M) [application/x-redhat-package-manager] Saving to: 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.3'

skelcd-installer-openSUSE-17.89-1.60.x8 100%[============================================================================>] 543.54M 10.2MB/s in 38s

2023-07-26 07:49:41 (14.2 MB/s) - 'skelcd-installer-openSUSE-17.89-1.60.x86_64.rpm.3' saved [569946669/569946669]

regards,

Jiri Slaby

07:48

On 26. 07. 23, 8:56, Fabian Vogt wrote:

...

Hi,

Am Mittwoch, 26. Juli 2023, 08:00:17 CEST schrieb Jiri Slaby:

...
Hi,

On 25. 07. 23, 13:43, Dirk Müller wrote:

...
As such, I would like to reinforce the use of http(s)://cdn.opensuse.org for your repos that are previously pointing to download.opensuse.org and please be critical in whether it actually improves latency, reliability and bandwidth for you or not.

It's slow as hell ;). 14 MB/s vs 56 MB/s of my close mirror.

...
$ wget http://download.opensuse.org/tumbleweed/repo/oss/x86_64/skelcd-installer-ope...

Tested from my VPS it's similar, but a bit weird. I tried downloading this a few times from download.o.o as well as cdn.o.o. Depending on the chosen mirror, with d.o.o I got between 75MB/s-133MB/s. With cdn.o.o, the first four tries were with <15MB/s but starting from the 5th download I get ~226MB/s (!) now.

I guess the CDN has several nodes and if the chosen node doesn't have the requested file, it's only transferred through a ~100MBit/s link.

Hm, so now I download the same only with < 1 MB/s. -- js suse labs

Dirk Müller

11:08

Hi Fabian, Am Mi., 26. Juli 2023 um 08:56 Uhr schrieb Fabian Vogt :

...

I guess the CDN has several nodes and if the chosen node doesn't have the requested file, it's only transferred through a ~100MBit/s link.

Yes, it fetches the files from our original infrastructure, which is often saturated or beyond capacity. so the initial fetch takes a while, while with d.o.org it redirects you to a different site that serves the actual file. In the targetted (not yet implemented) production usage the redirect would be cached by CDN, rather than the file itself, which leads to the same mirror like before, just with a much lower latency. Greetings, Dirk

Dirk Müller

11:05

Hi Jiri, Am Mi., 26. Juli 2023 um 08:00 Uhr schrieb Jiri Slaby :

...

It's slow as hell ;). 14 MB/s vs 56 MB/s of my close mirror.

Is this the first access? the first access is very slow as the CDN is caching files on first access. I get 112MB/s (full uplink speed) with the skelcd link on cdn.opensuse.org here. There might be a country specific issue, I'm asking if I can find a host in cz to debug this further. Greetings, Dirk

Jan Engelhardt

14 Jul 14 Jul

19:05

On Friday 2023-07-14 16:44, Dirk Müller wrote:

...

For a while I've been working in the background to get a sponsored CDN subscription. I am happy to report that Fastly.com has agreed to sponsor the openSUSE project with bandwidth. We primarily intend to use it to improve the reachability (latency) of download.opensuse.org in various areas of the world, but of course can expand it to other usecases as well.

Hm. Although it gives a 4 hop reduction and (with that) a 25% improvement in latency numbers for German Vodafone home networks, that seems also descriptive of the lackluster peering performanace of VF. I digress.. Now, if only libzypp would do parallel downloads, the extra latency of d.o.o would be much less of a non-issue since one would be offered at least a handful of files most of the time... ;^)

Jacob Michalskie

20:20

Am 14. Juli 2023 21:05:40 MESZ schrieb Jan Engelhardt :

...

On Friday 2023-07-14 16:44, Dirk Müller wrote:

...
For a while I've been working in the background to get a sponsored CDN subscription. I am happy to report that Fastly.com has agreed to sponsor the openSUSE project with bandwidth. We primarily intend to use it to improve the reachability (latency) of download.opensuse.org in various areas of the world, but of course can expand it to other usecases as well.

Hm. Although it gives a 4 hop reduction and (with that) a 25% improvement in latency numbers for German Vodafone home networks, that seems also descriptive of the lackluster peering performanace of VF. I digress..

Now, if only libzypp would do parallel downloads, the extra latency of d.o.o would be much less of a non-issue since one would be offered at least a handful of files most of the time... ;^)

I don't know if we can reduce the issues with libzypp just to that. It would be kind of nice to take some notes from other software in the space and allow zypp to query a single file with a list of mirror urls to avoid redirections altogether. That is a feature that existed in dnf for ever, and the only reason why it never worked on openSUSE distros (with dnf) was simply that this was never enabled in mirrorbrain. It also sounds a bit easier to only deal with one request per command instead of one per package on our side, and the local machine is going to be able to tell which mirror has better performance for it (be it latency or bandwidth) than we ever could tell from the redirector. LCP [Jake] https://lcp.world/

Luciano Santos

15 Jul 15 Jul

00:41

On Fri, 07/14/2023 at 21:05 +0200, Jan Engelhardt wrote:

...

Hm. Although it gives a 4 hop reduction and (with that) a 25% improvement in latency numbers for German Vodafone home networks, that seems also descriptive of the lackluster peering performanace of VF. I digress..

Now, if only libzypp would do parallel downloads, the extra latency of d.o.o would be much less of a non-issue since one would be offered at least a handful of files most of the time... ;^)

As far as I can tell [1]libzypp/Zypper devs have been working on that for a while now, a new media backend has basically been written to support parallel downloads already, now they are adapting the code to make use of this new implementation. 1. https://github.com/openSUSE/zypper/issues/104 -- Kind regards, Luciano (luc14n0), an openSUSE contributor.

Luciano Santos

26 Jul 26 Jul

01:24

On Fri, 07/14/2023 at 16:44 +0200, Dirk Müller wrote:

...

Hi all, ...

Happy to hear about any feedback that you might have.

Greetings, Dirk

Hi there Dirk, I've been getting timeouts quite frequently. They mostly occur during repository metadata refreshes, and sometimes during package downloads, like:

...

Retrieving: kernel-default-6.4.4-1.1.x86_64.rpm .....................................................[error (520.0 KiB/s)] Timeout exceeded when accessing 'http://cdn.opensuse.org/tumbleweed/repo/oss/x86_64/kernel-default-6.4.4-1.1....'. Retrying in 30 seconds...

And for repositories, they mostly occur in devel/home project repos. Are we already hitting bandwith limitations, or something? -- Kind regards, Luciano (luc14n0), an openSUSE contributor.

286

Age (days ago)

300

Last active (days ago)

List overview

Download

29 comments

14 participants

participants (14)

Andrei Borzenkov
Andrei Dziahel
Bernhard M. Wiedemann
Claudio Marcial Peon
Dirk Müller
Fabian Vogt
Felix Niederwanger
Jacob Michalskie
Jan Engelhardt
Jiri Slaby
Joe Salmeri
llyyr
Lubos Kocman
Luciano Santos

Trial to use a CDN for download.opensuse.org

tags

participants (14)