- openSUSE Mirrors

[mirror] Suboptimal (for mirrors) download pattern by opensuse clients
by Niklas Edmundsson 21 Feb '20

21 Feb '20

Hi, am I the only mirror admin that finds the current behavior of opensuse clients suboptimal? Requests by "ZYpp 17.11.4 (curl 7.60.0) openSUSE-Leap-15.1-x86_64" etc seem to be done with 256 kb chunk size, always, as an example: GET bytes=0-262143 /mirror/opensuse.org/tumbleweed/repo/oss/x86_64/libqt5-qtwebengine-5.14.1-1.5.x86_64.rpm GET bytes=262144-524287 /mirror/opensuse.org/tumbleweed/repo/oss/x86_64/libqt5-qtwebengine-5.14.1-1.5.x86_64.rpm That's a silly small size, since TCP won't be able to ramp window sizes and get good speed before those 256k are done. Also, we get int($filesize/256k) entries in our logs for each download. To make matters worse, the thing seems to do some kind of round robin between sites, with this pattern being the most ineffective looking from a mirror admin standpoint: GET bytes=2097152-2359295 /mirror/opensuse.org/tumbleweed/repo/oss/x86_64/libqt5-qtwebengine-5.14.1-1.5.x86_64.rpm GET bytes=2621440-2883583 /mirror/opensuse.org/tumbleweed/repo/oss/x86_64/libqt5-qtwebengine-5.14.1-1.5.x86_64.rpm Since the OS normally does read-ahead on file system reads, it will read-ahead after byte 2359295 in preparation for the next read(). In this case though, that's in vain as the request never comes but the next data read is instead byte 2621440 and forward... OS read-ahead is most commonly in the 64kB-1MB range, so it's not unlikely that the entire 256k gap inbetween is read from disk without being used... Downloading files this way is just plain stupid, IMHO. I don't know what problem this behavior is supposed so solve, but it's definitely not beneficial for us as a mirror, and I think it's hurting your end users as well. If you want more bandwidth from us, request larger chunks (or whole files). The TCP window will grow and you'll get the performance (within the limits of 10 gigabit networking for one download). If you want to spread the load between mirrors, use larger chunks, and specifically avoid small chunks and striped access. In any case, merge requests! If you're going to request a number of consecutive chunks, do it in one request, preferrably as one range, to make the most of the tcp connection you've set up. My minimum suggestion would be to bump the chunk-size to multiple megabytes at the minimum, possibly varying depending on download performance, aiming for each GET taking at least a couple of seconds to allow for TCP to ramp speed (and reduce the noise in our logs). In extreme cases we're seeing multiple tens of GET:s each second for some downloads, I'm guessing the rate throttles due to the RTT latency (ping time) and not some real bandwidth limit... /Nikke - admin of ftp.acc.umu.se -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke(a)acc.umu.se --------------------------------------------------------------------------- NOW (n), adv: A moment in time that has already passed. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

[mirror] openSuSE mirror problem
by ariel sabiguero yawelak 04 Dec '19

04 Dec '19

Hello. We had some trouble with one F.O. link to our faculty, thus, we had to "move" our mirror (espejito.fder.edu.uy) to a new IP address: 164.73.178.222 from a different provider. The provider states that the problem would be solved by 9th dec. As soon as it is fixed, we will return to previous operation conditions. Please let us know if there are further inconvinients or problems caused by this issue. regards ariel -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

[mirror] heads up: 15.2 will arrive on the download server soon
by Ludwig Nussel 07 Aug '19

07 Aug '19

Hi, Just a heads up. 15.2 development starts soon, therefore we'll have it on download.opensuse.org soon too. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

[mirror] packages builds not propagating to mirrors again -- for several days?
by PGNet Dev 11 Jul '19

11 Jul '19

My OBS kernel build's been through a couple of build cycles over the past few days. Currently, successfully built packages are not propagating to the mirrors; this is unfortunately a not-uncommon result, often going on for days. No idea what the actual problem is. What needs to be done to get propagation working consistently? E.g., this https://download.opensuse.org/repositories/home:/pgnd:/Kernel:/stable/openS… Mirrors for kernel-default-5.2.0-29.1.g7501499.x86_64.rpm File information Filename: kernel-default-5.2.0-29.1.g7501499.x86_64.rpm Path: /repositories/home:/pgnd:/Kernel:/stable/openSUSE_Leap_15.1/x86_64/kernel-default-5.2.0-29.1.g7501499.x86_64.rpm Size: 67M (70770360 bytes) Last modified: Wed, 10 Jul 2019 11:03:04 GMT (Unix time: 1562756584) Download file from preferred mirror Reliable downloads Metalink /repositories/home:/pgnd:/Kernel:/stable/openSUSE_Leap_15.1/x86_64/kernel-default-5.2.0-29.1.g7501499.x86_64.rpm.meta4 (IETF Metalink) /repositories/home:/pgnd:/Kernel:/stable/openSUSE_Leap_15.1/x86_64/kernel-default-5.2.0-29.1.g7501499.x86_64.rpm.metalink (old (v3) Metalink) Mirrors List of best mirrors for IP address 99.xx.xx.xx, located at 36.270241,-121.8074545 in United States (US). Map showing the closest mirrors No mirror was found I am very sorry, but no mirror was found. Feel free to download directly: /repositories/home:/pgnd:/Kernel:/stable/openSUSE_Leap_15.1/x86_64/kernel-default-5.2.0-29.1.g7501499.x86_64.rpm -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

2 4

[mirror] openSUSE Leap 15.1 on the way
by Ludwig Nussel 17 May '19

17 May '19

Hi, openSUSE Leap 15.1 is on the way to your servers! Release date is Wednesday 22nd 12:00 UTC. As usual access to iso images is limited until then. stage mirrors should be able to pull the iso nevertheless (and keep the restricted permission). So maybe a good chance to check your setup so it's ready for Wednesday. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

[mirror] openSUSE Leap 15.1 on the way
by Ludwig Nussel 17 May '19

17 May '19

Hi, openSUSE Leap 15.1 is on the way to your servers! Release date is Wednesday 22nd 12:00 UTC. As usual access to iso images is limited until then. stage mirrors should be able to pull the iso nevertheless (and keep the restricted permission). So maybe a good chance to check your setup so it's ready for Wednesday. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

[mirror] Connection issues?
by Matthias Hunstock 25 Mar '19

25 Mar '19

Hi all, since around Friday our mirror has some issues connecting to the master: rsync: failed to connect to stage.opensuse.org (195.135.221.130): Connection refused (111) Is there something wrong with the server or should I have a look on our end of the cable? Regards Matthias -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

4 5

[mirror] Error Message: SSL certificate problem: unable to get local issuer certificate
by Felix Miata 26 Jan '19

26 Jan '19

$SUBJECT is what happens trying to zypper ref TW's BS repos, such as mozilla and Base:/System. Is this explained by ICC device profile database major outage on https://status.opensuse.org/ ? -- Evolution as taught in public schools is religion, not science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

2 4

[mirror] Ensure using -H flag with rsync for duplicate hardlinks
by Jimmy Berry 12 Oct '18

12 Oct '18

As it was noted in "Announcing new module opensuse-history for your bandwidth consideration" thread and included in documentation I figure it might be worth a call out in case anyone missed it. If you are mirroring the history module or modules that include it the -H flag is crucial to avoid lots of duplicate files. -- Jimmy -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org To contact the owner, email: mirror+owner(a)opensuse.org

1 0

Re: [suse-mirror] Mirroring buildservice
by Stephan Kulow 07 Oct '18

07 Oct '18

Am Monday 04 October 2010 schrieb Carsten Otto: > Hi, > > On Mon, Oct 04, 2010 at 10:44:40AM +0200, Stephan Kulow wrote: > > > We get our data pushed from OpenSUSE directly. If there are files > > > missing, please push better :) > > > > I see that *Factory* is excluded for your mirror, that explains a good > > share of what I see as it matches the project name - one of the most > > busy ones actually. > > Oh, I thought we were mirroring everything OpenSUSE has to offer. What > do we need to do to actually get that? Could you just add Factory to the > pushes (and do a full sync)? We will extend disk space on demand, so it > may take a while until we are complete (as an alternative could you tell > us how much space we need?). > I don't think it makes sense to mirror everything. Many files are see so little traffic that mirroring them costs way too much. And now I'm really puzzled what your intention is - you strarted the thread with "remove all mirrors" and now you want even more stuff noone downloads? Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org For additional commands, e-mail: mirror+help(a)opensuse.org

7 20