Richard Brown (rbrown@suse.de) wrote on Tue, Feb 21, 2017 at 06:30:53AM BRT:

We recently had a problem with sync between stage.o.o and rsync.o.o, which effectively negated the effectiveness of that second tier of mirrors and also led to uses falling back to stage.o.o in the event of not finding a better mirror.

This problem could have been ongoing as long as the last few months.

Yes. We suffer A LOT with slowness too.

rsync.opensuse.org will remain as a public mirror source and should now be able to better fulfil it's intended role as the 'mirror of last resort' for our users instead of stage.o.o, leaving more of stage.o.o's bandwidth for mirror syncing.

I hope that this resolution & rebalancing of the secondary/public mirror tier over the last few days will have already sped up what you're seeing with stage.opensuse.org.

Not here (opensuse.c3sl.ufpr.br). Last update took 18h40m with a speed less than 1MB/s. That's not enough for the rate of change of your repository. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Am 21.02.2017 um 16:15 schrieb Richard Brown:

stage.opensuse.org is hosted at SUSE HQ which is currently served via 2x 300MBit links.

This is being augmented by an additional 1GBit link expected to go live on March 1st (ie. just over a week from now). Assuming that all goes well, one of the 300MBit links will be retired at the end of March

This is just stage 1 of improvements. In early May we should get a new fiber to the building, which is expected to go live on June 1st at a speed no less than 4Gbit. The remaining 300Mbit link will then be retired, leaving the 1GBit link as our backup

So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months.

Why aren't you asking teamix to mirror openSUSE on the mirror of N-IX (mirrors.n-ix.net)? You are directly linked there (at least the member list and the index page of the mirror are telling this). teamix is connected via two times 10 Gbit to the exchange point, maybe the mirror can also provide 10 Gbit. The mirror is providing Gentoo, fedora and grml over rsync and I am sure they would be able to add openSUSE. This server could act as a second rsync.o.o server. Greetings, Christian -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On Tue, 21 Feb 2017, ftp-admin@nluug.nl wrote:

...

...

So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months.

Regardless, I think that you have to do the math and factor in available bandwidth, change volume (current and future), expected mirror freshness and figure out a number of mirrors that stage.o.o can handle. Btw, if you have lots and lots of files you also need to cater for that as well since it takes time and some bandwidth to send the file lists... Developer awareness of the limitations also helps. Oh, and point the public rsync.o.o to a decent mirror somewhere that doesn't affect stage.o.o connectivity ...

...

Why aren't you asking teamix to mirror openSUSE on the mirror of N-IX (mirrors.n-ix.net)? You are directly linked there (at least the member list and the index page of the mirror are telling this). teamix is connected via two times 10 Gbit to the exchange point, maybe the mirror can also provide 10 Gbit. The mirror is providing Gentoo, fedora and grml over rsync and I am sure they would be able to add openSUSE. This server could act as a second rsync.o.o server. If more servers are needed next to rsync.o.o we can also help out. ftp.nluug.nl already is using stage.o.o and has mirrorred opensuse-full-with-factory and opensuse-ports We have a 10 Gbit connection, and can handle some more load.

I would say that being able to serve 10 Gbit is a sane prerequisite for a primary mirror. Also, connectivity/peering within the intended region is also important. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- */ \* <- Tribbles having a swordfight =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:

Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.

It seems the bandwidth increase didn't solve the problem.

Then please provide us with traceroutes and the source IP Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue Regards, -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Richard Brown (rbrown@suse.de) wrote on Fri, Mar 10, 2017 at 10:25:14AM BRT:

On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:

...

Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.

It seems the bandwidth increase didn't solve the problem.

Then please provide us with traceroutes and the source IP

Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue

IP is 2801:82:80ff:8000::2. Here's the traceroute: sagres#~[14:52] traceroute6 stage.opensuse.org traceroute to stage.opensuse.org (2001:67c:2178:8::10), 30 hops max, 80 byte packets 1 2801:82:80ff:8000::1 (2801:82:80ff:8000::1) 0.137 ms 0.140 ms 0.150 ms 2 p9-v1610-araucaria-c3sl.pop-pr.rnp.br (2801:82:0:f2::1) 0.659 ms 0.953 ms 1.278 ms 3 e1-5-v103-lapa-araucaria.pop-pr.rnp.br (2801:82:0:ff02::1) 10.162 ms 10.246 ms 10.196 ms 4 pr-lanpr.bkb-v6.rnp.br (2001:12f0:0:fe::81) 0.332 ms 0.322 ms 0.308 ms 5 sp2-pr-oi.bkb-v6.rnp.br (2001:12f0:0:fc::3d) 7.154 ms 7.309 ms 7.269 ms 6 sp-sp2.bkb-v6.rnp.br (2001:12f0:0:fd::25) 7.296 ms 7.445 ms 7.441 ms 7 2001:12f0:0:fc::22 (2001:12f0:0:fc::22) 112.978 ms 112.991 ms 112.998 ms 8 2001:478:124::176 (2001:478:124::176) 137.839 ms 137.573 ms 112.907 ms 9 100ge11-1.core1.atl1.he.net (2001:470:0:18d::1) 157.225 ms 157.234 ms 157.222 ms 10 100ge11-1.core1.ash1.he.net (2001:470:0:114::2) 206.721 ms 205.364 ms 205.355 ms 11 100ge3-1.core1.nyc4.he.net (2001:470:0:299::2) 170.035 ms 145.401 ms 170.202 ms 12 100ge10-1.core1.nyc6.he.net (2001:470:0:259::2) 193.476 ms 168.689 ms 193.458 ms 13 * * * 14 xe-2-0-7-2076.lon10.core-backbone.com (2a01:4a0:0:2076::19) 237.390 ms 237.595 ms 237.426 ms 15 xe-1-0-4-2077.fra10.core-backbone.com (2a01:4a0:0:2077::4) 248.469 ms 248.459 ms 223.714 ms 16 2a01:4a0:0:2003::2 (2a01:4a0:0:2003::2) 251.743 ms 226.968 ms 226.961 ms 17 2a01:4a0:1338:e8::2 (2a01:4a0:1338:e8::2) 276.947 ms 252.407 ms 276.895 ms 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Nothing obviously wrong. From here it seems the problem is in the final destination. I had to (yet again) stop the update because it had pulled only 2.1GB in 5.5hours. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On Wed, 22 Feb 2017, Carlos Carvalho wrote:

Lars Vogdt (Lars.Vogdt@suse.com) wrote on Wed, Feb 22, 2017 at 08:52:04AM BRT:

...

Hi

On Tue, 21 Feb 2017 10:10:39 +0100 (MET) Niklas Edmundsson wrote:

...

If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success.

We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side).

Direct push-rsync with write access is generally frowned upon by mirror admins.

This isn't necessary. It's enough to trigger the updates via a ssh connection that lauches the rsync from the mirror to the master. We do it with Debian.

Or you can do it the archlinux way and have a lastupdate timestamp that you download/check often and then trigger a full sync when it's changed. Sure, you get a slight delay compared to a true push syncscheme but it's much easier to set up (no additional auth steps). As long as your sync script uses a local lock file to avoid running multiple syncs concurrently you can run it say every 10 minutes without much harm. See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first. Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- COMMITTEE: 12 people doing the work of one. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Hi On Thu, 23 Feb 2017 08:45:36 +0100 (MET) Niklas Edmundsson wrote:

...

...

We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side).

Direct push-rsync with write access is generally frowned upon by mirror admins.

I like the Debian way. :-) I will bring that up to the admins that maintain the stage.o.o machine.

Or you can do it the archlinux way and have a lastupdate timestamp that you download/check often and then trigger a full sync when it's changed. Sure, you get a slight delay compared to a true push syncscheme but it's much easier to set up (no additional auth steps). As long as your sync script uses a local lock file to avoid running multiple syncs concurrently you can run it say every 10 minutes without much harm.

See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first.

Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO.

Also an idea - but I need to check with the admins of the stage.o.o server first. A problem might be that OBS alone has >80.000 repositories aka directories - that change all the time. But I'm sure now that we will find a solution that works for everyone. Thanks for all your input! With kind regards, Lars -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Quoting Niklas Edmundsson :

On Thu, 23 Feb 2017, Lars Vogdt wrote:

...

...

See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first.

Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO.

Also an idea - but I need to check with the admins of the stage.o.o server first. A problem might be that OBS alone has >80.000 repositories aka directories - that change all the time.

You'll have the same issue with a push-approach.

The workaround is to trigger the update when needed, be it via push or updating the lastupdate file.

By the way, it might be beneficial to point out to the developers that gazillions of files and directories that are to be kept up to date at all times isn't really a design that's easy to mirror/distribute...

"But it works on my laptop" isn't really a good argument, even though it's frequently used by devs ;-)

I've seen various efforts on partial updates of large repositories, generally with the master server providing lists of what's changed.

For example, I've been intending to look into setting up instant mirroring for CPAN (perl). See http://www.cpan.org/misc/how-to-mirror.html#Instant_mirroring for their take on solving the problem. Looks easy to use on the client side at least, I have no clue on what's needed on the server side... We are using imm for that for a long time, works like a charm http://www.staff.science.uu.nl/~penni101/iim/ On the client it is very easy to setup, we used to rsync CPAN in the past. Since imm was installed we did not see long running rsyncs to CPAN anymore.

...

But I'm sure now that we will find a solution that works for everyone.

Thanks for all your input!

Glad it's appreciated :-)

/Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- * <- Regular Tribble * <- Odo disguised as a tribble =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

-- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On Thu, Feb 23, 2017 at 10:03:30AM +0100, Adrian Reber wrote:

On Wed, Feb 22, 2017 at 08:46:53PM -0300, Carlos Carvalho wrote:

...

And now this! Oh well. It's in your best interest to fix it asap...

@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)

Same for my mirror since yesterday. The last three sync have now failed.

I've had the same for a while, so I've been using the -4 flag to force IPv4. Tim. -- Tim Bishop, Computing Officer, School of Computing, University of Kent. PGP Key: 0x6C226B37FDF38D55 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On 02/23/2017 12:46 AM, Carlos Carvalho wrote:

And now this! Oh well. It's in your best interest to fix it asap... @ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)

Just a 'me three' here, access denied since yesterday evening, both on IPv4 and IPv6 though we are a registered mirror. The reason is probably this (from the rsync output):

You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution. Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;) stage.opensuse.org has address 195.135.221.130 stage.opensuse.org has IPv6 address 2001:67c:2178:8::10 That's the IPs that all authoritative nameservers for opensuse.org (all under novell.com?) claim...

And while I'm already writing a mail, my n cents on the recent discussion: - I very much appreciate stage.opensuse.org being restricted to registered mirrors again, this was actually the case in the past - The performance problems on stage.o.o currently are most certainly not due to bad peering as some have assumed. Our FTP is just 7 hops and 8 ms away from stage, I know for a fact that none of the lines between us and their ISP is in any way close to capacity, and we still see sync times of 12hours+ for opensuse-full-with-factory (see history on https://ftp.fau.de/cgi-bin/show-mirror-status.cgi?mirror=opensuse). This is just an overloaded connection to SuSE headquarters, also evidenced by the fact that their GW often pings at around 100 ms instead of 8. For most mirrors problems should disappear as soon as they receive their bandwidth upgrade. Regards, -- Michael Meier, FTP-Admin Friedrich-Alexander-Universitaet Erlangen-Nuernberg Regionales Rechenzentrum Erlangen Martensstrasse 1, 91058 Erlangen, Germany Tel.: +49 9131 85-28973, Fax: +49 9131 302941 rrze-ftp-admins@fau.de blogs.fau.de/ftp/ -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

I've asked our admins to look into these ACL reports urgently Thank you for letting us know On Thu, 2017-02-23 at 10:36 +0100, Matthias Hunstock wrote:

Am 23.02.2017 um 10:25 schrieb Michael Meier (FTP Admin):

...

The reason is probably this (from the rsync output):

...

You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution.

Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;)

We did setup a new mirror recently, so I can tell you that this message is not appearing just from yesterday onward, but is shown all the time.

We requested a change in the ACLs some days ago. This has been done yesterday according to the ticket.. maybe that is related. It doesn't work out by the way, maybe someone messed up the ACLs.

Matthias

-- Dipl.-Inf. Matthias Hunstock UniRZ der TU Ilmenau, Raum 07 Tel.: +49 3677 69-1289

-- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

On Wed, 22 Feb 2017 20:46:53 -0300 Carlos Carvalho wrote:

And now this! Oh well. It's in your best interest to fix it asap...

@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) --

I wanted to fix this - and checked the logs: 2017/02/23 08:20:02 [2602] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 08:20:03 [2602] rsync denied on module opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) But as I see: 2017/02/23 12:20:02 [32766] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 11:20:10 [32766] rsync on opensuse-full-really-everything/opensuse from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) I guess someone from the admins already fixed the issue. with kind regards, Lars -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Richard Brown (rbrown@suse.de) wrote on Thu, Feb 23, 2017 at 08:49:53AM BRT:

Yup, we think we've fixed the issues - though if anyone wants to disagree we will be VERY interested

Our update at 11h20 UTC did get started. It's quite large, it'll take forever due to the slowness of stage but it's better than no updates. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org