[suse-mirror] Opensuse master rsync server performance
Related to the heads up about rolling updates for the upcoming release: Have you investigated speeding up your main rsync servers? ftp.acc.umu.se syncs from stage.opensuse.org, and we are seeing sync-times in excess of 24 hours at times. Sync-bandwidth on our end is limited to 1 Gbps, but the speeds we are getting are much lower so we suspect that the upstream server is the bottleneck. If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- I parked my harddisk - and got a ticket! =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Tue, 2017-02-21 at 10:10 +0100, Niklas Edmundsson wrote:
Related to the heads up about rolling updates for the upcoming release:
Have you investigated speeding up your main rsync servers?
ftp.acc.umu.se syncs from stage.opensuse.org, and we are seeing sync-times in excess of 24 hours at times.
Sync-bandwidth on our end is limited to 1 Gbps, but the speeds we are getting are much lower so we suspect that the upstream server is the bottleneck.
If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success.
/Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- I parked my harddisk - and got a ticket! =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Hi Niklas, We have a somewhat tiered system. Registered mirrors are meant to be using stage.opensuse.org meanwhile anyone can use rsync.opensuse.org. We recently had a problem with sync between stage.o.o and rsync.o.o, which effectively negated the effectiveness of that second tier of mirrors and also led to uses falling back to stage.o.o in the event of not finding a better mirror. This problem could have been ongoing as long as the last few months. This obviously resulted in a lot more contention on stage.opensuse.org, making things worse for both our registered mirror sync and our users. We've resolved that problem and to prevent a repeat of users flooding stage.o.o we are going to be asking everyone registered to use stage.o.o to do so (I have the email drafted, but you beat me to the question). rsync.opensuse.org will remain as a public mirror source and should now be able to better fulfil it's intended role as the 'mirror of last resort' for our users instead of stage.o.o, leaving more of stage.o.o's bandwidth for mirror syncing. I hope that this resolution & rebalancing of the secondary/public mirror tier over the last few days will have already sped up what you're seeing with stage.opensuse.org.
From my personal monitoring it seems to be the case, but there is a small risk that we could be temporarily making things a little worse, if a lot more consistent.
But with a consistent baseline of all of our registered mirror hosts we can come up with a sensible strategy for a more robust & effectively tiering. Debian's way doesn't sound like a bad idea at all. Meanwhile, in parallel to everything I said above, we are actively working on increasing our bandwidth for stage.opensuse.org. Regards, -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Richard Brown (rbrown@suse.de) wrote on Tue, Feb 21, 2017 at 06:30:53AM BRT:
We recently had a problem with sync between stage.o.o and rsync.o.o, which effectively negated the effectiveness of that second tier of mirrors and also led to uses falling back to stage.o.o in the event of not finding a better mirror.
This problem could have been ongoing as long as the last few months.
Yes. We suffer A LOT with slowness too.
rsync.opensuse.org will remain as a public mirror source and should now be able to better fulfil it's intended role as the 'mirror of last resort' for our users instead of stage.o.o, leaving more of stage.o.o's bandwidth for mirror syncing.
I hope that this resolution & rebalancing of the secondary/public mirror tier over the last few days will have already sped up what you're seeing with stage.opensuse.org.
Not here (opensuse.c3sl.ufpr.br). Last update took 18h40m with a speed less than 1MB/s. That's not enough for the rate of change of your repository. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Tue, 2017-02-21 at 08:27 -0300, Carlos Carvalho wrote:
Richard Brown (rbrown@suse.de) wrote on Tue, Feb 21, 2017 at 06:30:53AM BRT:
We recently had a problem with sync between stage.o.o and rsync.o.o, which effectively negated the effectiveness of that second tier of mirrors and also led to uses falling back to stage.o.o in the event of not finding a better mirror.
This problem could have been ongoing as long as the last few months.
Yes. We suffer A LOT with slowness too.
rsync.opensuse.org will remain as a public mirror source and should now be able to better fulfil it's intended role as the 'mirror of last resort' for our users instead of stage.o.o, leaving more of stage.o.o's bandwidth for mirror syncing. I hope that this resolution & rebalancing of the secondary/public mirror tier over the last few days will have already sped up what you're seeing with stage.opensuse.org.
Not here (opensuse.c3sl.ufpr.br). Last update took 18h40m with a speed less than 1MB/s. That's not enough for the rate of change of your repository.
Thank you Carlos, very useful to know the issue is more widespread I've just been talking with our network guys and the plans to resolve the situation are even more advanced than I already referred to. stage.opensuse.org is hosted at SUSE HQ which is currently served via 2x 300MBit links. This is being augmented by an additional 1GBit link expected to go live on March 1st (ie. just over a week from now). Assuming that all goes well, one of the 300MBit links will be retired at the end of March This is just stage 1 of improvements. In early May we should get a new fiber to the building, which is expected to go live on June 1st at a speed no less than 4Gbit. The remaining 300Mbit link will then be retired, leaving the 1GBit link as our backup So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months. I still want to investigate Niklas' suggestion because I think it's a good idea regardless, and generally we're making an effort to better keep an eye on what is going on with our mirrors, but I wanted to give a full breakdown of where we are so everyone knows where everything fits into context. Thanks again, -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Am 21.02.2017 um 16:15 schrieb Richard Brown:
stage.opensuse.org is hosted at SUSE HQ which is currently served via 2x 300MBit links.
This is being augmented by an additional 1GBit link expected to go live on March 1st (ie. just over a week from now). Assuming that all goes well, one of the 300MBit links will be retired at the end of March
This is just stage 1 of improvements. In early May we should get a new fiber to the building, which is expected to go live on June 1st at a speed no less than 4Gbit. The remaining 300Mbit link will then be retired, leaving the 1GBit link as our backup
So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months.
Why aren't you asking teamix to mirror openSUSE on the mirror of N-IX (mirrors.n-ix.net)? You are directly linked there (at least the member list and the index page of the mirror are telling this). teamix is connected via two times 10 Gbit to the exchange point, maybe the mirror can also provide 10 Gbit. The mirror is providing Gentoo, fedora and grml over rsync and I am sure they would be able to add openSUSE. This server could act as a second rsync.o.o server. Greetings, Christian -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Quoting "Estelmann, Christian" <c.estelmann@gmx.net>:
Am 21.02.2017 um 16:15 schrieb Richard Brown:
stage.opensuse.org is hosted at SUSE HQ which is currently served via 2x 300MBit links.
This is being augmented by an additional 1GBit link expected to go live on March 1st (ie. just over a week from now). Assuming that all goes well, one of the 300MBit links will be retired at the end of March
This is just stage 1 of improvements. In early May we should get a new fiber to the building, which is expected to go live on June 1st at a speed no less than 4Gbit. The remaining 300Mbit link will then be retired, leaving the 1GBit link as our backup
So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months.
Why aren't you asking teamix to mirror openSUSE on the mirror of N-IX (mirrors.n-ix.net)? You are directly linked there (at least the member list and the index page of the mirror are telling this). teamix is connected via two times 10 Gbit to the exchange point, maybe the mirror can also provide 10 Gbit. The mirror is providing Gentoo, fedora and grml over rsync and I am sure they would be able to add openSUSE. This server could act as a second rsync.o.o server. If more servers are needed next to rsync.o.o we can also help out. ftp.nluug.nl already is using stage.o.o and has mirrorred opensuse-full-with-factory and opensuse-ports We have a 10 Gbit connection, and can handle some more load.
Grtz, Mike
Greetings, Christian -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
-- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Tue, 21 Feb 2017, ftp-admin@nluug.nl wrote:
So in short, we're going to have double bandwidth in a week, and 8x our current bandwidth in a few months.
Regardless, I think that you have to do the math and factor in available bandwidth, change volume (current and future), expected mirror freshness and figure out a number of mirrors that stage.o.o can handle. Btw, if you have lots and lots of files you also need to cater for that as well since it takes time and some bandwidth to send the file lists... Developer awareness of the limitations also helps. Oh, and point the public rsync.o.o to a decent mirror somewhere that doesn't affect stage.o.o connectivity ...
Why aren't you asking teamix to mirror openSUSE on the mirror of N-IX (mirrors.n-ix.net)? You are directly linked there (at least the member list and the index page of the mirror are telling this). teamix is connected via two times 10 Gbit to the exchange point, maybe the mirror can also provide 10 Gbit. The mirror is providing Gentoo, fedora and grml over rsync and I am sure they would be able to add openSUSE. This server could act as a second rsync.o.o server. If more servers are needed next to rsync.o.o we can also help out. ftp.nluug.nl already is using stage.o.o and has mirrorred opensuse-full-with-factory and opensuse-ports We have a 10 Gbit connection, and can handle some more load.
I would say that being able to serve 10 Gbit is a sane prerequisite for a primary mirror. Also, connectivity/peering within the intended region is also important. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- */ \* <- Tribbles having a swordfight =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart. It seems the bandwidth increase didn't solve the problem. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:
Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.
It seems the bandwidth increase didn't solve the problem.
Then please provide us with traceroutes and the source IP Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue Regards, -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Fri, 10 Mar 2017, Richard Brown wrote:
On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:
Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.
It seems the bandwidth increase didn't solve the problem.
Then please provide us with traceroutes and the source IP
Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue
A tip would also be to investigate Round Trip Times and TCP buffer tunings. The Linux/distro defaults are sometimes way too small. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- I like it better in the dark. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Richard Brown (rbrown@suse.de) wrote on Fri, Mar 10, 2017 at 10:25:14AM BRT:
On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:
Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.
It seems the bandwidth increase didn't solve the problem.
Then please provide us with traceroutes and the source IP
Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue
IP is 2801:82:80ff:8000::2. Here's the traceroute: sagres#~[14:52] traceroute6 stage.opensuse.org traceroute to stage.opensuse.org (2001:67c:2178:8::10), 30 hops max, 80 byte packets 1 2801:82:80ff:8000::1 (2801:82:80ff:8000::1) 0.137 ms 0.140 ms 0.150 ms 2 p9-v1610-araucaria-c3sl.pop-pr.rnp.br (2801:82:0:f2::1) 0.659 ms 0.953 ms 1.278 ms 3 e1-5-v103-lapa-araucaria.pop-pr.rnp.br (2801:82:0:ff02::1) 10.162 ms 10.246 ms 10.196 ms 4 pr-lanpr.bkb-v6.rnp.br (2001:12f0:0:fe::81) 0.332 ms 0.322 ms 0.308 ms 5 sp2-pr-oi.bkb-v6.rnp.br (2001:12f0:0:fc::3d) 7.154 ms 7.309 ms 7.269 ms 6 sp-sp2.bkb-v6.rnp.br (2001:12f0:0:fd::25) 7.296 ms 7.445 ms 7.441 ms 7 2001:12f0:0:fc::22 (2001:12f0:0:fc::22) 112.978 ms 112.991 ms 112.998 ms 8 2001:478:124::176 (2001:478:124::176) 137.839 ms 137.573 ms 112.907 ms 9 100ge11-1.core1.atl1.he.net (2001:470:0:18d::1) 157.225 ms 157.234 ms 157.222 ms 10 100ge11-1.core1.ash1.he.net (2001:470:0:114::2) 206.721 ms 205.364 ms 205.355 ms 11 100ge3-1.core1.nyc4.he.net (2001:470:0:299::2) 170.035 ms 145.401 ms 170.202 ms 12 100ge10-1.core1.nyc6.he.net (2001:470:0:259::2) 193.476 ms 168.689 ms 193.458 ms 13 * * * 14 xe-2-0-7-2076.lon10.core-backbone.com (2a01:4a0:0:2076::19) 237.390 ms 237.595 ms 237.426 ms 15 xe-1-0-4-2077.fra10.core-backbone.com (2a01:4a0:0:2077::4) 248.469 ms 248.459 ms 223.714 ms 16 2a01:4a0:0:2003::2 (2a01:4a0:0:2003::2) 251.743 ms 226.968 ms 226.961 ms 17 2a01:4a0:1338:e8::2 (2a01:4a0:1338:e8::2) 276.947 ms 252.407 ms 276.895 ms 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Nothing obviously wrong. From here it seems the problem is in the final destination. I had to (yet again) stop the update because it had pulled only 2.1GB in 5.5hours. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Fri, 2017-03-10 at 14:58 -0300, Carlos Carvalho wrote:
Richard Brown (rbrown@suse.de) wrote on Fri, Mar 10, 2017 at 10:25:14AM BRT:
On Fri, 2017-03-10 at 10:05 -0300, Carlos Carvalho wrote:
Slow downloads persist at opensuse.c3sl.ufpr.br :-( :-( We're getting speeds from ~7MB/s to 200KB/s. I still have to interrupt and restart.
It seems the bandwidth increase didn't solve the problem.
Then please provide us with traceroutes and the source IP
Without that we have no way to diagnose where the issue could possibly be now we can rule out the host and our direct bandwidth as the issue
IP is 2801:82:80ff:8000::2. Here's the traceroute:
sagres#~[14:52] traceroute6 stage.opensuse.org traceroute to stage.opensuse.org (2001:67c:2178:8::10), 30 hops max, 80 byte packets 1 2801:82:80ff:8000::1 (2801:82:80ff:8000::1) 0.137 ms 0.140 ms 0.150 ms 2 p9-v1610-araucaria-c3sl.pop-pr.rnp.br (2801:82:0:f2::1) 0.659 ms 0.953 ms 1.278 ms 3 e1-5-v103-lapa-araucaria.pop-pr.rnp.br (2801:82:0:ff02::1) 10.162 ms 10.246 ms 10.196 ms 4 pr-lanpr.bkb-v6.rnp.br (2001:12f0:0:fe::81) 0.332 ms 0.322 ms 0.308 ms 5 sp2-pr-oi.bkb-v6.rnp.br (2001:12f0:0:fc::3d) 7.154 ms 7.309 ms 7.269 ms 6 sp-sp2.bkb-v6.rnp.br (2001:12f0:0:fd::25) 7.296 ms 7.445 ms 7.441 ms 7 2001:12f0:0:fc::22 (2001:12f0:0:fc::22) 112.978 ms 112.991 ms 112.998 ms 8 2001:478:124::176 (2001:478:124::176) 137.839 ms 137.573 ms 112.907 ms 9 100ge11-1.core1.atl1.he.net (2001:470:0:18d::1) 157.225 ms 157.234 ms 157.222 ms 10 100ge11-1.core1.ash1.he.net (2001:470:0:114::2) 206.721 ms 205.364 ms 205.355 ms 11 100ge3-1.core1.nyc4.he.net (2001:470:0:299::2) 170.035 ms 145.401 ms 170.202 ms 12 100ge10-1.core1.nyc6.he.net (2001:470:0:259::2) 193.476 ms 168.689 ms 193.458 ms 13 * * * 14 xe-2-0-7-2076.lon10.core-backbone.com (2a01:4a0:0:2076::19) 237.390 ms 237.595 ms 237.426 ms 15 xe-1-0-4-2077.fra10.core-backbone.com (2a01:4a0:0:2077::4) 248.469 ms 248.459 ms 223.714 ms 16 2a01:4a0:0:2003::2 (2a01:4a0:0:2003::2) 251.743 ms 226.968 ms 226.961 ms 17 2a01:4a0:1338:e8::2 (2a01:4a0:1338:e8::2) 276.947 ms 252.407 ms 276.895 ms 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Nothing obviously wrong. From here it seems the problem is in the final destination. I had to (yet again) stop the update because it had pulled only 2.1GB in 5.5hours.
And it looks the same with mtr? I'm not sure we have enough information for me to go to our network provider and blame them for the last hop.. -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Hi On Tue, 21 Feb 2017 10:10:39 +0100 (MET) Niklas Edmundsson wrote:
Have you investigated speeding up your main rsync servers?
We tried that multiple times, but the limited amount of available personal resources and bandwidth (beside the fact that we do not had access to some core infrastructure resources like DNS and other stuff) is not easily fixable for the openSUSE Heroes.
If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success.
We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side). This worked fine until we got all those tons of repositories: OBS (the open build service) was only able to support one "push server" in the past. So the workflow ended up like: OBS pushing to stage.opensuse.org and stage.opensuse.org pushing to our dedicated "push mirrors" (while other mirrors still pull from the same storage). After that, a scan is triggered to check if the content on the mirrors is correct - and only after that, the content is added to the MirrorBrain database, which is behind the redirector on download.o.o You might notice multiple bottle necks just by reading the paragraph above. While the developer resources to fix issues in OBS and in MirrorBrain (both open source, both welcome patches, btw) are still low, we try fix other bottle necks that are available. Any ideas, any support is welcome and we hope that we see some progress here and in many other areas, too, in the next weeks. With kind regards, Lars *member_of_the_openSUSE_heroes_team* -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Lars Vogdt (Lars.Vogdt@suse.com) wrote on Wed, Feb 22, 2017 at 08:52:04AM BRT:
Hi
On Tue, 21 Feb 2017 10:10:39 +0100 (MET) Niklas Edmundsson wrote:
If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success.
We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side).
This isn't necessary. It's enough to trigger the updates via a ssh connection that lauches the rsync from the mirror to the master. We do it with Debian. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Wed, 22 Feb 2017, Carlos Carvalho wrote:
Lars Vogdt (Lars.Vogdt@suse.com) wrote on Wed, Feb 22, 2017 at 08:52:04AM BRT:
Hi
On Tue, 21 Feb 2017 10:10:39 +0100 (MET) Niklas Edmundsson wrote:
If you can't increase the performance of the main rsync server it might be beneficial to investigate a tiered approach where a few select primary high-bandwidth mirrors sync from the master site and the rest sync from the closest primary mirror. This is the way Debian has been doing it for quite some time with great success.
We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side).
Direct push-rsync with write access is generally frowned upon by mirror admins.
This isn't necessary. It's enough to trigger the updates via a ssh connection that lauches the rsync from the mirror to the master. We do it with Debian.
Or you can do it the archlinux way and have a lastupdate timestamp that you download/check often and then trigger a full sync when it's changed. Sure, you get a slight delay compared to a true push syncscheme but it's much easier to set up (no additional auth steps). As long as your sync script uses a local lock file to avoid running multiple syncs concurrently you can run it say every 10 minutes without much harm. See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first. Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- COMMITTEE: 12 people doing the work of one. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Niklas Edmundsson (nikke@acc.umu.se) wrote on Thu, Feb 23, 2017 at 04:45:36AM BRT:
Or you can do it the archlinux way and have a lastupdate timestamp that you download/check often and then trigger a full sync when it's changed. Sure, you get a slight delay compared to a true push syncscheme but it's much easier to set up (no additional auth steps). As long as your sync script uses a local lock file to avoid running multiple syncs concurrently you can run it say every 10 minutes without much harm.
Exactly. We do it for a bunch of repositories. Unfortunately o.o doesn't have such a timestamp. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Hi On Thu, 23 Feb 2017 08:45:36 +0100 (MET) Niklas Edmundsson wrote:
We have a few mirror admins that allowed us to push content to them (simply via separate rsync module on their side).
Direct push-rsync with write access is generally frowned upon by mirror admins.
I like the Debian way. :-) I will bring that up to the admins that maintain the stage.o.o machine.
Or you can do it the archlinux way and have a lastupdate timestamp that you download/check often and then trigger a full sync when it's changed. Sure, you get a slight delay compared to a true push syncscheme but it's much easier to set up (no additional auth steps). As long as your sync script uses a local lock file to avoid running multiple syncs concurrently you can run it say every 10 minutes without much harm.
See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first.
Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO.
Also an idea - but I need to check with the admins of the stage.o.o server first. A problem might be that OBS alone has >80.000 repositories aka directories - that change all the time. But I'm sure now that we will find a solution that works for everyone. Thanks for all your input! With kind regards, Lars -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Thu, 23 Feb 2017, Lars Vogdt wrote:
See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first.
Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO.
Also an idea - but I need to check with the admins of the stage.o.o server first. A problem might be that OBS alone has >80.000 repositories aka directories - that change all the time.
You'll have the same issue with a push-approach. The workaround is to trigger the update when needed, be it via push or updating the lastupdate file. By the way, it might be beneficial to point out to the developers that gazillions of files and directories that are to be kept up to date at all times isn't really a design that's easy to mirror/distribute... "But it works on my laptop" isn't really a good argument, even though it's frequently used by devs ;-) I've seen various efforts on partial updates of large repositories, generally with the master server providing lists of what's changed. For example, I've been intending to look into setting up instant mirroring for CPAN (perl). See http://www.cpan.org/misc/how-to-mirror.html#Instant_mirroring for their take on solving the problem. Looks easy to use on the client side at least, I have no clue on what's needed on the server side...
But I'm sure now that we will find a solution that works for everyone.
Thanks for all your input!
Glad it's appreciated :-) /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- * <- Regular Tribble * <- Odo disguised as a tribble =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Quoting Niklas Edmundsson <nikke@acc.umu.se>:
On Thu, 23 Feb 2017, Lars Vogdt wrote:
See https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors and https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh for details. There are more projects using a timestamp file, but archlinux is the one that I came to think of first.
Their script uses curl to get the lastupdate file from the main mirror, which I personally think is a flaw since it can trigger a sync before the mirror you pull from has updated. It's easy enough to use rsync to pull the lastupdate file from the same mirror you sync from, a better option IMHO.
Also an idea - but I need to check with the admins of the stage.o.o server first. A problem might be that OBS alone has >80.000 repositories aka directories - that change all the time.
You'll have the same issue with a push-approach.
The workaround is to trigger the update when needed, be it via push or updating the lastupdate file.
By the way, it might be beneficial to point out to the developers that gazillions of files and directories that are to be kept up to date at all times isn't really a design that's easy to mirror/distribute...
"But it works on my laptop" isn't really a good argument, even though it's frequently used by devs ;-)
I've seen various efforts on partial updates of large repositories, generally with the master server providing lists of what's changed.
For example, I've been intending to look into setting up instant mirroring for CPAN (perl). See http://www.cpan.org/misc/how-to-mirror.html#Instant_mirroring for their take on solving the problem. Looks easy to use on the client side at least, I have no clue on what's needed on the server side... We are using imm for that for a long time, works like a charm http://www.staff.science.uu.nl/~penni101/iim/ On the client it is very easy to setup, we used to rsync CPAN in the past. Since imm was installed we did not see long running rsyncs to CPAN anymore.
But I'm sure now that we will find a solution that works for everyone.
Thanks for all your input!
Glad it's appreciated :-)
/Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- * <- Regular Tribble * <- Odo disguised as a tribble =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
-- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
ftpmirror-beheer@nluug.nl (ftpmirror-beheer@nluug.nl) wrote on Thu, Feb 23, 2017 at 09:59:40AM BRT:
Quoting Niklas Edmundsson <nikke@acc.umu.se>:
I've seen various efforts on partial updates of large repositories, generally with the master server providing lists of what's changed.
For example, I've been intending to look into setting up instant mirroring for CPAN (perl). See http://www.cpan.org/misc/how-to-mirror.html#Instant_mirroring for their take on solving the problem. Looks easy to use on the client side at least, I have no clue on what's needed on the server side... We are using imm for that for a long time, works like a charm http://www.staff.science.uu.nl/~penni101/iim/ On the client it is very easy to setup, we used to rsync CPAN in the past. Since imm was installed we did not see long running rsyncs to CPAN anymore.
We do something similar: if the master provides a list of all objects, in rsync format, we use it and do not scan the whole repository upstream. I can give the rsync command (just a few options) if the admins of stage are interested. OTOH, we have a sophisticated mirror update script and won't use anything else here. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
And now this! Oh well. It's in your best interest to fix it asap... @ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Wed, Feb 22, 2017 at 08:46:53PM -0300, Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap...
@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
Same for my mirror since yesterday. The last three sync have now failed. Adrian
On Thu, Feb 23, 2017 at 10:03:30AM +0100, Adrian Reber wrote:
On Wed, Feb 22, 2017 at 08:46:53PM -0300, Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap...
@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
Same for my mirror since yesterday. The last three sync have now failed.
I've had the same for a while, so I've been using the -4 flag to force IPv4. Tim. -- Tim Bishop, Computing Officer, School of Computing, University of Kent. PGP Key: 0x6C226B37FDF38D55 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Thu, Feb 23, 2017 at 09:10:20AM +0000, Tim Bishop wrote:
On Thu, Feb 23, 2017 at 10:03:30AM +0100, Adrian Reber wrote:
On Wed, Feb 22, 2017 at 08:46:53PM -0300, Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap...
@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
Same for my mirror since yesterday. The last three sync have now failed.
I've had the same for a while, so I've been using the -4 flag to force IPv4.
My problems are also IPv4, I am not sure my v6 address is even in the ACLs Adrian
On 02/23/2017 12:46 AM, Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap... @ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
Just a 'me three' here, access denied since yesterday evening, both on IPv4 and IPv6 though we are a registered mirror. The reason is probably this (from the rsync output):
You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution. Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;) stage.opensuse.org has address 195.135.221.130 stage.opensuse.org has IPv6 address 2001:67c:2178:8::10 That's the IPs that all authoritative nameservers for opensuse.org (all under novell.com?) claim...
And while I'm already writing a mail, my n cents on the recent discussion: - I very much appreciate stage.opensuse.org being restricted to registered mirrors again, this was actually the case in the past - The performance problems on stage.o.o currently are most certainly not due to bad peering as some have assumed. Our FTP is just 7 hops and 8 ms away from stage, I know for a fact that none of the lines between us and their ISP is in any way close to capacity, and we still see sync times of 12hours+ for opensuse-full-with-factory (see history on https://ftp.fau.de/cgi-bin/show-mirror-status.cgi?mirror=opensuse). This is just an overloaded connection to SuSE headquarters, also evidenced by the fact that their GW often pings at around 100 ms instead of 8. For most mirrors problems should disappear as soon as they receive their bandwidth upgrade. Regards, -- Michael Meier, FTP-Admin Friedrich-Alexander-Universitaet Erlangen-Nuernberg Regionales Rechenzentrum Erlangen Martensstrasse 1, 91058 Erlangen, Germany Tel.: +49 9131 85-28973, Fax: +49 9131 302941 rrze-ftp-admins@fau.de blogs.fau.de/ftp/ -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Am 23.02.2017 um 10:25 schrieb Michael Meier (FTP Admin):
The reason is probably this (from the rsync output):
You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution. Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;)
We did setup a new mirror recently, so I can tell you that this message is not appearing just from yesterday onward, but is shown all the time. We requested a change in the ACLs some days ago. This has been done yesterday according to the ticket.. maybe that is related. It doesn't work out by the way, maybe someone messed up the ACLs. Matthias -- Dipl.-Inf. Matthias Hunstock UniRZ der TU Ilmenau, Raum 07 Tel.: +49 3677 69-1289 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
I've asked our admins to look into these ACL reports urgently Thank you for letting us know On Thu, 2017-02-23 at 10:36 +0100, Matthias Hunstock wrote:
Am 23.02.2017 um 10:25 schrieb Michael Meier (FTP Admin):
The reason is probably this (from the rsync output):
You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution.
Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;)
We did setup a new mirror recently, so I can tell you that this message is not appearing just from yesterday onward, but is shown all the time.
We requested a change in the ACLs some days ago. This has been done yesterday according to the ticket.. maybe that is related. It doesn't work out by the way, maybe someone messed up the ACLs.
Matthias
-- Dipl.-Inf. Matthias Hunstock UniRZ der TU Ilmenau, Raum 07 Tel.: +49 3677 69-1289
-- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Thu, 2017-02-23 at 10:25 +0100, Michael Meier (FTP Admin) wrote:
You are accessing rsync.opensuse.org under an outdated IP. Please update your configuration or use DNS resolution.
Well we naturally DO use DNS resolution, and we're trying to access stage.opensuse.org not rsync.opensuse.org ;)
Yeah, I fixed the motd from stage.opensuse.org - it will no longer erroneously complain -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Wed, 22 Feb 2017 20:46:53 -0300 Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap...
@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) --
I wanted to fix this - and checked the logs: 2017/02/23 08:20:02 [2602] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 08:20:03 [2602] rsync denied on module opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) But as I see: 2017/02/23 12:20:02 [32766] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 11:20:10 [32766] rsync on opensuse-full-really-everything/opensuse from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) I guess someone from the admins already fixed the issue. with kind regards, Lars -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
On Thu, 2017-02-23 at 12:45 +0100, Lars Vogdt wrote:
On Wed, 22 Feb 2017 20:46:53 -0300 Carlos Carvalho wrote:
And now this! Oh well. It's in your best interest to fix it asap...
@ERROR: access denied to opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) --
I wanted to fix this - and checked the logs: 2017/02/23 08:20:02 [2602] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 08:20:03 [2602] rsync denied on module opensuse-full-really-everything from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
But as I see: 2017/02/23 12:20:02 [32766] connect from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2) 2017/02/23 11:20:10 [32766] rsync on opensuse-full-really-everything/opensuse from sagres.c3sl.ufpr.br (2801:82:80ff:8000::2)
I guess someone from the admins already fixed the issue.
with kind regards, Lars
Yup, we think we've fixed the issues - though if anyone wants to disagree we will be VERY interested I'm currently setting up an external canary to help get to the bottom of exactly what went wrong so we can prevent it happening again. -- Richard Brown Technical Lead - openQA openSUSE Chairman Phone +4991174053-361 SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Richard Brown (rbrown@suse.de) wrote on Thu, Feb 23, 2017 at 08:49:53AM BRT:
Yup, we think we've fixed the issues - though if anyone wants to disagree we will be VERY interested
Our update at 11h20 UTC did get started. It's quite large, it'll take forever due to the slowness of stage but it's better than no updates. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
participants (11)
-
Adrian Reber -
Carlos Carvalho -
Estelmann, Christian -
ftp-admin＠nluug.nl -
ftpmirror-beheer＠nluug.nl -
Lars Vogdt -
Matthias Hunstock -
Michael Meier (FTP Admin) -
Niklas Edmundsson -
Richard Brown -
Tim Bishop