Hi, On Mon, Oct 04, 2010 at 11:02:24AM +0200, Carsten Otto wrote:

Well, this was just a bugfix, because our intention was to mirror everything. I'd like to reevaluate the situation after a while of serving the complete data set.

It seems we still need to synchronize way more data than we actually send out to users (-3.5 MBit/sec average in the last week): http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.pn... Based on these numbers I'd like to remove OBS from our mirror to save disk space, bandwidth and some other ressources. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211

On Tue, Oct 26, 2010 at 02:29:13PM +0200, Stephan Kulow wrote:

Well, as I tried to convince you last time: don't mirror everything but interesting OBS repos. OBS repos are just too large and differ too much in use to make this a all or nothing decision.

Sounds good, too. Do you have any idea what is "interesting"? Or should we rather analyze our log files? Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211

Hi, I'd like to revive the discussion regarding the traffic generated by OBS. We have enough disk space to mirror everything that OpenSUSE has to offer and we are happy to upload as much as there is demand. However, the OBS suite we get from stage::buildservice-repos is not a good thing to mirror. Based on our statistics we roughly have the same amount of outgoing traffic (users downloading from our mirror) as incoming traffic (rsync pushes): http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23year.pn... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23month.p... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.pn... Depending on the current situation, sometimes we upload more than we download (good!), sometimes we do not. Because of all the rsync pushes (involving disk writes and scans) there is a high price (performance-wise) we pay to have the data, although there is not enough demand for that "expensive" data. In order to ease the burden on our disk backend (filling all the caches, slowing down other accesses) caused by traffic without much use we would like to have any of the following solutions: - we do not mirror OBS anymore - you increase our traffic substantially - you offer a "hot" subset of data that is high in demand (we can provide logfiles, but we do not want to invest much time to compute this subset (and re-compute it over time?)) Please discuss/pick. Thanks, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211

Am Donnerstag, 19. Mai 2011 schrieb Carsten Otto:

On Mon, May 09, 2011 at 12:23:18PM +0200, Carsten Otto wrote:

...

Please discuss/pick.

Any news? Hi,

If you want to work on that, I can give you some log file snippets to experiment with. Other than that, I'll wait for another 7 months if your oppinion changes again and won't rush into a solution. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org

Dear Stephan and all others, we removed the rsync push module and configured to mirror buildservice-repos-main (every six hours). Here, too, we will keep an eye on the traffic ratio. Please keep us informed if you have any results or opionions about the previously stated questions and problems. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211

On Wed, 1 Jun 2011 20:29:33 -0300 Carlos Carvalho wrote:

It seems stage has got a v6 IP, 2001:67c:2178:8::10.

yes we support v6 and v4 now on most of our infrastructure. the remaining pieces should be ready for the IPv6 day.

However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name?

yes your entry is using the hostname, what would be the hostname of your v6 address? darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org

Marcus Rückert (mrueckert@suse.de) wrote on 3 June 2011 11:47:

On Wed, 1 Jun 2011 20:29:33 -0300 Carlos Carvalho wrote:

...

It seems stage has got a v6 IP, 2001:67c:2178:8::10.

yes we support v6 and v4 now on most of our infrastructure. the remaining pieces should be ready for the IPv6 day.

...

However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name?

yes your entry is using the hostname, what would be the hostname of your v6 address?

sagres.c3sl.ufpr.br At the moment there's a problem with the delegation of reverse dns for our zones, so the name won't work. There are two possibilities: continue using v4 until our dns resolves or put in the IPv6 (which you can get from the name). -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org

Marcus Rückert (mrueckert@suse.de) wrote on 3 June 2011 17:00:

added the ipv6 IP aswell. can you please test if that works for you?

Working, thanks.

and please let me know when i can remove it again. thank you in advance.

Sure. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org

Carsten Otto wrote:

Dear all,

I'm one of the admins of ftp.halifax.rwth-aachen.de. We've been mirroring opensuse content for quite a while, including opensuse-buildservice.

As a follow up to our discussion started around 8 years ago, I'd like to remove opensuse-buildservice from our mirror. In the past years, the server rarely uploaded more data to opensuse users (devs?) than what we needed to download using rsync (the difference is around -13.6 MBit/sec on average). The previous attempts at solving this issue by switching modules and getting data via rsync pushes sadly did not help.

Hallo Carsten

In case you have an idea of how to properly make use of our resources (we're not an archive of stale data!), please let me know.

I would suggest you add suitable "--exclude"s to your rsync command. Unless I have completely misunderstood, I think that will solve the issue. Best regards Per -- Per Jessen, Zürich (17.1°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org

Carsten Otto wrote:

Hi Per,

we already had that discussion in 2011, without any solution. I still don't think it is the mirror admins' responsibility to dig through log files, make up some statistics, or anything similar to find what to mirror and what not. If you want us to provide a mirror, provide a suitable rsync repository.

Right now we're serving around 60 projects, and there simply is no time to do this kind of additional work.

As a consequence I'll remove OBS from ftp.halifax.rwth-aachen.de now.

Alright, thanks for letting us know. -- Per Jessen, Zürich (19.5°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org