Re: [suse-mirror] Mirroring buildservice
Am Monday 04 October 2010 schrieb Carsten Otto:
Hi,
On Mon, Oct 04, 2010 at 10:44:40AM +0200, Stephan Kulow wrote:
We get our data pushed from OpenSUSE directly. If there are files missing, please push better :)
I see that *Factory* is excluded for your mirror, that explains a good share of what I see as it matches the project name - one of the most busy ones actually.
Oh, I thought we were mirroring everything OpenSUSE has to offer. What do we need to do to actually get that? Could you just add Factory to the pushes (and do a full sync)? We will extend disk space on demand, so it may take a while until we are complete (as an alternative could you tell us how much space we need?).
I don't think it makes sense to mirror everything. Many files are see so little traffic that mirroring them costs way too much. And now I'm really puzzled what your intention is - you strarted the thread with "remove all mirrors" and now you want even more stuff noone downloads? Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Mon, Oct 04, 2010 at 10:58:55AM +0200, Stephan Kulow wrote:
I don't think it makes sense to mirror everything. Many files are see so little traffic that mirroring them costs way too much. And now I'm really puzzled what your intention is - you strarted the thread with "remove all mirrors" and now you want even more stuff noone downloads?
Well, this was just a bugfix, because our intention was to mirror everything. I'd like to reevaluate the situation after a while of serving the complete data set. Bye, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Hi, On Mon, Oct 04, 2010 at 11:02:24AM +0200, Carsten Otto wrote:
Well, this was just a bugfix, because our intention was to mirror everything. I'd like to reevaluate the situation after a while of serving the complete data set.
It seems we still need to synchronize way more data than we actually send out to users (-3.5 MBit/sec average in the last week): http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.pn... Based on these numbers I'd like to remove OBS from our mirror to save disk space, bandwidth and some other ressources. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Am Dienstag 26 Oktober 2010 schrieb Carsten Otto:
Hi,
On Mon, Oct 04, 2010 at 11:02:24AM +0200, Carsten Otto wrote:
Well, this was just a bugfix, because our intention was to mirror everything. I'd like to reevaluate the situation after a while of serving the complete data set.
It seems we still need to synchronize way more data than we actually send out to users (-3.5 MBit/sec average in the last week): http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.p ng
Based on these numbers I'd like to remove OBS from our mirror to save disk space, bandwidth and some other ressources.
Well, as I tried to convince you last time: don't mirror everything but interesting OBS repos. OBS repos are just too large and differ too much in use to make this a all or nothing decision. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Tue, Oct 26, 2010 at 02:29:13PM +0200, Stephan Kulow wrote:
Well, as I tried to convince you last time: don't mirror everything but interesting OBS repos. OBS repos are just too large and differ too much in use to make this a all or nothing decision.
Sounds good, too. Do you have any idea what is "interesting"? Or should we rather analyze our log files? Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Am Dienstag 26 Oktober 2010 schrieb Carsten Otto:
On Tue, Oct 26, 2010 at 02:29:13PM +0200, Stephan Kulow wrote:
Well, as I tried to convince you last time: don't mirror everything but interesting OBS repos. OBS repos are just too large and differ too much in use to make this a all or nothing decision.
Sounds good, too. Do you have any idea what is "interesting"? Or should we rather analyze our log files?
Well, checking log files would surely help. But we have buildservice-repos and buildservice-repos-main and we can add more. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, I'd like to revive the discussion regarding the traffic generated by OBS. We have enough disk space to mirror everything that OpenSUSE has to offer and we are happy to upload as much as there is demand. However, the OBS suite we get from stage::buildservice-repos is not a good thing to mirror. Based on our statistics we roughly have the same amount of outgoing traffic (users downloading from our mirror) as incoming traffic (rsync pushes): http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23year.pn... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23month.p... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.pn... Depending on the current situation, sometimes we upload more than we download (good!), sometimes we do not. Because of all the rsync pushes (involving disk writes and scans) there is a high price (performance-wise) we pay to have the data, although there is not enough demand for that "expensive" data. In order to ease the burden on our disk backend (filling all the caches, slowing down other accesses) caused by traffic without much use we would like to have any of the following solutions: - we do not mirror OBS anymore - you increase our traffic substantially - you offer a "hot" subset of data that is high in demand (we can provide logfiles, but we do not want to invest much time to compute this subset (and re-compute it over time?)) Please discuss/pick. Thanks, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
On Mon, May 09, 2011 at 12:23:18PM +0200, Carsten Otto wrote:
Please discuss/pick.
Any news? -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Am Donnerstag, 19. Mai 2011 schrieb Carsten Otto:
On Mon, May 09, 2011 at 12:23:18PM +0200, Carsten Otto wrote:
Please discuss/pick.
Any news? Hi,
If you want to work on that, I can give you some log file snippets to experiment with. Other than that, I'll wait for another 7 months if your oppinion changes again and won't rush into a solution. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Thu, May 19, 2011 at 02:01:24PM +0200, Stephan Kulow wrote:
If you want to work on that, I can give you some log file snippets to experiment with. Other than that, I'll wait for another 7 months if your oppinion changes again and won't rush into a solution.
As said, we do not want to invest time on that. I do not understand your comment regarding the opinion change. The problem is the same as originally stated in 2010 and the situation did not change (the traffic ratio did not increase, although changes were made to the set of available files [which was kind of a bug fix that might have changed the ratio - but did not]). Your offer "I can also offer that we offer a push mode only containing popular software gathered from our redirect logs. That should increase your http/rsync ratio too :)" still sounds good. This would, indeed, solve the problem. To summarize: - there is no available solution known to me - we do not want to analyze our log files (we can still given them to you, of course) - the situation ("opinion") did not change in the last few months The original suggestion to disable mirroring of OBS on other mirrors, too, still is a good idea (from my point of view). Feel free to think about it (again?). Please clarify your view on this. Thanks, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Dear Stephan and all others, we removed the rsync push module and configured to mirror buildservice-repos-main (every six hours). Here, too, we will keep an eye on the traffic ratio. Please keep us informed if you have any results or opionions about the previously stated questions and problems. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
It seems stage has got a v6 IP, 2001:67c:2178:8::10. However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name? -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Wed, 1 Jun 2011 20:29:33 -0300 Carlos Carvalho <carlos@fisica.ufpr.br> wrote:
It seems stage has got a v6 IP, 2001:67c:2178:8::10.
yes we support v6 and v4 now on most of our infrastructure. the remaining pieces should be ready for the IPv6 day.
However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name?
yes your entry is using the hostname, what would be the hostname of your v6 address? darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Might as well add us to v6 as well. mirror.metrocast.net / 65.175.128.102 ipv6-mirror.metrocast.net / 2606:f400:400:0:65:175:128:102 Both of those hosts and ips need access to the opensuse-full acl. Eventually ipv6-mirror will go away and just be a AAAA record for mirror. -- Ben On 6/3/2011 5:47 AM, Marcus Rückert wrote:
On Wed, 1 Jun 2011 20:29:33 -0300 Carlos Carvalho <carlos@fisica.ufpr.br> wrote:
It seems stage has got a v6 IP, 2001:67c:2178:8::10.
yes we support v6 and v4 now on most of our infrastructure. the remaining pieces should be ready for the IPv6 day.
However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name?
yes your entry is using the hostname, what would be the hostname of your v6 address?
darix
-- Benjamin Hamilton, RHCE Systems Engineer MetroCast Cablevision (p) 603-330-7799 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Marcus Rückert (mrueckert@suse.de) wrote on 3 June 2011 11:47:
On Wed, 1 Jun 2011 20:29:33 -0300 Carlos Carvalho <carlos@fisica.ufpr.br> wrote:
It seems stage has got a v6 IP, 2001:67c:2178:8::10.
yes we support v6 and v4 now on most of our infrastructure. the remaining pieces should be ready for the IPv6 day.
However it doesn't allow sagres.c3sl.ufpr.br to update via v6. Did you forget to update the access list or are you using the name?
yes your entry is using the hostname, what would be the hostname of your v6 address?
sagres.c3sl.ufpr.br At the moment there's a problem with the delegation of reverse dns for our zones, so the name won't work. There are two possibilities: continue using v4 until our dns resolves or put in the IPv6 (which you can get from the name). -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Fri, 3 Jun 2011 10:32:44 -0300 Carlos Carvalho <carlos@fisica.ufpr.br> wrote:
sagres.c3sl.ufpr.br
At the moment there's a problem with the delegation of reverse dns for our zones, so the name won't work. There are two possibilities: continue using v4 until our dns resolves or put in the IPv6 (which you can get from the name).
added the ipv6 IP aswell. can you please test if that works for you? and please let me know when i can remove it again. thank you in advance. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Marcus Rückert (mrueckert@suse.de) wrote on 3 June 2011 17:00:
added the ipv6 IP aswell. can you please test if that works for you?
Working, thanks.
and please let me know when i can remove it again. thank you in advance.
Sure. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Dear all, I'm one of the admins of ftp.halifax.rwth-aachen.de. We've been mirroring opensuse content for quite a while, including opensuse-buildservice. As a follow up to our discussion started around 8 years ago, I'd like to remove opensuse-buildservice from our mirror. In the past years, the server rarely uploaded more data to opensuse users (devs?) than what we needed to download using rsync (the difference is around -13.6 MBit/sec on average). The previous attempts at solving this issue by switching modules and getting data via rsync pushes sadly did not help. In case you have an idea of how to properly make use of our resources (we're not an archive of stale data!), please let me know. Best regards Carsten -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/
Carsten Otto wrote:
Dear all,
I'm one of the admins of ftp.halifax.rwth-aachen.de. We've been mirroring opensuse content for quite a while, including opensuse-buildservice.
As a follow up to our discussion started around 8 years ago, I'd like to remove opensuse-buildservice from our mirror. In the past years, the server rarely uploaded more data to opensuse users (devs?) than what we needed to download using rsync (the difference is around -13.6 MBit/sec on average). The previous attempts at solving this issue by switching modules and getting data via rsync pushes sadly did not help.
Hallo Carsten
In case you have an idea of how to properly make use of our resources (we're not an archive of stale data!), please let me know.
I would suggest you add suitable "--exclude"s to your rsync command. Unless I have completely misunderstood, I think that will solve the issue. Best regards Per -- Per Jessen, Zürich (17.1°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
Hi Per, we already had that discussion in 2011, without any solution. I still don't think it is the mirror admins' responsibility to dig through log files, make up some statistics, or anything similar to find what to mirror and what not. If you want us to provide a mirror, provide a suitable rsync repository. Right now we're serving around 60 projects, and there simply is no time to do this kind of additional work. As a consequence I'll remove OBS from ftp.halifax.rwth-aachen.de now. Best regards Carsten -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/
Carsten Otto wrote:
Hi Per,
we already had that discussion in 2011, without any solution. I still don't think it is the mirror admins' responsibility to dig through log files, make up some statistics, or anything similar to find what to mirror and what not. If you want us to provide a mirror, provide a suitable rsync repository.
Right now we're serving around 60 projects, and there simply is no time to do this kind of additional work.
As a consequence I'll remove OBS from ftp.halifax.rwth-aachen.de now.
Alright, thanks for letting us know. -- Per Jessen, Zürich (19.5°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org To contact the owner, email: mirror+owner@opensuse.org
participants (7)
-
Benjamin Hamilton -
Carlos Carvalho -
Carsten Otto -
Carsten Otto -
Marcus Rückert -
Per Jessen -
Stephan Kulow