October 2010 - openSUSE Mirrors - openSUSE Mailing Lists

Re: [suse-mirror] Mirroring buildservice
by Stephan Kulow 07 Oct '18

07 Oct '18

Am Monday 04 October 2010 schrieb Carsten Otto: > Hi, > > On Mon, Oct 04, 2010 at 10:44:40AM +0200, Stephan Kulow wrote: > > > We get our data pushed from OpenSUSE directly. If there are files > > > missing, please push better :) > > > > I see that *Factory* is excluded for your mirror, that explains a good > > share of what I see as it matches the project name - one of the most > > busy ones actually. > > Oh, I thought we were mirroring everything OpenSUSE has to offer. What > do we need to do to actually get that? Could you just add Factory to the > pushes (and do a full sync)? We will extend disk space on demand, so it > may take a while until we are complete (as an alternative could you tell > us how much space we need?). > I don't think it makes sense to mirror everything. Many files are see so little traffic that mirroring them costs way too much. And now I'm really puzzled what your intention is - you strarted the thread with "remove all mirrors" and now you want even more stuff noone downloads? Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org For additional commands, e-mail: mirror+help(a)opensuse.org

7 20

Re: [suse-mirror] Mirroring buildservice
by Stephan Kulow 09 May '11

09 May '11

Am Monday 04 October 2010 schrieb Carsten Otto: > Dear Stephan, > > On Mon, Oct 04, 2010 at 10:05:03AM +0200, Stephan Kulow wrote: > > But I'm not even able to find how often you sync, because e.g. files in > > the KDE:Distro:Factory repo published on 29.9 are only on 2 german > > mirrors - files published on 12.9 are on 5. So I would think that mirror > > admins syncing so seldomly aren't doing themselves a favor in syncing > > opensuse repositories, but I'm not sure. > > We get our data pushed from OpenSUSE directly. If there are files > missing, please push better :) I see that *Factory* is excluded for your mirror, that explains a good share of what I see as it matches the project name - one of the most busy ones actually. > > > We have no way of telling how many mirrors are available when people > > download repositories, even though it would be surely a good number to > > know. I don't even know how many mirrors we need in the world ;( > > Use ?mirrorlist: > http://download.opensuse.org/repositories/KDE:/Distro:/Factory/openSUSE_Fac > tory/KDE4-BASIS.ymp?mirrorlist > That's the _current_ status, not the one the user saw. If you could find out a pattern of files that is causing a lot of rsync traffic, but few http traffic, we can easily add this to the exclude pattern. I can imagine e.g. home:/ having a bad ratio. If you have no time for that, perhaps you dig out a the http traffic logs for a given day (weekday preferred), so I can match it against the repos pushed to you. I can also offer that we offer a push mode only containing popular software gathered from our redirect logs. That should increase your http/rsync ratio too :) Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org For additional commands, e-mail: mirror+help(a)opensuse.org

2 3

[suse-mirror] Accessing stage.opensuse.org
by LKD FTP 26 Oct '10

26 Oct '10

Hi, We are getting the following errors while mirroring from stage.opensuse.org. ======================================================================= rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.5] ======================================================================= Is this a problem on our end, or is there new acess rules in effect? Did the sync source server change ? Thanks, - Gokdeniz Karadag ftp.linux.org.tr -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org For additional commands, e-mail: mirror+help(a)opensuse.org

2 2

[suse-mirror] Mirroring buildservice
by Carsten Otto 04 Oct '10

04 Oct '10

Dear all, we (ftp.halifax.rwth-aachen.de) are mirroring quite a bit from the openSUSE project, including the build service (OBS). Due to a fast connection and good hardware, we're set up with a large weight in the mirrorbrain system distributing the download requests to the mirrors. This normally means we are serving a high percentage of all traffic for the files that are requested and also reside on our mirror. Based on the traffic we saw in the last month (and before that) I have doubts that mirroring OBS is a real advantage to the community. I understand that it is nice to have redundancy (e.g. if the servers at openSUSE are down) and to have distributed servers to increase download speed due to latency and routing issues. The following graphs show the traffic we see for OBS at our mirror. The yellow part (rsyncd) shows the (incoming) traffic due to openSUSE syncing new content to our mirror. The blue part (HTTP/Nginx) reflects the downloads from actual users (developers?). http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23month.… http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.p… http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23day.png As you can see, there is a fair amount of traffic served to the users, but there is also a large amount of traffic needed to synchronize the data. In total, when subtracting the incoming traffic from the outgoing traffic, this boils down to an average of about 2.3 MBit/sec in the last month. This value is positive, meaning our mirror serves more than what is needed to keep it synchronized - but it is very little traffic. For comparison, the openSUSE part of our mirror (excluding OBS) has a monthly average of 56 MBit/sec and Mozilla leads with 968 MBit/sec on average in the last month. I assume there are more mirrors facing similar traffic (maybe not knowing about it, since you need to analyze the log files). I also guess that the openSUSE server/mirror infrastructure is good enough to handle the requests to OBS content without the help from external mirrors (based on the fact that they are good enough to serve all the mirrors with rsync-pushes). Please think about the following aspects: a) some traffic is wasted and maybe paid for (we do not have to pay, though) b) disk space is used c) administrative work needs to be done I could think of several ways to face the current situation: d) leave it as it is e) remove some mirrors with low weight (in areas with good coverage) f) remove all mirrors Option e) would give the remaining mirrors more meaning, although even tripling the traffic would not really be a great step in my opinion. Option f) would not really change the outgoing traffic served by the openSUSE mirrors, although the traffic is shifted from rsync-pushes to HTTP downloads. Best regards, -- Carsten Otto otto(a)informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211

2 2

[suse-mirror] Milestone2
by Stephan Kulow 01 Oct '10

01 Oct '10

Hi, I'm uploading 11.4-Milestone2 right now, I hope to have it on enough mirrors this afternoon to release. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe(a)opensuse.org For additional commands, e-mail: mirror+help(a)opensuse.org

2 2