[suse-mirror] Mirroring buildservice
Dear all, we (ftp.halifax.rwth-aachen.de) are mirroring quite a bit from the openSUSE project, including the build service (OBS). Due to a fast connection and good hardware, we're set up with a large weight in the mirrorbrain system distributing the download requests to the mirrors. This normally means we are serving a high percentage of all traffic for the files that are requested and also reside on our mirror. Based on the traffic we saw in the last month (and before that) I have doubts that mirroring OBS is a real advantage to the community. I understand that it is nice to have redundancy (e.g. if the servers at openSUSE are down) and to have distributed servers to increase download speed due to latency and routing issues. The following graphs show the traffic we see for OBS at our mirror. The yellow part (rsyncd) shows the (incoming) traffic due to openSUSE syncing new content to our mirror. The blue part (HTTP/Nginx) reflects the downloads from actual users (developers?). http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23month.p... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23week.pn... http://ftp.halifax.rwth-aachen.de/~cotto/rrd/opensuse-buildservice%23day.png As you can see, there is a fair amount of traffic served to the users, but there is also a large amount of traffic needed to synchronize the data. In total, when subtracting the incoming traffic from the outgoing traffic, this boils down to an average of about 2.3 MBit/sec in the last month. This value is positive, meaning our mirror serves more than what is needed to keep it synchronized - but it is very little traffic. For comparison, the openSUSE part of our mirror (excluding OBS) has a monthly average of 56 MBit/sec and Mozilla leads with 968 MBit/sec on average in the last month. I assume there are more mirrors facing similar traffic (maybe not knowing about it, since you need to analyze the log files). I also guess that the openSUSE server/mirror infrastructure is good enough to handle the requests to OBS content without the help from external mirrors (based on the fact that they are good enough to serve all the mirrors with rsync-pushes). Please think about the following aspects: a) some traffic is wasted and maybe paid for (we do not have to pay, though) b) disk space is used c) administrative work needs to be done I could think of several ways to face the current situation: d) leave it as it is e) remove some mirrors with low weight (in areas with good coverage) f) remove all mirrors Option e) would give the remaining mirrors more meaning, although even tripling the traffic would not really be a great step in my opinion. Option f) would not really change the outgoing traffic served by the openSUSE mirrors, although the traffic is shifted from rsync-pushes to HTTP downloads. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Am Saturday 02 October 2010 schrieb Carsten Otto:
As you can see, there is a fair amount of traffic served to the users, but there is also a large amount of traffic needed to synchronize the data. In total, when subtracting the incoming traffic from the outgoing traffic, this boils down to an average of about 2.3 MBit/sec in the last month. This value is positive, meaning our mirror serves more than what is needed to keep it synchronized - but it is very little traffic. For comparison, the openSUSE part of our mirror (excluding OBS) has a monthly average of 56 MBit/sec and Mozilla leads with 968 MBit/sec on average in the last month.
I assume there are more mirrors facing similar traffic (maybe not knowing about it, since you need to analyze the log files). I also guess that the openSUSE server/mirror infrastructure is good enough to handle the requests to OBS content without the help from external mirrors (based on the fact that they are good enough to serve all the mirrors with rsync-pushes).
Please think about the following aspects: a) some traffic is wasted and maybe paid for (we do not have to pay, though) b) disk space is used c) administrative work needs to be done
I could think of several ways to face the current situation: d) leave it as it is e) remove some mirrors with low weight (in areas with good coverage) f) remove all mirrors
Option e) would give the remaining mirrors more meaning, although even tripling the traffic would not really be a great step in my opinion. Option f) would not really change the outgoing traffic served by the openSUSE mirrors, although the traffic is shifted from rsync-pushes to HTTP downloads.
Hi, Thanks for bringing this up. Unfortunately I can't check how much traffic of the overall traffic you got as download.o.o only serves metalinks and the client then picks the mirrors. But I'm not even able to find how often you sync, because e.g. files in the KDE:Distro:Factory repo published on 29.9 are only on 2 german mirrors - files published on 12.9 are on 5. So I would think that mirror admins syncing so seldomly aren't doing themselves a favor in syncing opensuse repositories, but I'm not sure. We have no way of telling how many mirrors are available when people download repositories, even though it would be surely a good number to know. I don't even know how many mirrors we need in the world ;( But I find it very useful that you bring it up, so every mirror admin can question himself if he really wants to mirror repositories - as many have way less points than halifax has, so will see even less traffic. I'm trying to keep the most busy repositories in the 160Gb rsync module, the rest should be fine on a couple of mirrors around the world. Of course I don't want to stop anyone from mirroring, so I will not block mirrors - if people stop mirroring, it's their own choice. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Dear Stephan, On Mon, Oct 04, 2010 at 10:05:03AM +0200, Stephan Kulow wrote:
But I'm not even able to find how often you sync, because e.g. files in the KDE:Distro:Factory repo published on 29.9 are only on 2 german mirrors - files published on 12.9 are on 5. So I would think that mirror admins syncing so seldomly aren't doing themselves a favor in syncing opensuse repositories, but I'm not sure.
We get our data pushed from OpenSUSE directly. If there are files missing, please push better :)
We have no way of telling how many mirrors are available when people download repositories, even though it would be surely a good number to know. I don't even know how many mirrors we need in the world ;(
Use ?mirrorlist: http://download.opensuse.org/repositories/KDE:/Distro:/Factory/openSUSE_Fact... Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
participants (2)
-
Carsten Otto
-
Stephan Kulow