[suse-mirror] Blogging about 11.3 launch - THANKS!
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea If you have anything to share regarding bandwidth/download numbers, please tell me. thanks a lot to all of you, this went really smooth! Andreas -- Andreas Jaeger, Program Manager openSUSE, aj@{novell.com,opensuse.org} Twitter: jaegerandi | Identica: jaegerandi SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 17:18:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
I hope you're not using Akamai for Brazil and South America... We (opensuse.c3sl.ufpr.br) could have easily transmitted several times more. Also, mirrorbrain only sends us isos. There are two possibilities for this. The first is that opensuse almost doesn't have package updates, differently from other distros. The second is that it's sending the packages directly to the clients, since they're much smaller. In this case it's quite bad since clients will have to pull from Europe, which is probably a lot slower than pulling from us. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Friday 16 July 2010 17:52:27 Carlos Carvalho wrote:
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 17:18:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
I hope you're not using Akamai for Brazil and South America... We (opensuse.c3sl.ufpr.br) could have easily transmitted several times more.
Also, mirrorbrain only sends us isos. There are two possibilities for this. The first is that opensuse almost doesn't have package updates, differently from other distros. The second is that it's sending the packages directly to the clients, since they're much smaller. In this case it's quite bad since clients will have to pull from Europe, which is probably a lot slower than pulling from us.
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.: http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please? Thanks, Andreas -- Andreas Jaeger, Program Manager openSUSE, aj@{novell.com,opensuse.org} Twitter: jaegerandi | Identica: jaegerandi SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
On Friday 16 July 2010 17:52:27 Carlos Carvalho wrote:
Also, mirrorbrain only sends us isos. There are two possibilities for this. The first is that opensuse almost doesn't have package updates, differently from other distros. The second is that it's sending the packages directly to the clients, since they're much smaller. In this case it's quite bad since clients will have to pull from Europe, which is probably a lot slower than pulling from us.
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
From here we appear at the top of the list, as it should be. This shows that mirrorbrain knows we have the file. If we don't show up in your list at all it's not a database problem, it's in the selection algorithm... -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
Forgot to say that our opensuse traffic dropped by approximately half since May. And the proportion of traffic that goes abroad didn't change. It could be that there are only half downloaders now but the change is too abrupt for this to be the reason. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 16 Jul 2010, Carlos Carvalho wrote:
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
Forgot to say that our opensuse traffic dropped by approximately half since May. And the proportion of traffic that goes abroad didn't change. It could be that there are only half downloaders now but the change is too abrupt for this to be the reason.
My guess is mirrorbrain is working better then ever. At GWDG, I also have seen less traffic than before at release moments, but ftp5.gwdg.de had the first 6 hours 2 gbit/s full. Then it went down to 1 gbit/s, and it is still there. That was not by mirrorbrain redirection - ftp3 and ftp4 had a higher mirrorbrain score than ftp5, but both needed less than half of their capacity. So it seems to me ftp5.gwdg.de has his own fan community, and mirrorbrain for the first time had more than enough mirror servers configured. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 ------------------------------------------------------------------------- -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Eberhard Moenkeberg (emoenke@gwdg.de) wrote on 16 July 2010 23:48:
On Fri, 16 Jul 2010, Carlos Carvalho wrote:
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
Forgot to say that our opensuse traffic dropped by approximately half since May. And the proportion of traffic that goes abroad didn't change. It could be that there are only half downloaders now but the change is too abrupt for this to be the reason.
My guess is mirrorbrain is working better then ever.
At GWDG, I also have seen less traffic than before at release moments, but ftp5.gwdg.de had the first 6 hours 2 gbit/s full. Then it went down to 1 gbit/s, and it is still there.
I've given 3 objective reasons for asking: - Andreas says that we don't appear in the mirrorlist but we should; - traffic has dropped here not only during releases but all the time. As I said, a 40% reduction since May; - it should not be because there are more mirrors around, because mirrorbrain doesn't show any from our side. There is a new one in Argentina but not in Brazil, and our stats show that the proportion of traffic going abroad didn't change. This means that Brazilian traffic dropped in the same proportion without any new mirror. It could be that there are less downloads of opensuse now from Brazil. Novell people can compare the mirrorbrain stats of the last 3 months to see if it's true. I find it unlikely. The other possibility is that downloads from here are being sent to other mirrors, which they shouldn't. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 16 Jul 2010, Carlos Carvalho wrote:
Eberhard Moenkeberg (emoenke@gwdg.de) wrote on 16 July 2010 23:48:
On Fri, 16 Jul 2010, Carlos Carvalho wrote:
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
Forgot to say that our opensuse traffic dropped by approximately half since May. And the proportion of traffic that goes abroad didn't change. It could be that there are only half downloaders now but the change is too abrupt for this to be the reason.
My guess is mirrorbrain is working better then ever.
At GWDG, I also have seen less traffic than before at release moments, but ftp5.gwdg.de had the first 6 hours 2 gbit/s full. Then it went down to 1 gbit/s, and it is still there.
I've given 3 objective reasons for asking: - Andreas says that we don't appear in the mirrorlist but we should;
Needs inspection, of course.
- traffic has dropped here not only during releases but all the time. As I said, a 40% reduction since May; - it should not be because there are more mirrors around, because mirrorbrain doesn't show any from our side. There is a new one in Argentina but not in Brazil, and our stats show that the proportion of traffic going abroad didn't change. This means that Brazilian traffic dropped in the same proportion without any new mirror.
It could be that there are less downloads of opensuse now from Brazil. Novell people can compare the mirrorbrain stats of the last 3 months to see if it's true. I find it unlikely. The other possibility is that downloads from here are being sent to other mirrors, which they shouldn't.
An "official" answer with statistics numbers could help. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 ------------------------------------------------------------------------- -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi Carlos, On Fri, Jul 16, 2010 at 07:34:28 -0300, Carlos Carvalho wrote:
Eberhard Moenkeberg (emoenke@gwdg.de) wrote on 16 July 2010 23:48:
On Fri, 16 Jul 2010, Carlos Carvalho wrote:
Andreas Jaeger (aj@novell.com) wrote on 16 July 2010 21:59:
You can just append "?mirrorlist" to a download link to see what mirrorbrain would do, e.g.:
http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon- Lang-i586.iso?mirrorlist
Looking at that one, I do not see your mirror at all right now - this is something we should investigate. Darix, could you do so, please?
Forgot to say that our opensuse traffic dropped by approximately half since May. And the proportion of traffic that goes abroad didn't change. It could be that there are only half downloaders now but the change is too abrupt for this to be the reason.
My guess is mirrorbrain is working better then ever.
At GWDG, I also have seen less traffic than before at release moments, but ftp5.gwdg.de had the first 6 hours 2 gbit/s full. Then it went down to 1 gbit/s, and it is still there.
I've given 3 objective reasons for asking: - Andreas says that we don't appear in the mirrorlist but we should;
Compare http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon-Lang-... http://download.opensuse.org/distribution/11.3/iso/openSUSE-11.3-Addon-Lang-... Andreas does not see you in the mirrorlist because he looks from Germany. You see your mirror because you look from Brazil. That's because the mirror is configured to be eligible for clients from the South American continent, not from other continents. There's a number of countries that have sparse international connectivity, except to Germany and the US maybe. The assumption is that they are better served by sending them there then to Brazil. Even if your mirror generally has good international connectivity. This doesn't necessarily affect arbitrary client countries, because many are configured in a way which makes sense for them. It affects clients that cannot be geolocated, which are unknown to GeoIP or connected via Satellite link. Without any geolocation, it is hard to redirect to a useful mirror [*]. [*] Maybe such clients should always be sent to the US or Germany, instead of picking a mirror from anywhere (random from the world map). An improvement in MirrorBrain would not be hard and a contribution would be most welcome. Anyhow, this is all about an edge case, and nothing that steals you any significant traffic. Just the odd client that might find its way to your mirror from far away. Either way, you get no requests from US or German clients, just as US mirrors don't get requests from German clients, and so on. As long as there is at least one German mirror, that is ;-)
- traffic has dropped here not only during releases but all the time. As I said, a 40% reduction since May;
No idea what could cause that. Peter
Hi, On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
Thanks for referencing Fujitsu as our sponsor, I hope they like it! Some minor remarks or wishes: - a few examples -> two examples - sat -> set - with 4 gbit/s -> with peaks of 4 gbit/s - gbit/s -> GBit/sec or GBit/s - by Fujitsu. -> by Fujitsu and will be upgraded from 2 GBit/sec to 10 GBit/sec in the next few days. [if you wait long enough I can tell you if the upgrade already happened] Btw, our mirror was quite bored by this release with barely any disk or CPU activity. I hope this changes with the 10 GBit/sec line. Do you know when the next release is scheduled? :) Bye, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
Hi, On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
Thanks for referencing Fujitsu as our sponsor, I hope they like it!
Some minor remarks or wishes:
- a few examples -> two examples
I hope GWDG will join too. It was really great.
- sat -> set - with 4 gbit/s -> with peaks of 4 gbit/s - gbit/s -> GBit/sec or GBit/s
gbit/s is OK.
- by Fujitsu. -> by Fujitsu and will be upgraded from 2 GBit/sec to 10 GBit/sec in the next few days. [if you wait long enough I can tell you if the upgrade already happened]
Btw, our mirror was quite bored by this release with barely any disk or CPU activity. I hope this changes with the 10 GBit/sec line.
How should it - if you have enough RAM to cache the whole tree, and I guess you have 32 GB or more. My first astonished experience this way was at the release of staroffice-3 for linux about 1995. The whole tree was 32 MB size, ftp.gwdg.de had 64 MB RAM, and as I looked at the server noises and leds, there was nothing... Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 ------------------------------------------------------------------------- -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Fri, Jul 16, 2010 at 11:14:36PM +0200, Eberhard Moenkeberg wrote:
Hi,
On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
My case is that it seemed like all traffic was directed to the novell site, with a prio of 500, outweighting our 1 gbit/s site prio of 100. there were some downloads but not that much. The irony was that our site at ftp.klid.dk was about 10 times as fast as the novell site. maybe the novell site also took over quite some trafffic in other countries. Best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sat, Jul 17, 2010 at 09:51:54AM +0200, Keld Simonsen wrote:
On Fri, Jul 16, 2010 at 11:14:36PM +0200, Eberhard Moenkeberg wrote:
Hi,
On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
My case is that it seemed like all traffic was directed to the novell site, with a prio of 500, outweighting our 1 gbit/s site prio of 100. there were some downloads but not that much. The irony was that our site at ftp.klid.dk was about 10 times as fast as the novell site.
maybe the novell site also took over quite some trafffic in other countries.
cdn.novell.com is actually Akamai. Ciao, Marcus -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sat, Jul 17, 2010 at 10:03:06AM +0200, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 09:51:54AM +0200, Keld Simonsen wrote:
On Fri, Jul 16, 2010 at 11:14:36PM +0200, Eberhard Moenkeberg wrote:
Hi,
On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
My case is that it seemed like all traffic was directed to the novell site, with a prio of 500, outweighting our 1 gbit/s site prio of 100. there were some downloads but not that much. The irony was that our site at ftp.klid.dk was about 10 times as fast as the novell site.
maybe the novell site also took over quite some traffic in other countries.
cdn.novell.com is actually Akamai.
yes, that is understood. So they should have bandwidth. But it seems like Akamai does not perform nearly as good as some of the volunteer mirrors. Anyway what will a prio of 500 meqan compared to a prio of 100+ Would it mean that in a round-robin queue that the 500 prio site would be chosen 5 times as frequently as a 100 prio site? (500/100 = 5). best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Peter On Sat, Jul 17, 2010 at 12:50:12 +0200, Keld Simonsen wrote:
On Sat, Jul 17, 2010 at 10:03:06AM +0200, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 09:51:54AM +0200, Keld Simonsen wrote:
On Fri, Jul 16, 2010 at 11:14:36PM +0200, Eberhard Moenkeberg wrote:
Hi,
On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
My case is that it seemed like all traffic was directed to the novell site, with a prio of 500, outweighting our 1 gbit/s site prio of 100. there were some downloads but not that much. The irony was that our site at ftp.klid.dk was about 10 times as fast as the novell site.
maybe the novell site also took over quite some traffic in other countries.
cdn.novell.com is actually Akamai.
yes, that is understood. So they should have bandwidth. But it seems like Akamai does not perform nearly as good as some of the volunteer mirrors.
Indeed, that's what I observed at previous openSUSE release. Akamai seems to invest precisely only as much bandwidth as is needed to keep people marginally happy (and their service certainly works), but they won't make people as happy as a network of fast mirrors do. So during the first day, many users have to download from "decently performing" Akamai network, while they certainly could get stuff faster from a mirror. But then again, it's probably still fast enough for most people that they don't bother too much.
Anyway what will a prio of 500 meqan compared to a prio of 100+ Would it mean that in a round-robin queue that the 500 prio site would be chosen 5 times as frequently as a 100 prio site? (500/100 = 5).
It's not a linear relationship, and it depends on the rest of the mirrors that are selected for a client. 500/100 would roughly be the result in your case, if you are the only Danish mirror and there is one other (with prio 500). The higher the prio, the more requests get assigned to a mirror. Peter
On Sat, Jul 17, 2010 at 04:16:22PM +0200, Peter Poeml wrote:
Peter
On Sat, Jul 17, 2010 at 12:50:12 +0200, Keld Simonsen wrote:
On Sat, Jul 17, 2010 at 10:03:06AM +0200, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 09:51:54AM +0200, Keld Simonsen wrote:
On Fri, Jul 16, 2010 at 11:14:36PM +0200, Eberhard Moenkeberg wrote:
Hi,
On Fri, 16 Jul 2010, Carsten Otto wrote:
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
>I'm blogging about the launch and will publish my article later today. The >current version is here: >http://pastie.org/private/orcmz76zjcybhy0tdki6ea > >If you have anything to share regarding bandwidth/download numbers, please >tell me.
My case is that it seemed like all traffic was directed to the novell site, with a prio of 500, outweighting our 1 gbit/s site prio of 100. there were some downloads but not that much. The irony was that our site at ftp.klid.dk was about 10 times as fast as the novell site.
maybe the novell site also took over quite some traffic in other countries.
cdn.novell.com is actually Akamai.
yes, that is understood. So they should have bandwidth. But it seems like Akamai does not perform nearly as good as some of the volunteer mirrors.
Indeed, that's what I observed at previous openSUSE release. Akamai seems to invest precisely only as much bandwidth as is needed to keep people marginally happy (and their service certainly works), but they won't make people as happy as a network of fast mirrors do.
So during the first day, many users have to download from "decently performing" Akamai network, while they certainly could get stuff faster from a mirror. But then again, it's probably still fast enough for most people that they don't bother too much.
Anyway what will a prio of 500 meqan compared to a prio of 100+ Would it mean that in a round-robin queue that the 500 prio site would be chosen 5 times as frequently as a 100 prio site? (500/100 = 5).
It's not a linear relationship, and it depends on the rest of the mirrors that are selected for a client. 500/100 would roughly be the result in your case, if you are the only Danish mirror and there is one other (with prio 500). The higher the prio, the more requests get assigned to a mirror.
If not linear, how is the relationship then? Anyway it looks like the novell site has been taken off the list for denmark. And I read a little more about the cdn.novell.com site. I understand now that this is not a Suse site, but some other department in Novell. For a moment I thought that Novell was putting money into a Akamai site which was in fact just competing with their voluntary openSUSE mirrors, creating some worries about what was causing a drop down in the mirror traffic by seanoed mirror maintainers. But if it is another Novell project - well then all resources are well received, and of cause bandwidth always costs - be it donated by a university or a private firm. best regards Keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sat, Jul 17, 2010 at 05:07:50PM +0200, Keld Simonsen wrote:
If not linear, how is the relationship then?
Eligible mirrors are looked up in the database with their prio, and then each mirror is ranked according to this little formula: rank = (rand()>>16) * ((RAND_MAX>>16) / prio); The mirror with the lowest rank is the one that's chosen. Thus, the choice is made by weighted randomization. The probabilities of selection should be: 3 mirrors with prios 100, 100, 50: 41.7%, 41.7%, 16.7% 3 mirrors with prios 200, 100, 10: 73.5%, 24.4%, 2.1% 2 mirrors with prios 100, 50: 75.0%, 25.0% 2 mirrors with prios 500, 100: 90.0%, 10.0%
Anyway it looks like the novell site has been taken off the list for denmark.
It's not a Novell site as such - it's just a CNAME entry in the Novell DNS which points to Akamai's content server(s) (a CNAME itself, pointing to variable edge servers depending on your location). This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.) After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
And I read a little more about the cdn.novell.com site. I understand now that this is not a Suse site, but some other department in Novell. For a moment I thought that Novell was putting money into a Akamai site which was in fact just competing with their voluntary openSUSE mirrors, creating some worries about what was causing a drop down in the mirror traffic by seanoed mirror maintainers. But if it is another Novell project - well then all resources are well received, and of cause bandwidth always costs - be it donated by a university or a private firm.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector. Peter
Hi Peter Thx for the explanations! Keld On Sat, Jul 17, 2010 at 08:55:45PM +0200, Peter Poeml wrote:
On Sat, Jul 17, 2010 at 05:07:50PM +0200, Keld Simonsen wrote:
If not linear, how is the relationship then?
Eligible mirrors are looked up in the database with their prio, and then each mirror is ranked according to this little formula:
rank = (rand()>>16) * ((RAND_MAX>>16) / prio);
The mirror with the lowest rank is the one that's chosen. Thus, the choice is made by weighted randomization.
The probabilities of selection should be: 3 mirrors with prios 100, 100, 50: 41.7%, 41.7%, 16.7% 3 mirrors with prios 200, 100, 10: 73.5%, 24.4%, 2.1% 2 mirrors with prios 100, 50: 75.0%, 25.0% 2 mirrors with prios 500, 100: 90.0%, 10.0%
Anyway it looks like the novell site has been taken off the list for denmark.
It's not a Novell site as such - it's just a CNAME entry in the Novell DNS which points to Akamai's content server(s) (a CNAME itself, pointing to variable edge servers depending on your location).
This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.)
After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
And I read a little more about the cdn.novell.com site. I understand now that this is not a Suse site, but some other department in Novell. For a moment I thought that Novell was putting money into a Akamai site which was in fact just competing with their voluntary openSUSE mirrors, creating some worries about what was causing a drop down in the mirror traffic by seanoed mirror maintainers. But if it is another Novell project - well then all resources are well received, and of cause bandwidth always costs - be it donated by a university or a private firm.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector.
Peter
-- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sat, Jul 17, 2010 at 08:55:45PM +0200, Peter Poeml wrote:
pointing to variable edge servers depending on your location).
This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.)
After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
AFAIK 48 hours actually, so 2 days.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector.
Yes and we really melted the mirrors at that time. Probably coolo has the Akamai stats when he is back in the office on Monday on how much they served out. Ciao, Marcus -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Sun, 18 Jul 2010, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 08:55:45PM +0200, Peter Poeml wrote:
pointing to variable edge servers depending on your location).
This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.)
After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
AFAIK 48 hours actually, so 2 days.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector.
Yes and we really melted the mirrors at that time.
Probably coolo has the Akamai stats when he is back in the office on Monday on how much they served out.
Please evaluate that with the fact in mind that all mirrors have seen lower throughput than before - with exception of ftp5.gwdg.de, but that was an effect outside of mirrorbrain. My guess is mirrorbrain is so sensitive and complete now that Akamai will be no win next time, as it already was no win this time regarding individual throughput. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 ------------------------------------------------------------------------- -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sun, Jul 18, 2010 at 12:32:28AM +0200, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 08:55:45PM +0200, Peter Poeml wrote:
pointing to variable edge servers depending on your location).
This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.)
After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
AFAIK 48 hours actually, so 2 days.
36 hours only, minus ample time (~4-6hours) to finish running downloads before the file are removed from Akamai.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector.
Yes and we really melted the mirrors at that time.
"melted the mirrors" simply because only 8 or 10 mirrors were used at the time, as far as I remember. Worldwide. That's what the infrastructure could handle. Since then, MirrorBrain wasn't the only change. The other change was that I persistently collected mirrors, one by one, day after day, actively searching them, keeping them running. Now there are ~150 mirrors.
Probably coolo has the Akamai stats when he is back in the office on Monday on how much they served out.
We can safely assume a maximum throughput of 20GB/s delivered via Akamai. For a popular openSUSE release. (They could handle a hundred times more, if need be.) This is to be regarded as a peak value, occurring for some hours at most. (GB, not GBit) Now everybody can easily calculate: How many 1GBit/s mirrors does it need to provide 20GB/s total throughput? :-) ~150. However, only fast mirrors will have 1GBit (or more). The majority is still 100 MBit I'd say. There aren't 150 1GBit mirrors. So, offloading some requests to Akamai may indeed have its use during the peak demand on the release day. Only then, though. Outside this presumed peak, we have seen often enough that the mirror network is clearly able to handle the demand. During the peak, we don't know because it hasn't been tried out. I'd say, it should be tried! It is definitely not necessary to send _all_ traffic to Akamai. Of course that makes it easy to get download counts for that first day. However, those counts could also be accurately obtained directly on download.opensuse.org. I implemented that for OpenOffice.org's download server, and the elimination of skew by repetitive/partial requests seems to work well. (Maybe it works better then Akamai's ;) I don't know how they count exactly, but since their business is "big bucks", I guess they know very well how to count bytes instead of requests.) IMO, it would be interesting to have download counts at all times and not only for day 1. Peter
On Sun, Jul 18, 2010 at 02:14:56PM +0200, Peter Poeml wrote:
On Sun, Jul 18, 2010 at 12:32:28AM +0200, Marcus Meissner wrote:
On Sat, Jul 17, 2010 at 08:55:45PM +0200, Peter Poeml wrote:
pointing to variable edge servers depending on your location).
This cname cdn.novell.com is handled like any other mirror in MirrorBrain, with the exception that it is not configured with a certain country, but with a wildcard, which matches all countries. Thus it's considered for requests from all countries. (For the files that it serves, namely those few ISOs.)
After the first release day, this special mirror is simply switched off in MirrorBrain, and from that point everything is redirected to the mirrors as normal.
AFAIK 48 hours actually, so 2 days.
36 hours only, minus ample time (~4-6hours) to finish running downloads before the file are removed from Akamai.
To explain this: the bandwidth is surplus bandwidth that is available to Novell anyway (as a customer), and it donates this bandwidth to the openSUSE project. It's a nice gesture albeit it's of arguable use. I guess the motivation needs to be understood in a historic context; there was a time before openSUSE had a good download redirector.
Yes and we really melted the mirrors at that time.
"melted the mirrors" simply because only 8 or 10 mirrors were used at the time, as far as I remember. Worldwide. That's what the infrastructure could handle.
Since then, MirrorBrain wasn't the only change. The other change was that I persistently collected mirrors, one by one, day after day, actively searching them, keeping them running. Now there are ~150 mirrors.
Probably coolo has the Akamai stats when he is back in the office on Monday on how much they served out.
We can safely assume a maximum throughput of 20GB/s delivered via Akamai. For a popular openSUSE release. (They could handle a hundred times more, if need be.) This is to be regarded as a peak value, occurring for some hours at most.
(GB, not GBit)
Now everybody can easily calculate: How many 1GBit/s mirrors does it need to provide 20GB/s total throughput? :-)
~150.
However, only fast mirrors will have 1GBit (or more). The majority is still 100 MBit I'd say. There aren't 150 1GBit mirrors.
So, offloading some requests to Akamai may indeed have its use during the peak demand on the release day.
Only then, though. Outside this presumed peak, we have seen often enough that the mirror network is clearly able to handle the demand. During the peak, we don't know because it hasn't been tried out. I'd say, it should be tried!
It is definitely not necessary to send _all_ traffic to Akamai. Of course that makes it easy to get download counts for that first day. However, those counts could also be accurately obtained directly on download.opensuse.org. I implemented that for OpenOffice.org's download server, and the elimination of skew by repetitive/partial requests seems to work well. (Maybe it works better then Akamai's ;) I don't know how they count exactly, but since their business is "big bucks", I guess they know very well how to count bytes instead of requests.) IMO, it would be interesting to have download counts at all times and not only for day 1.
Peter
Above, I claimed "20GB/s" as maximum bandwidth served by Akamai in the past. I now have the suspicion that the number was wrong. I took the number from memory (the peak ranged from 13-20 in the past) but the scale I verified with a presentation from 2008. However, at http://news.opensuse.org/2010/07/16/opensuse-11-3-launch-information/ Andreas gives 13GBit/s (not GB) for the last release. He also says that 90TB were distributed by Akamai in 24 hours, which results in about 1GB/s or 10 GBit/s and makes the 13GBit/s peak more plausible. Thus, all I wrote above is put into new perspective and needs to corrected by a factor of about 10. So the contribution of delivery via Akamai's services is a tenth of what I reckoned, and it shouldn't pose much difficulty to the mirrors at all to yield the same. Peter
On Thu, Jul 22, 2010 at 12:57:03PM +0200, Peter Poeml wrote:
Above, I claimed "20GB/s" as maximum bandwidth served by Akamai in the past. I now have the suspicion that the number was wrong. I took the number from memory (the peak ranged from 13-20 in the past) but the scale I verified with a presentation from 2008. However, at http://news.opensuse.org/2010/07/16/opensuse-11-3-launch-information/ Andreas gives 13GBit/s (not GB) for the last release. He also says that 90TB were distributed by Akamai in 24 hours, which results in about 1GB/s or 10 GBit/s and makes the 13GBit/s peak more plausible.
Thus, all I wrote above is put into new perspective and needs to corrected by a factor of about 10. So the contribution of delivery via Akamai's services is a tenth of what I reckoned, and it shouldn't pose much difficulty to the mirrors at all to yield the same.
Still interesting info. I hope those in the know will return to the list with a more detailed analysis of the 11.3 release, and come forward with their proposals and decisions for the next release. The main purpose should be that the users would have a good experience with the next release, and then also that we treat the volunteers well - the mirror administrators should have a feeling that their work is appreciated. And in my mind we could have also a look to costs, I am not sure if the Akamai service costs something, I would expect it to, but it could be part of Novell's normal agreement with Akamai. Maybe such money could be better spent on something else, eg. some further development of mirroring software. I actually think that the user experience with 11.3 downloads in peek time was pretty good - if users get about 20 Mbit/s then they are generally happy, I think. Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP. Best regards Keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Thursday 22 July 2010 13:23:01 Keld Simonsen wrote:
On Thu, Jul 22, 2010 at 12:57:03PM +0200, Peter Poeml wrote:
Above, I claimed "20GB/s" as maximum bandwidth served by Akamai in the past. I now have the suspicion that the number was wrong. I took the number from memory (the peak ranged from 13-20 in the past) but the scale I verified with a presentation from 2008. However, at http://news.opensuse.org/2010/07/16/opensuse-11-3-launch-information/ Andreas gives 13GBit/s (not GB) for the last release. He also says that 90TB were distributed by Akamai in 24 hours, which results in about 1GB/s or 10 GBit/s and makes the 13GBit/s peak more plausible.
Thus, all I wrote above is put into new perspective and needs to corrected by a factor of about 10. So the contribution of delivery via Akamai's services is a tenth of what I reckoned, and it shouldn't pose much difficulty to the mirrors at all to yield the same.
Still interesting info. I hope those in the know will return to the list with a more detailed analysis of the 11.3 release, and come forward with their proposals and decisions for the next release.
Darix, will you drive this?
The main purpose should be that the users would have a good experience with the next release, and then also that we treat the volunteers well - the mirror administrators should have a feeling that their work is appreciated. And in my mind we could have also a look to costs,
Your work is definitely appreciated!
I am not sure if the Akamai service costs something, I would expect it to, but it could be part of Novell's normal agreement with Akamai. Maybe such money could be better spent on something else, eg. some further development of mirroring software.
The way we used Akamai for the launch, it is indeed covered by our contract.
I actually think that the user experience with 11.3 downloads in peek time was pretty good - if users get about 20 Mbit/s then they are generally happy, I think.
Yes, think so, too.
Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP.
Ah... Thanks, Andreas -- Andreas Jaeger, Program Manager openSUSE, aj@{novell.com,opensuse.org} Twitter: jaegerandi | Identica: jaegerandi SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
Hi all, On Thu, Jul 22, 2010 at 01:23:01PM +0200, Keld Simonsen wrote:
Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP.
Seeding via BitTorrent is something we at ftp.halifax.rwth-aachen.de are doing for a long time now. With every single big release also advertising BitTorrent - OpenSUSE and Ubuntu come to mind - we were able to hit the limits of our line (1 GBit/sec in the past, 1,7 GBit/sec in the meantime and 2 GBit/sec with OpenSUSE 11.3). With OpenSUSE 11.3 we also added a few additional machines, each serving a single ISO file. This resulted in about 3 GBit/sec just via BitTorrent at peaks. With enough RAM, which should not be a problem for proper mirrors, the CD/DVD files are cached and the random accesses caused by BitTorrent do not harm at all. By referencing the very same file for HTTP, FTP and BitTorrent you get BitTorrent "for free" (disregarding CPU usage). However, if the mirror can only fit one or two DVDs into RAM, also handling normal HTTP traffic accessing other files could really hurt performance. Here, the disk subsystem is trashed with random read accesses. In fact, we needed to put the image into RAM (using "cat") before starting rtorrent, because it started seeding the uncached file right from the start and was unresponsive because of that. We made very good experiences with rtorrent. You may need to spawn several instances for different files if you hit CPU limits, since rtorrent is not multi threaded, yet. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
On Thu, Jul 22, 2010 at 01:38:50PM +0200, Carsten Otto wrote:
Hi all,
On Thu, Jul 22, 2010 at 01:23:01PM +0200, Keld Simonsen wrote:
Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP.
Seeding via BitTorrent is something we at ftp.halifax.rwth-aachen.de are doing for a long time now. With every single big release also advertising BitTorrent - OpenSUSE and Ubuntu come to mind - we were able to hit the limits of our line (1 GBit/sec in the past, 1,7 GBit/sec in the meantime and 2 GBit/sec with OpenSUSE 11.3). With OpenSUSE 11.3 we also added a few additional machines, each serving a single ISO file. This resulted in about 3 GBit/sec just via BitTorrent at peaks.
With enough RAM, which should not be a problem for proper mirrors, the CD/DVD files are cached and the random accesses caused by BitTorrent do not harm at all. By referencing the very same file for HTTP, FTP and BitTorrent you get BitTorrent "for free" (disregarding CPU usage).
However, if the mirror can only fit one or two DVDs into RAM, also handling normal HTTP traffic accessing other files could really hurt performance. Here, the disk subsystem is trashed with random read accesses. In fact, we needed to put the image into RAM (using "cat") before starting rtorrent, because it started seeding the uncached file right from the start and was unresponsive because of that.
We made very good experiences with rtorrent. You may need to spawn several instances for different files if you hit CPU limits, since rtorrent is not multi threaded, yet.
Sounds interesting! So this is dependent on all isos in RAM, to avoid random disk access to clutter up the disks... Would it be possible to eg use bigger block sizes for BT so that the random access would not be so performance impacting? I was thinking about maybe block sizes of 1 MB, without investigating what BT currently does, and whether it is easy to change block sizes wth BT. My server does only have 1.5 TB of RAM, and I think what you suggest would need much more RAM. Anyway, RAM is cheap these days. What would be the recommended amount of RAM? And would it scale? I was thinking about having most of the mirror data available via BT, in an automated and distributed way, and then having all data in RAM seems futile, I have about 5 TB data on my mirror server. I would think bigger sizes could be a better way to scale. Anyway - again without consulting the net, I think it could be useful with a wiki article on mirroring software and practices, eg on wikipedia, which could be a neutral place to gather the technology and experience of distribution and mirror maintainers on this subject. best regards Keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Thu, Jul 22, 2010 at 02:37:21PM +0200, Keld Simonsen wrote:
Sounds interesting! So this is dependent on all isos in RAM, to avoid random disk access to clutter up the disks...
Yes.
Would it be possible to eg use bigger block sizes for BT so that the random access would not be so performance impacting? I was thinking about maybe block sizes of 1 MB, without investigating what BT currently does, and whether it is easy to change block sizes wth BT.
As far as I know, this is not possible. However, we were thinking (when we had far less than 60 GByte of RAM) to limit the amount of data that we seed (and announce and get from disk/RAM). This amount can be configured so that it fits into RAM and should be changed based on demand (there's no sense to seed the first half and ignore the second half of the file). This would need some changes in the local BitTorrent client, so this is still only an idea.
My server does only have 1.5 TB of RAM,
Wow? :)
and I think what you suggest would need much more RAM. Anyway, RAM is cheap these days. What would be the recommended amount of RAM? And would it scale? I was thinking about having most of the mirror data available via BT, in an automated and distributed way, and then having all data in RAM seems futile, I have about 5 TB data on my mirror server. I would think bigger sizes could be a better way to scale.
I'd suggest at least 4 GByte for the system, 4 GByte for standard caches (depending on your traffic, amount of data, disk backend speed) and X GByte for as many X as you want to seed. If you seed files that are already served via HTTP, you can save on the intersection, of course. Having most of the mirror data availabe via BitTorrent is not such a good idea, I think. BitTorrent is especially useful for 'hot' files, e.g. OpenSUSE DVDs in the first few days after a release. This way you 'only' need about 10 GByte of RAM for these files.
Anyway - again without consulting the net, I think it could be useful with a wiki article on mirroring software and practices, eg on wikipedia, which could be a neutral place to gather the technology and experience of distribution and mirror maintainers on this subject.
I don't think Wikipedia is the right place for technical documentation, but I'd be glad to participate in whatever documentation project. Best regards, -- Carsten Otto carsten@c-otto.de www.c-otto.de
On Thu, Jul 22, 2010 at 02:47:28PM +0200, Carsten Otto wrote:
On Thu, Jul 22, 2010 at 02:37:21PM +0200, Keld Simonsen wrote:
Sounds interesting! So this is dependent on all isos in RAM, to avoid random disk access to clutter up the disks...
Yes.
Would it be possible to eg use bigger block sizes for BT so that the random access would not be so performance impacting? I was thinking about maybe block sizes of 1 MB, without investigating what BT currently does, and whether it is easy to change block sizes wth BT.
As far as I know, this is not possible. However, we were thinking (when we had far less than 60 GByte of RAM) to limit the amount of data that we seed (and announce and get from disk/RAM). This amount can be configured so that it fits into RAM and should be changed based on demand (there's no sense to seed the first half and ignore the second half of the file). This would need some changes in the local BitTorrent client, so this is still only an idea.
My server does only have 1.5 TB of RAM,
Wow? :)
I am sorry, I meant 1.5 GB RAM... It is a small machine, but I am quite amazed about the performance I get out of it.
and I think what you suggest would need much more RAM. Anyway, RAM is cheap these days. What would be the recommended amount of RAM? And would it scale? I was thinking about having most of the mirror data available via BT, in an automated and distributed way, and then having all data in RAM seems futile, I have about 5 TB data on my mirror server. I would think bigger sizes could be a better way to scale.
I'd suggest at least 4 GByte for the system, 4 GByte for standard caches (depending on your traffic, amount of data, disk backend speed) and X GByte for as many X as you want to seed. If you seed files that are already served via HTTP, you can save on the intersection, of course. Having most of the mirror data availabe via BitTorrent is not such a good idea, I think. BitTorrent is especially useful for 'hot' files, e.g. OpenSUSE DVDs in the first few days after a release. This way you 'only' need about 10 GByte of RAM for these files.
Hmm, this would mean a new server for us, and I am not sure we can find the money to finance it. Anyway our performance is pretty good, we have max download speeds of about 300 Mbit/s. And I wonder if it would be any good for us to do our own torrents. I tried it once, without any significant results/traffic. I think if we made our own torrents, then users need to know the specific torrents URLs to utilize them, and that would not be well advertised. We rely much on central redistribution services like mirrorbrain to get our traffic. And I think the torrents should be set up automatically, and not by hand, and torrents should probably go away after the peak time, if every torrent need to be in RAM.
Anyway - again without consulting the net, I think it could be useful with a wiki article on mirroring software and practices, eg on wikipedia, which could be a neutral place to gather the technology and experience of distribution and mirror maintainers on this subject.
I don't think Wikipedia is the right place for technical documentation, but I'd be glad to participate in whatever documentation project.
Wikipedia seems to be well maintained, and it is a neutral place. And I see a number of technical overview articles there, Anyway, would there be other places better suited for such content? best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Dear Keld and all, On Thu, Jul 22, 2010 at 03:43:23PM +0200, Keld Simonsen wrote:
I am sorry, I meant 1.5 GB RAM... It is a small machine, but I am quite amazed about the performance I get out of it.
I guessed so.
Anyway our performance is pretty good, we have max download speeds of about 300 Mbit/s.
With decent hardware you should be able to provide about 300 MBit/sec via BitTorrent, even if you only have 1,5 GByte of RAM. Try it out :)
Wikipedia seems to be well maintained, and it is a neutral place. And I see a number of technical overview articles there, Anyway, would there be other places better suited for such content?
I don't know where this should be done. Start a project! Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
On Thu, Jul 22, 2010 at 01:38:50PM +0200, Carsten Otto wrote:
Hi all,
On Thu, Jul 22, 2010 at 01:23:01PM +0200, Keld Simonsen wrote:
Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP.
Seeding via BitTorrent is something we at ftp.halifax.rwth-aachen.de are doing for a long time now. With every single big release also advertising BitTorrent - OpenSUSE and Ubuntu come to mind - we were able to hit the limits of our line (1 GBit/sec in the past, 1,7 GBit/sec in the meantime and 2 GBit/sec with OpenSUSE 11.3). With OpenSUSE 11.3 we also added a few additional machines, each serving a single ISO file. This resulted in about 3 GBit/sec just via BitTorrent at peaks.
With enough RAM, which should not be a problem for proper mirrors, the CD/DVD files are cached and the random accesses caused by BitTorrent do not harm at all. By referencing the very same file for HTTP, FTP and BitTorrent you get BitTorrent "for free" (disregarding CPU usage).
I am interested in learning more about this. I tried downloading http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11... It gave a download speed of 1.15 MB/S or 9 Mbit/s with wget. You have a 10 Gbit/s line and the overall bandwidth load at your site was about 1.5 Gbit/s. So there should be enough excessive bandwidth to play with. Do you have a limit on the line speed per connection? I then tried my own site: http://ftp.klid.dk/ftp/opensuse/distribution/11.3/iso/openSUSE-11.3-KDE4-Liv... This was with wget 29.8 MB/s or about 240 Mbit/s - my system has a 1 Gbit/s connection and the load was 100 Mbit/s. I then tried to download via bittorrent from your site http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11... It was something like 3.2 MB/s = 25 Mbit/s. Better than 9 Mbit/s So is this why your users use BT? They know it is faster than ftp? I did not dee any big visible advertisement of the torrents. So you have educated your users to use BT? I am just wondering how this is done from your side. best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Keld Simonsen (keld@keldix.com) wrote on 23 July 2010 18:24:
I tried downloading
http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It gave a download speed of 1.15 MB/S or 9 Mbit/s with wget.
You have a 10 Gbit/s line and the overall bandwidth load at your site was about 1.5 Gbit/s.
How did you get this number?
So there should be enough excessive bandwidth to play with. Do you have a limit on the line speed per connection?
I then tried my own site:
http://ftp.klid.dk/ftp/opensuse/distribution/11.3/iso/openSUSE-11.3-KDE4-Liv...
This was with wget 29.8 MB/s or about 240 Mbit/s - my system has a 1 Gbit/s connection and the load was 100 Mbit/s.
I then tried to download via bittorrent from your site http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It was something like 3.2 MB/s = 25 Mbit/s. Better than 9 Mbit/s
How do you know that you pulled only from halifax.rwth-aachen? When a site gets the iso it registers as a seeder with the tracker; it'll only be one more seeder among many. The torrent client will get a list of machines from the tracker and pull from many simultaneously, won't it? So if the bottleneck is not in your link you may be able to pull faster than from a single location. If the bottleneck is somewhere between you and the single source you may have lower flux even if both you and the source could be faster. -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Fri, Jul 23, 2010 at 03:07:21PM -0300, Carlos Carvalho wrote:
Keld Simonsen (keld@keldix.com) wrote on 23 July 2010 18:24:
I tried downloading
http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It gave a download speed of 1.15 MB/S or 9 Mbit/s with wget.
You have a 10 Gbit/s line and the overall bandwidth load at your site was about 1.5 Gbit/s.
How did you get this number?
I got the load figure and line speed from http://ftp.halifax.rwth-aachen.de
So there should be enough excessive bandwidth to play with. Do you have a limit on the line speed per connection?
I then tried my own site:
http://ftp.klid.dk/ftp/opensuse/distribution/11.3/iso/openSUSE-11.3-KDE4-Liv...
This was with wget 29.8 MB/s or about 240 Mbit/s - my system has a 1 Gbit/s connection and the load was 100 Mbit/s.
I then tried to download via bittorrent from your site http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It was something like 3.2 MB/s = 25 Mbit/s. Better than 9 Mbit/s
How do you know that you pulled only from halifax.rwth-aachen? When a site gets the iso it registers as a seeder with the tracker; it'll only be one more seeder among many. The torrent client will get a list of machines from the tracker and pull from many simultaneously, won't it? So if the bottleneck is not in your link you may be able to pull faster than from a single location. If the bottleneck is somewhere between you and the single source you may have lower flux even if both you and the source could be faster.
My link for the testing machine running the BT client is 1 Gbit/s. So it should not be the bottleneck. Yes, I cannot ensure that the 25 Mbit/s is only coming from the ftp.halifax.rwth-aachen.de machine. But I think this is how the BT that they operate is supposed to work. How else could I measure the BT speed for the halifax seeds? Anyway it is the user experience I am interested in. Best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Dear Keld and all, On Fri, Jul 23, 2010 at 06:24:51PM +0200, Keld Simonsen wrote:
It gave a download speed of 1.15 MB/S or 9 Mbit/s with wget.
That is not much. How good is your connection to the German/European research network? How fast in general is your connection to Germany? I don't think we can deliver 10 GBit/sec to Mexico, just to give some example. Maybe this is problematic for Denmark, too.
You have a 10 Gbit/s line and the overall bandwidth load at your site was about 1.5 Gbit/s. So there should be enough excessive bandwidth to play with.
Yes.
Do you have a limit on the line speed per connection?
No, there are not real limits. In my tests it is possible to get 1 GBit/sec with many machines at the same time.
http://ftp.klid.dk/ftp/opensuse/distribution/11.3/iso/openSUSE-11.3-KDE4-Liv...
This was with wget 29.8 MB/s or about 240 Mbit/s - my system has a 1 Gbit/s connection and the load was 100 Mbit/s.
I just tried downloading it from ftp.halifax.rwth-aachen.de and got about 4,7 MByte/sec (roughly 50 MBit/sec). So I guess the connection between our servers is not the best.
I then tried to download via bittorrent from your site http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It was something like 3.2 MB/s = 25 Mbit/s. Better than 9 Mbit/s So is this why your users use BT? They know it is faster than ftp?
I don't know why users actually use BitTorrent. It should be faster, because many peers participate and provide upstream. This is the theory and for hot files (think warez?) this works out just fine. We can reach more than 1 GBit/sec with BitTorrent, there is no real difference to the HTTP download you tried - I think it suffered the same problem. Besides, with BitTorrent you need to get the right peer list. Maybe you did not even download a single byte from our server.
I did not dee any big visible advertisement of the torrents.
Try www.opensuse.org and www.ubuntu.com (or whereever they announce their downloads). We as a mirror do not advertise, we just seed (upload).
So you have educated your users to use BT?
No. This is not our job.
I am just wondering how this is done from your side.
We are seeding where we feel like it. This is mostly OpenSUSE, Ubuntu and CentOS, as far as I remember. On Fri, Jul 23, 2010 at 03:07:21PM -0300, Carlos Carvalho wrote:
How did you get this number?
http://ftp.halifax.rwth-aachen.de shows some basic stats and http://ftp.halifax.rwth-aachen.de/~cotto/ gives more in-depth statistics. On Sat, Jul 24, 2010 at 10:46:48AM +0200, Keld Simonsen wrote:
My link for the testing machine running the BT client is 1 Gbit/s. So it should not be the bottleneck.
1 GBit/sec is not the universal answer. I guess your international routing is not well-equipped. You should not judge other servers or protocols if you can't manage to download _anything_ from _some_ server located in the same country or region with the speed you want.
Yes, I cannot ensure that the 25 Mbit/s is only coming from the ftp.halifax.rwth-aachen.de machine. But I think this is how the BT that they operate is supposed to work. How else could I measure the BT speed for the halifax seeds?
Most clients show speed per client (or IP). The IP for our server currently is 137.226.34.42. If this appears in your peer list, you can check how fast we upload to your client.
Anyway it is the user experience I am interested in.
We made the experience that a huge number of users actually use BitTorrent. We uploaded about 5 TByte of the 11.3 OpenSUSE DVDs only via BitTorrent, that is roughly 500-600x per DVD. With CentOS 5.5 I see 2.5 TByte in Total and roughly 300x uploads per DVD. This is peanuts compared to the HTTP traffic (30 TByte in one day for Mozilla!), but still significant enough so that it makes sense to support BitTorrent. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
On Sun, Jul 25, 2010 at 05:46:29PM +0200, Carsten Otto wrote:
Dear Keld and all,
On Fri, Jul 23, 2010 at 06:24:51PM +0200, Keld Simonsen wrote:
It gave a download speed of 1.15 MB/S or 9 Mbit/s with wget.
That is not much. How good is your connection to the German/European research network? How fast in general is your connection to Germany? I don't think we can deliver 10 GBit/sec to Mexico, just to give some example. Maybe this is problematic for Denmark, too.
My machines there are on the Danish research network, and close to the Danish Internet exchange (DIX) - some 100 meters from it. We have 10 Gbit on the university campus, but not (yet) to our machines. I would think there is good connection to German research network - I would think it would be a multi gigabit connection. It should be good for multi 100 mbit/s traffic. - not just 10 Mbit. But of coarse there can be bottlenecks - I am not sure where they are and I am interested in finding out.
You have a 10 Gbit/s line and the overall bandwidth load at your site was about 1.5 Gbit/s. So there should be enough excessive bandwidth to play with.
Yes.
Do you have a limit on the line speed per connection?
No, there are not real limits. In my tests it is possible to get 1 GBit/sec with many machines at the same time.
We are talking about 2 things here: 1. the total bandwidth consumed at a server machine 2. the obtainable bandwidth for an extra machine (a user). I am interested in both aspects. Do you get 1 Gbit/s from a testing client machine towards your server? Do you get it at this time (Sunday afternoon)?
http://ftp.klid.dk/ftp/opensuse/distribution/11.3/iso/openSUSE-11.3-KDE4-Liv...
This was with wget 29.8 MB/s or about 240 Mbit/s - my system has a 1 Gbit/s connection and the load was 100 Mbit/s.
I just tried downloading it from ftp.halifax.rwth-aachen.de and got about 4,7 MByte/sec (roughly 50 MBit/sec). So I guess the connection between our servers is not the best.
Hmm, I think you should not try benchmarking from your ftp machine, it could be that your file systems are very busy. I think your tests on your own machine was conducted on a client machine?
I then tried to download via bittorrent from your site http://ftp.halifax.rwth-aachen.de/opensuse/distribution/11.3/iso/openSUSE-11...
It was something like 3.2 MB/s = 25 Mbit/s. Better than 9 Mbit/s So is this why your users use BT? They know it is faster than ftp?
I don't know why users actually use BitTorrent. It should be faster, because many peers participate and provide upstream. This is the theory and for hot files (think warez?) this works out just fine. We can reach more than 1 GBit/sec with BitTorrent, there is no real difference to the HTTP download you tried - I think it suffered the same problem. Besides, with BitTorrent you need to get the right peer list. Maybe you did not even download a single byte from our server.
Hmm, how do I control the peer list to be just from your ftp site?
I did not dee any big visible advertisement of the torrents.
Try www.opensuse.org and www.ubuntu.com (or whereever they announce their downloads). We as a mirror do not advertise, we just seed (upload).
So you are just part of the general opensuse or ubuntu seed? That should not be too difficult for me to do also.
We are seeding where we feel like it. This is mostly OpenSUSE, Ubuntu and CentOS, as far as I remember.
On Fri, Jul 23, 2010 at 03:07:21PM -0300, Carlos Carvalho wrote:
How did you get this number?
http://ftp.halifax.rwth-aachen.de shows some basic stats and http://ftp.halifax.rwth-aachen.de/~cotto/ gives more in-depth statistics.
On Sat, Jul 24, 2010 at 10:46:48AM +0200, Keld Simonsen wrote:
My link for the testing machine running the BT client is 1 Gbit/s. So it should not be the bottleneck.
1 GBit/sec is not the universal answer. I guess your international routing is not well-equipped.
I don't know how well equipped things are here, I am not in charge of routing. Anyway I would expect it to be quite standard, they run a 10 Gbit backbone at the university campus, and they are hosting the national internet exchange, wgeer they then also have a number of international lines. Are you talking about international connection bandwidth?
You should not judge other servers or protocols if you can't manage to download _anything_ from _some_ server located in the same country or region with the speed you want.
Well, I am not trying to blame anybody. I am just puzzled over some things. And then I am asking people on this list as I think they are quite knowledgeable in what is going on. I do get fine speeds from other servers in Denmark, say 200 - 300 Mbit/s. So you are saying that the bottlenecks are not the ftp servers, but rather the national infrastructure, at least in some cases. This could give med some insights in what mirrors to chose for rsyncing, eg. I should prefer Danish or Nordic servers to European servers. And BT could also give and advantage. This also have inpact on normal users, many users have 20 - 50 - 100 Mbit/s download connectivity, and having only 10 Mbit/s because of national infrastructure issues could make the users want to have particular priorities.
Yes, I cannot ensure that the 25 Mbit/s is only coming from the ftp.halifax.rwth-aachen.de machine. But I think this is how the BT that they operate is supposed to work. How else could I measure the BT speed for the halifax seeds?
Most clients show speed per client (or IP). The IP for our server currently is 137.226.34.42. If this appears in your peer list, you can check how fast we upload to your client.
good!
Anyway it is the user experience I am interested in.
We made the experience that a huge number of users actually use BitTorrent. We uploaded about 5 TByte of the 11.3 OpenSUSE DVDs only via BitTorrent, that is roughly 500-600x per DVD. With CentOS 5.5 I see 2.5 TByte in Total and roughly 300x uploads per DVD. This is peanuts compared to the HTTP traffic (30 TByte in one day for Mozilla!), but still significant enough so that it makes sense to support BitTorrent.
So what are the main types of traffic for you? 30 TB a day is almost 3 gigabit... Thanks for your advice. I will have a look into it. best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sun, Jul 25, 2010 at 07:32:54PM +0200, Keld Simonsen wrote:
On Sun, Jul 25, 2010 at 05:46:29PM +0200, Carsten Otto wrote:
On Sat, Jul 24, 2010 at 10:46:48AM +0200, Keld Simonsen wrote:
My link for the testing machine running the BT client is 1 Gbit/s. So it should not be the bottleneck.
1 GBit/sec is not the universal answer. I guess your international routing is not well-equipped.
I don't know how well equipped things are here, I am not in charge of routing. Anyway I would expect it to be quite standard, they run a 10 Gbit backbone at the university campus, and they are hosting the national internet exchange, where they then also have a number of international lines. Are you talking about international connection bandwidth?
You should not judge other servers or protocols if you can't manage to download _anything_ from _some_ server located in the same country or region with the speed you want.
Well, I am not trying to blame anybody. I am just puzzled over some things. And then I am asking people on this list as I think they are quite knowledgeable in what is going on.
I do get fine speeds from other servers in Denmark, say 200 - 300 Mbit/s.
I am still puzzled. I did a traceroute traceroute to ftp.halifax.rwth-aachen.de (137.226.34.42), 30 hops max, 40 byte packets 1 192.38.78.161 (192.38.78.161) 0.462 ms 0.475 ms 0.526 ms 2 10g-dtu.ly0.core.fsknet.dk (130.225.166.241) 0.321 ms 0.352 ms 0.367 ms 3 dk-uni.nordu.net (109.105.102.37) 0.280 ms 0.296 ms 0.306 ms 4 dk-ore.nordu.net (109.105.97.13) 0.755 ms 0.778 ms 0.746 ms 5 nordunet.rt2.cop.dk.geant2.net (62.40.124.45) 0.900 ms 0.880 ms 0.878 ms 6 so-7-3-0.rt1.fra.de.geant2.net (62.40.112.49) 14.999 ms 15.014 ms 15.030 ms 7 dfn-gw.rt1.fra.de.geant2.net (62.40.124.34) 16.398 ms 16.409 ms 16.378 ms 8 xr-aac1-te2-2.x-win.dfn.de (188.1.144.109) 19.106 ms 19.236 ms 19.324 ms 9 kr-rwth-aachen.x-win.dfn.de (188.1.43.110) 19.757 ms 19.719 ms 19.736 ms 10 pfo-stone-2.noc.RWTH-Aachen.DE (134.130.9.234) 19.105 ms 19.013 ms 18.969 ms 11 n7k-lssnord-1.noc.RWTH-Aachen.DE (134.130.9.252) 19.519 ms 19.360 ms 19.314 ms 12 n6k-rz-1.noc.RWTH-Aachen.DE (137.226.118.50) 21.749 ms 21.716 ms 21.721 ms 13 ftp.halifax.RWTH-Aachen.DE (137.226.34.42) 19.251 ms 19.215 ms 19.182 ms So apart from the 2 first links, which are only 1 gigabit, the rest should be 10 gigabit. This is geant2 http://www.geant2.net/server/show/nav.941 - and a direct international link Copenhagen - Frankfurt of 10 gigabit is used. How can this be a bottleneck? Best regards Keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Dear Keld, On Sun, Jul 25, 2010 at 07:32:54PM +0200, Keld Simonsen wrote:
My machines there are on the Danish research network, and close to the Danish Internet exchange (DIX) - some 100 meters from it. We have 10 Gbit on the university campus, but not (yet) to our machines. I would think there is good connection to German research network - I would think it would be a multi gigabit connection. It should be good for multi 100 mbit/s traffic. - not just 10 Mbit.
Should, yes. In reality it is not.
No, there are not real limits. In my tests it is possible to get 1 GBit/sec with many machines at the same time.
We are talking about 2 things here:
1. the total bandwidth consumed at a server machine 2. the obtainable bandwidth for an extra machine (a user).
The server is able to provide at least 8 GBit/sec, we tested that. The university network currently is limited to about 400.000 packets/sec. Here we still need to experiment. Right now, and I guess all around the clock, I can get several GBit/sec from the machine when downloading some file. I don't see when/why this should not be the case (apart from the 400kp/s issue). 2010-07-25 20:05:12 (111 MB/s) - `/dev/null' saved [4697126912/4697126912] If you find a machine with 10 GBit/sec or more, a decent connection to our university and the 400kp/s issue is not there, I guess you can download with 5 GBit/sec and more. I can't test this, though.
Do you get 1 Gbit/s from a testing client machine towards your server? Do you get it at this time (Sunday afternoon)?
Yes. I get 1 GBit/sec each for several machines, if I want to. As said, we reached 8 GBit/sec already.
Hmm, I think you should not try benchmarking from your ftp machine, it could be that your file systems are very busy. I think your tests on your own machine was conducted on a client machine?
I downloaded to /dev/null, which is fast. Besides, we have a very fast file system and a lot of RAM to cache the download. This is clearly not the issue. The other tests were taken using other machines in the university, yes. A local download currently gives 8 GBit/sec: 2010-07-25 20:08:15 (1.01 GB/s) - `/dev/null' saved [4697126912/4697126912]
Hmm, how do I control the peer list to be just from your ftp site?
You don't. Instead you just look at the number for our server and ignore the others (and the total).
So you are just part of the general opensuse or ubuntu seed? That should not be too difficult for me to do also.
Yes, that is how BitTorrent works. One huge swarm for a single file.
I don't know how well equipped things are here, I am not in charge of routing. Anyway I would expect it to be quite standard, they run a 10 Gbit backbone at the university campus, and they are hosting the national internet exchange, wgeer they then also have a number of international lines. Are you talking about international connection bandwidth?
Yes, since we are in Germany and you are in Denmrak.
I do get fine speeds from other servers in Denmark, say 200 - 300 Mbit/s.
Denmark is not Germany. Try other servers in Germany. If you can find _any_ with a decent performance, we can start thinking about problems at our end.
So you are saying that the bottlenecks are not the ftp servers, but rather the national infrastructure, at least in some cases. This could give med some insights in what mirrors to chose for rsyncing, eg. I should prefer Danish or Nordic servers to European servers. And BT could also give and advantage.
Correct.
This also have inpact on normal users, many users have 20 - 50 - 100 Mbit/s download connectivity, and having only 10 Mbit/s because of national infrastructure issues could make the users want to have particular priorities.
I would not say that the connection between Denmark and Germany is 10 MBit/sec. But when you want to download something, you need to think of those limitations.
So what are the main types of traffic for you? 30 TB a day is almost 3 gigabit...
It is. In this case this was the automatic update for Firefox 3.6.7, so roughly 20% of all (European?) Firefox users downloaded from our server. This is a lot and in average we had 3 GBit/sec. Bye, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211
On Sun, Jul 25, 2010 at 08:14:19PM +0200, Carsten Otto wrote:
Dear Keld,
On Sun, Jul 25, 2010 at 07:32:54PM +0200, Keld Simonsen wrote:
My machines there are on the Danish research network, and close to the Danish Internet exchange (DIX) - some 100 meters from it. We have 10 Gbit on the university campus, but not (yet) to our machines. I would think there is good connection to German research network - I would think it would be a multi gigabit connection. It should be good for multi 100 mbit/s traffic. - not just 10 Mbit.
Should, yes. In reality it is not.
Hmm, I don't understand why. I have contacted local geant2 people and university people for an explanation. Do other mirror maintaners experience the same - that international connections on eg. 10 gigabit geant2 lines are quite slow? I have a mirror speed test program for pclinuxos that shows me that the max speed I ever get out of international connections are about 40 Mbit/s, and generally these mirrors have at least 1 gigabit connectivity and they are located on the European research network. I made some tests from my 2 servers with other Danish ISPs for the Aachen site, and they were generally faster than the geant2 lines, but still below 30 Mbit/s.
It is. In this case this was the automatic update for Firefox 3.6.7, so roughly 20% of all (European?) Firefox users downloaded from our server. This is a lot and in average we had 3 GBit/sec.
Impressive. How did you make the file system go that fast? Would your 1 Gigabyte/s speed come out of the RAM or out of the filesystem? Best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
That looks like a case for a PERT (Performance Enhancement and Response Team): http://edupert.geant.net/ NORDUnet has unfortunately no PERT listed here: http://edupert.geant.net/PERTs/pert_details.html However, DFN is listed to start the investigation at the other end. Thomas On 27.07.10 10:30, Keld Simonsen wrote:
On Sun, Jul 25, 2010 at 08:14:19PM +0200, Carsten Otto wrote:
Dear Keld,
On Sun, Jul 25, 2010 at 07:32:54PM +0200, Keld Simonsen wrote:
My machines there are on the Danish research network, and close to the Danish Internet exchange (DIX) - some 100 meters from it. We have 10 Gbit on the university campus, but not (yet) to our machines. I would think there is good connection to German research network - I would think it would be a multi gigabit connection. It should be good for multi 100 mbit/s traffic. - not just 10 Mbit.
Should, yes. In reality it is not.
Hmm, I don't understand why. I have contacted local geant2 people and university people for an explanation.
Do other mirror maintaners experience the same - that international connections on eg. 10 gigabit geant2 lines are quite slow? I have a mirror speed test program for pclinuxos that shows me that the max speed I ever get out of international connections are about 40 Mbit/s, and generally these mirrors have at least 1 gigabit connectivity and they are located on the European research network.
I made some tests from my 2 servers with other Danish ISPs for the Aachen site, and they were generally faster than the geant2 lines, but still below 30 Mbit/s.
It is. In this case this was the automatic update for Firefox 3.6.7, so roughly 20% of all (European?) Firefox users downloaded from our server. This is a lot and in average we had 3 GBit/sec.
Impressive. How did you make the file system go that fast? Would your 1 Gigabyte/s speed come out of the RAM or out of the filesystem?
Best regards keld
-- SWITCH Serving Swiss Universities -------------------------- Thomas Lenggenhager P.O. Box, 8021 Zurich, Switzerland http://mirror.switch.ch -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Tue, Jul 27, 2010 at 10:30:42AM +0200, Keld Simonsen wrote:
Impressive. How did you make the file system go that fast? Would your 1 Gigabyte/s speed come out of the RAM or out of the filesystem?
We have a good disk backend, but for really interesting speeds you need to have the files in RAM. Since we have 60 GByte of DDR3, this works out quite fine. Best regards, -- Carsten Otto otto@informatik.rwth-aachen.de LuFG Informatik 2 http://verify.rwth-aachen.de/otto/ RWTH Aachen phone: +49 241 80-21211 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Thu, Jul 22, 2010 at 01:23:01PM +0200, Keld Simonsen wrote:
Personally I would like to exploit the use of bittorrent seeds hosted by mirrors - maybe that would scale well, and give quite evenly spread bandwidth use on the mirror infrastructure. Maybe gwdg.de can tell us more, and whether there could be some generalized way to do this in a distributed way. And also about performance - I have had bad experience with a lot of HTTP connections and therefore I redirected all .iso downloads to FTP.
During the past months, I have been working on native torrent support in MirrorBrain. It is relatively complete. Contrary to the current torrents that openSUSE offers, the torrents generated by MirrorBrain will contain mirror URLs for web seeding. That should help, and as far as I understand, save mirror admins the work of seeding manually. Of course, the listed mirrors will automatically be the ones closest to the user. The next MirrorBrain release will ship the new code. I can't say when I find the time to accomplish the release, but the code is already running at OOo's download server, e.g.: http://download.services.openoffice.org/files/stable/3.2.1/OOo_3.2.1_Linux_x... It cannot really be tried out, though, because no involved tracker knows about the torrents yet. I thought this features might be interesting for openSUSE. I'm relatively new to torrents though, and torrents are badly standardized and I found it hard to create a "good" torrent. There might be bugs or things that can be improved. Hints appreciated. Peter
On Saturday 17 July 2010 17:07:50 Keld Simonsen wrote:
Anyway it looks like the novell site has been taken off the list for denmark.
We use Akamai only for the first day of the release. Let's see for the next release whether we should give it a lower priority initially - or what are your suggestions? What seems missing IMHO from mirrorbrain is dynamic adjustments - so if gwdg has bandwidth, we should raise it's priority and if it's saturated, lower it. That way it could balance load even better... Andreas -- Andreas Jaeger, Program Manager openSUSE, aj@{novell.com,opensuse.org} Twitter: jaegerandi | Identica: jaegerandi SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
On Sun, Jul 18, 2010 at 02:37:03PM +0200, Andreas Jaeger wrote:
On Saturday 17 July 2010 17:07:50 Keld Simonsen wrote:
Anyway it looks like the novell site has been taken off the list for denmark.
We use Akamai only for the first day of the release. Let's see for the next release whether we should give it a lower priority initially - or what are your suggestions?
What seems missing IMHO from mirrorbrain is dynamic adjustments - so if gwdg has bandwidth, we should raise it's priority and if it's saturated, lower it. That way it could balance load even better...
If I remember correctly, we once where thinking about offering mirror admins a way to set their mirrobrain score. However, as the score is relative to other mirrors in the same country / region, we would need another way for mirror admins to set their available bandwidth. Do you guys have thoughts on this? Best Christoph -- Christoph Thiel, Tech. Project Management, Research & Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Sun, Jul 18, 2010 at 02:37:03PM +0200, Andreas Jaeger wrote:
On Saturday 17 July 2010 17:07:50 Keld Simonsen wrote:
Anyway it looks like the novell site has been taken off the list for denmark.
We use Akamai only for the first day of the release. Let's see for the next release whether we should give it a lower priority initially - or what are your suggestions?
Well, some thoughts: Try to estimate the total bandwidth use for the next release, given that it would be an even greater success than 11.3. That is: look at the previous couple of releases, extimate the total bandwith use, over the peek days, and extrapolate from there. then see if there is a need for the Akamai service, and what the need should be. Also bear in mind that the mirrors in the opensuse infrastructure have other obligations, eg serving other distributions, and that they may not be able to serve full nominal bandwidth. I know other distributions have some reopsitory bandwidth testing scripts, eg pclinuxos, and that could probably be exploited to measure the individual mirrors download speed.
What seems missing IMHO from mirrorbrain is dynamic adjustments - so if gwdg has bandwidth, we should raise it's priority and if it's saturated, lower it. That way it could balance load even better...
Yes, a good idea. I would like something that is well documented and general enough for also other distributions to use. It seems that almost every distribution has their own way of administering mirrors and administering downloads. I would like at least some building bricks that could ease the work with distribution distribution (sic!). best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Carsten Otto <ftp@halifax.rwth-aachen.de> writes:
Hi,
On Fri, Jul 16, 2010 at 05:18:12PM +0200, Andreas Jaeger wrote:
I'm blogging about the launch and will publish my article later today. The current version is here: http://pastie.org/private/orcmz76zjcybhy0tdki6ea
If you have anything to share regarding bandwidth/download numbers, please tell me.
Thanks for referencing Fujitsu as our sponsor, I hope they like it!
Some minor remarks or wishes:
- a few examples -> two examples - sat -> set - with 4 gbit/s -> with peaks of 4 gbit/s - gbit/s -> GBit/sec or GBit/s
- by Fujitsu. -> by Fujitsu and will be upgraded from 2 GBit/sec to 10 GBit/sec in the next few days. [if you wait long enough I can tell you if the upgrade already happened]
I updated the post.
Btw, our mirror was quite bored by this release with barely any disk or CPU activity. I hope this changes with the 10 GBit/sec line. Do you know when the next release is scheduled? :)
That's bad ;) Next release is in 8month time - mid of March 2011, thanks, Andreas -- Andreas Jaeger, aj@{novell.com,opensuse.org} Twitter: jaegerandi | Identica: jaegerandi SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
participants (10)
-
Andreas Jaeger -
Andreas Jaeger -
Carlos Carvalho -
Carsten Otto -
Christoph Thiel -
Eberhard Moenkeberg -
Keld Simonsen -
Marcus Meissner -
Peter Poeml -
SWITCHmirror Admins