On Thu, 1 Sep 2005, Lode Vermeiren wrote:

Why can't openSUSE use a wider mirror system, using 'established' mirrors like mirrorservice.org, ibiblio, ftp.belnet.be, planetmirror, ... ? Most of these are SUSE mirrors anyway, why not put openSUSE under this tree and make it get mirrored automagically?

The problem we are facing here is that the openSUSE trees are really huge. We decided not to put the openSUSE stuff into the existing SUSE trees, in order to keep the size of that trees on the same level.

...

...

I can personally offer a server to be online at the same time as the announcement is done (as I have access to the iso's), but to be honest it is not near as grunty as Eberhard's server, but maybe all together we can make a difference? And I can get the torrent live as soon as the announcement is done.

After the first beta I put one of my boxes online as a mirror as well for a day, but it proved to be too big of a load on the rest of the sites. I burned about 100 Gb of bandwidth in one day, so there's definitely a need for fast and reliable mirrors.

Yes, ftp.gwdg.de is delivering some 1.5-2 TB per day...! Regards Christoph

I agree, for example I only need the iso's for mirroring purposes, but I am using the inst-source to update my machine to the next beta and offer a local installation source for those who need one, and these should be easier to mirror (rsync) and could be split from the plain iso download... Daniel Adrian Schroeter wrote:

We have currently too much ouput per month and the mirrors need longer syncing times than our beta schedule does allow.

We will address this via seperating the rpm trees from the isos and offering a new syncing mechanism for the trees which do only need to download the deltas.

But this is something for post 10.0 RC1 ...

For now there is only BitTorrent.

bye adrian

hello, Lode Vermeiren wrote:

Bittorrent is indeed a good option, but it can't be the only one. Lots of us, at least here in Belgium, are on ADSL. This means that uploading directly affects your download speed, and that upload speed is generally capped at a fraction of the maximum download speed, which is 'punished' by the way BT works.

if you allow a maximum speed for uploads, your download speed will increase, and you'll be able to share sooner, so the community will agree ... the default setting for clients like azureus is not optimal. with an adsl connection in france, it works great.

Why can't openSUSE use a wider mirror system, using 'established' mirrors like mirrorservice.org, ibiblio, ftp.belnet.be, planetmirror, ... ? Most of these are SUSE mirrors anyway, why not put openSUSE under this tree and make it get mirrored automagically?

like the others, ftp is no use for such big projects as opensuse. i think bt is much more scallable and efficient than ftp (for opensuse at least). ftp is good only for updates, until bt can be used for that too. paul.

I'n on the line with Pascal - the only thing the community can do is to use the torrents. not everyone in the community has access to a server that coulc act as a sort mirror (not even a slow one). to get the situation with BitTorrent (BT) better, there is something both - Novell / SUSE and the community can do: the "backbone" of BT are the seeders - users who have the complette files and are online as long as possible and so fast as possible. the *trick* could be as fallows: - at time X, all needed stuff is made by Novell/SUSE - RPMs, ISOs etc. the files will find there way to eberhard, so he can setup the main mirror - at time X+1 a working torrent-file is created, and on some place there will be a _main-seeder_ - maybe at Novell/SUSE, maybe at GWDG. importend: it has to be a *fast* seeder. and it has to accept as mutch clients as possible. at this time, no ISOs are public aviable jet, we are still before launchtime. - at time X+2 the torrent-file will be send out to the community - to all on this list, or to "special" users of the community. how ever, everyone starts his BT-client and downloads the ISOs. when there is a download speed around 50 - 100 kb/s, the complete files will be out in 10 - 14 hours. (this is the speed I have currently at 10 AM local german time ...). as near as possible as the main backbone is internal up und ready, also the resnc of mirrors can start. the mirror admins have to hold back all files and stuff till [launchtime+Y] so [X+2] should be arrount [launchtime - 12 hours] - at [launchtime] the BT-file will be announced at openSUSE.org. at this time there will be out sufficient data in the BT-network, so the users will have a acceptable download-speed. - at [launchtime+Y] when allmost more mirrors are ready the ISO- files will be announced at openSUSE.org. behind the downlaod- link at the wiki there will be run the load balancer that has his "eye" on all the mirrors and makes the redict to one og them. so not only af gwdg.de. Y should be arround 24 hours - I think that will be a timeframe that solves some of the problems that a) users and community wants to have the ISOs and b) the needed time to get more mirrors in sync with gwdg.de there is also the option *jigdo* that will be look good to solve some piece of the problem. I've worked with it for the win-GUI the last night, and I think that will be a good option (even I think, the need of a working GUI on other plattforms then Win is realy needed, and the main downloader jigdo-lite will realy need some improvemends (for example: it should be possible to have more then one wget running at the same time - even on slow connections). and last but not least: I agree with others that Novell/SUSE realy should think about the option for more support at gwdg.de for the situation *mirroring* - but that will be aout of the timeframe of this release I think. at least: the numbers of seeds with complette ISOs is up from 1 to 3 at 10.55 AM local german time - so, if users of / or the the community get more involved into the BT-track it will improve the BT-track, regards, JBScout

Hi, On Fri, 2 Sep 2005, jdd wrote:

Pascal Bleser wrote:

...

BTW, Eberhard, thanks again for all your hard work, we (almost) all very much appreciate what you are doing :)

must be said. And if beta is a little late, don't bother, it's only a proof of success.

what is necessary is a solution for the final release (this is also beta :-)

ftp.gwdg.de is "almost complete" since the night - meaning: only some *-debuginfo-* RPMs are still missing. Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)

Yeah ... the question is : Is a bittorrent only realease at first still considered an on schedule realease & then fully synced servers can serve the iso's later. Is it rude to say that people have to use BT for the first day or two ? --- Alexander Grujic wrote:

Hi List,

wouldn't it maybe help to at first only release the isos via bittorrent and add the ftp servers a few days later?

That way all the "I need the new version NOW" people wouldn't rush the ftp servers, bittorrent would serv up faster (those people usually have hight bandwiths) and those people that have a bit more patience can download conveniently from the ftp servers at a later point.

The mirrors would have a bit more time to sync and especially gwdg wouldn't suffer so much from the first onslaught...

Just an Idea,

Alexander

___________________________________________________________ How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos http://uk.photos.yahoo.com

Yes I will also give my support to an initial BitTorrent only release. The clever way to do this for future releases, is to call the ftp release day the "launch" date, and have that date on the project schedule. The BitTorrent release is then the Community release date a day or so earlier. Peter. On 02/09/05, Alexander Grujic wrote:

I think you can call it a release as soon as you make it available to a broad public, and i wouldn't think it's rude... at least i would not feel offended ;-)

JBScout [Thomas Lodewick] had the same idea (and postet it earlier on the list in a much more elaborte way, so I think there are at least two of us that would not be offended ;-))

Alex

Am Donnerstag, 1. September 2005 19:31 schrieb Winston Graeme:

...

Yeah ... the question is : Is a bittorrent only realease at first still considered an on schedule realease & then fully synced servers can serve the iso's later.

Is it rude to say that people have to use BT for the first day or two ?

On Thu, Sep 01, 2005 at 03:06:36PM +0200, Siegbert Baude wrote:

For me the main problem seems to be, that the mirrors don't manage to rsync in a fast way (that there is not enough funding for Eberhard's server is a problem which can't be solved technically ;-) ). Isn't it possible to allow the mirrors to rsync with full bandwith by traffic shaping or does traffic shaping simply not scale up to this level?

The point is that the main problem here is not the network bandwidth but the disk I/O resulting from the enormous size of the data. This is similar to your desktop machine with 512 MB memory when you open and use so many applications at the same time that they use 3 GB if memory. In that case the machine starts swapping in and out memory pages resulting in a real slow system. Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de

On Thu, Sep 01, 2005 at 05:51:30PM +0200, Siegbert Baude wrote:

So, is traffic shaping or packet filtering to prioritize the mirrors no option?

In theory this option does exist but in practice it is almost impossible to find useful limit values for the traffic shaper that do not make the server useless for other people as long as traffic is not the limiting factor. Note that controlling a complex system is only effective when you controll the limiting factor. The method would actually work if the traffic shaper would actually have control over disk I/O scheduling priority as well but AFAIK there is no such implementation for linux availlable yet. Robert -- Robert Schiele Tel.: +49-621-181-2214 Dipl.-Wirtsch.informatiker mailto:rschiele@uni-mannheim.de

Guten Abend Eberhard (Morgen hier)

Please take my apologies back; it was my hardest night ever, with lots of faults at my side while trying to parallelize as much rsyncs as possible, and only 2 hours sleep between the days.

Totally accepted and understood. I have been there and can really feel for you, mate! Take care of yourself ...... All the best Andreas openSUSE is SUPER: To help in the SUSE Performance Enhanced Release project visit http://www.opensuse.org/index.php/SUPER