Mailinglist Archive: mirror (76 mails)

< Previous Next >
Re: [suse-mirror] Blogging about 11.3 launch - THANKS!
On Thu, Jul 22, 2010 at 02:47:28PM +0200, Carsten Otto wrote:
On Thu, Jul 22, 2010 at 02:37:21PM +0200, Keld Simonsen wrote:
Sounds interesting! So this is dependent on all isos in RAM, to avoid
random disk access to clutter up the disks...

Yes.

Would it be possible to
eg use bigger block sizes for BT so that the random access would not
be so performance impacting? I was thinking about maybe block sizes of 1
MB,
without investigating what BT currently does, and whether it is
easy to change block sizes wth BT.

As far as I know, this is not possible. However, we were thinking (when
we had far less than 60 GByte of RAM) to limit the amount of data that
we seed (and announce and get from disk/RAM). This amount can be
configured so that it fits into RAM and should be changed based on
demand (there's no sense to seed the first half and ignore the second
half of the file). This would need some changes in the local BitTorrent
client, so this is still only an idea.

My server does only have 1.5 TB of RAM,

Wow? :)

I am sorry, I meant 1.5 GB RAM... It is a small machine, but I am quite
amazed about the performance I get out of it.


and I think what you suggest
would need much more RAM. Anyway, RAM is cheap these days. What would be the
recommended amount of RAM? And would it scale? I was thinking about having
most of the mirror data available via BT, in an automated and distributed
way, and then having all data in RAM seems futile, I have about 5 TB data on
my mirror server. I would think bigger sizes could be a better way to scale.

I'd suggest at least 4 GByte for the system, 4 GByte for standard caches
(depending on your traffic, amount of data, disk backend speed) and X
GByte for as many X as you want to seed. If you seed files that are
already served via HTTP, you can save on the intersection, of course.
Having most of the mirror data availabe via BitTorrent is not such a
good idea, I think. BitTorrent is especially useful for 'hot' files,
e.g. OpenSUSE DVDs in the first few days after a release. This way you
'only' need about 10 GByte of RAM for these files.

Hmm, this would mean a new server for us, and I am not sure we can find
the money to finance it.

Anyway our performance is pretty good, we have max download speeds of
about 300 Mbit/s.

And I wonder if it would be any good for us to do our own torrents.
I tried it once, without any significant results/traffic. I think if we
made our own torrents, then users need to know the specific torrents URLs
to utilize them, and that would not be well advertised. We rely much on central
redistribution services like mirrorbrain to get our traffic.
And I think the torrents should be set up automatically, and not by hand,
and torrents should probably go away after the peak time, if every torrent
need to be in RAM.

Anyway - again without consulting the net, I think it could be useful
with a wiki article on mirroring software and practices, eg on wikipedia,
which could be a neutral place to gather the technology and experience
of distribution and mirror maintainers on this subject.

I don't think Wikipedia is the right place for technical documentation,
but I'd be glad to participate in whatever documentation project.

Wikipedia seems to be well maintained, and it is a neutral place.
And I see a number of technical overview articles there, Anyway, would there
be other places better suited for such content?

best regards
keld
--
To unsubscribe, e-mail: mirror+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: mirror+help@xxxxxxxxxxxx

< Previous Next >
List Navigation
Follow Ups