Mailinglist Archive: mirror (76 mails)

< Previous Next >
Re: [suse-mirror] Blogging about 11.3 launch - THANKS!
On Thu, Jul 22, 2010 at 02:37:21PM +0200, Keld Simonsen wrote:
Sounds interesting! So this is dependent on all isos in RAM, to avoid
random disk access to clutter up the disks...

Yes.

Would it be possible to
eg use bigger block sizes for BT so that the random access would not
be so performance impacting? I was thinking about maybe block sizes of 1 MB,
without investigating what BT currently does, and whether it is
easy to change block sizes wth BT.

As far as I know, this is not possible. However, we were thinking (when
we had far less than 60 GByte of RAM) to limit the amount of data that
we seed (and announce and get from disk/RAM). This amount can be
configured so that it fits into RAM and should be changed based on
demand (there's no sense to seed the first half and ignore the second
half of the file). This would need some changes in the local BitTorrent
client, so this is still only an idea.

My server does only have 1.5 TB of RAM,

Wow? :)

and I think what you suggest
would need much more RAM. Anyway, RAM is cheap these days. What would be the
recommended amount of RAM? And would it scale? I was thinking about having
most of the mirror data available via BT, in an automated and distributed
way, and then having all data in RAM seems futile, I have about 5 TB data on
my mirror server. I would think bigger sizes could be a better way to scale.

I'd suggest at least 4 GByte for the system, 4 GByte for standard caches
(depending on your traffic, amount of data, disk backend speed) and X
GByte for as many X as you want to seed. If you seed files that are
already served via HTTP, you can save on the intersection, of course.
Having most of the mirror data availabe via BitTorrent is not such a
good idea, I think. BitTorrent is especially useful for 'hot' files,
e.g. OpenSUSE DVDs in the first few days after a release. This way you
'only' need about 10 GByte of RAM for these files.

Anyway - again without consulting the net, I think it could be useful
with a wiki article on mirroring software and practices, eg on wikipedia,
which could be a neutral place to gather the technology and experience
of distribution and mirror maintainers on this subject.

I don't think Wikipedia is the right place for technical documentation,
but I'd be glad to participate in whatever documentation project.

Best regards,
--
Carsten Otto
carsten@xxxxxxxxx
www.c-otto.de
< Previous Next >
List Navigation
Follow Ups