On Thu, Jul 22, 2010 at 02:37:21PM +0200, Keld Simonsen wrote:
Sounds interesting! So this is dependent on all isos in RAM, to avoid random disk access to clutter up the disks...
Would it be possible to eg use bigger block sizes for BT so that the random access would not be so performance impacting? I was thinking about maybe block sizes of 1 MB, without investigating what BT currently does, and whether it is easy to change block sizes wth BT.
As far as I know, this is not possible. However, we were thinking (when we had far less than 60 GByte of RAM) to limit the amount of data that we seed (and announce and get from disk/RAM). This amount can be configured so that it fits into RAM and should be changed based on demand (there's no sense to seed the first half and ignore the second half of the file). This would need some changes in the local BitTorrent client, so this is still only an idea.
My server does only have 1.5 TB of RAM,
and I think what you suggest would need much more RAM. Anyway, RAM is cheap these days. What would be the recommended amount of RAM? And would it scale? I was thinking about having most of the mirror data available via BT, in an automated and distributed way, and then having all data in RAM seems futile, I have about 5 TB data on my mirror server. I would think bigger sizes could be a better way to scale.
I'd suggest at least 4 GByte for the system, 4 GByte for standard caches (depending on your traffic, amount of data, disk backend speed) and X GByte for as many X as you want to seed. If you seed files that are already served via HTTP, you can save on the intersection, of course. Having most of the mirror data availabe via BitTorrent is not such a good idea, I think. BitTorrent is especially useful for 'hot' files, e.g. OpenSUSE DVDs in the first few days after a release. This way you 'only' need about 10 GByte of RAM for these files.
Anyway - again without consulting the net, I think it could be useful with a wiki article on mirroring software and practices, eg on wikipedia, which could be a neutral place to gather the technology and experience of distribution and mirror maintainers on this subject.
I don't think Wikipedia is the right place for technical documentation, but I'd be glad to participate in whatever documentation project.