Hello, Just a query in relation to the build service. In relation to Ram(memory) and Disk, I know things get done quicker in ram than on disk.
Question: Could a ram disk be utilized do the compilation of packges/projects/rpms in faster memory instead of slower disk ?
The build logs/completed rpms/are then copied off to disk, and the ramdisk is cleared off before the next project starts compiling. Get more packages done in the same amount of time. Extra Ram costs money but there would be a benefit as well. Also see http://www.vanemery.com/Linux/Ramdisk/ramdisk.html http://www.linuxfocus.org/English/November1999/article124.html Just a suggestion, worth a try, comments welcome. Glenn
Am Sonntag, 2. August 2009 16:27:39 schrieb doiggl@velocitynet.com.au:
Hello, Just a query in relation to the build service. In relation to Ram(memory) and Disk, I know things get done quicker in ram than on disk.
Question: Could a ram disk be utilized do the compilation of packges/projects/rpms in faster memory instead of slower disk ?
yes
The build logs/completed rpms/are then copied off to disk, and the ramdisk is cleared off before the next project starts compiling. Get more packages done in the same amount of time. Extra Ram costs money but there would be a benefit as well.
Exactly. And you need mainboards which can take this amount of memory at all (32GB * 16 = 0.5 TB RAM)
So this is too expensive, but the systems should use the existing ram for caching as much as possible in our configuration.
bye adrian
Dne Po 3. srpna 2009 09:21:46 Adrian Schröter napsal(a):
Am Sonntag, 2. August 2009 16:27:39 schrieb doiggl@velocitynet.com.au:
Hello, Just a query in relation to the build service. In relation to Ram(memory) and Disk, I know things get done quicker in ram than on disk.
Question: Could a ram disk be utilized do the compilation of packges/projects/rpms in faster memory instead of slower disk ?
yes
The build logs/completed rpms/are then copied off to disk, and the ramdisk is cleared off before the next project starts compiling. Get more packages done in the same amount of time. Extra Ram costs money but there would be a benefit as well.
Exactly. And you need mainboards which can take this amount of memory at all (32GB * 16 = 0.5 TB RAM)
So this is too expensive, but the systems should use the existing ram for caching as much as possible in our configuration.
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in a ram disk. The build root creation should be speeded up using similar technique as Yast uses - installation from images.
Regards Michal Vyskocil
Am Montag, 3. August 2009 09:53:14 schrieb Michal Vyskocil:
Dne Po 3. srpna 2009 09:21:46 Adrian Schröter napsal(a):
Am Sonntag, 2. August 2009 16:27:39 schrieb doiggl@velocitynet.com.au:
Hello, Just a query in relation to the build service. In relation to Ram(memory) and Disk, I know things get done quicker in ram than on disk.
Question: Could a ram disk be utilized do the compilation of packges/projects/rpms in faster memory instead of slower disk ?
yes
The build logs/completed rpms/are then copied off to disk, and the ramdisk is cleared off before the next project starts compiling. Get more packages done in the same amount of time. Extra Ram costs money but there would be a benefit as well.
Exactly. And you need mainboards which can take this amount of memory at all (32GB * 16 = 0.5 TB RAM)
So this is too expensive, but the systems should use the existing ram for caching as much as possible in our configuration.
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in a ram disk. The build root creation should be speeded up using similar technique as Yast uses - installation from images.
Where is the advantage ?
We use the hdd, but if there is enough memory, the kernel will not write out anything. Or at a point of time which is best for it.
We played with images based setup, but given the speed of current CPUs, the only platform where this helped was s390. Okay, the ARM builds would be a good candidate maybe too.
bye adrian
Dne Po 3. srpna 2009 09:59:25 Adrian Schröter napsal(a):
Am Montag, 3. August 2009 09:53:14 schrieb Michal Vyskocil:
[snip]
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in a ram disk. The build root creation should be speeded up using similar technique as Yast uses - installation from images.
Where is the advantage ?
We use the hdd, but if there is enough memory, the kernel will not write out anything. Or at a point of time which is best for it.
That was just an idea. I use tmpfs for my local builds and have a good experience with it, but I can imagine that this is not doable in a build farm.
We played with images based setup, but given the speed of current CPUs, the only platform where this helped was s390. Okay, the ARM builds would be a good candidate maybe too.
Good to know.
Best regards Michal Vyskocil
Adrian wrote:
Exactly. And you need mainboards which can take this amount of memory
at
all (32GB * 16 = 0.5 TB RAM) So this is too expensive, but the systems should use the existing ram for caching as much as possible in our configuration.
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in
a
ram disk. The build root creation should be speeded up using similar technique as Yast uses - installation from images.
Where is the advantage ? We use the hdd, but if there is enough memory, the kernel will not write out anything. Or at a point of time which is best for it. We played with images based setup, but given the speed of current CPUs, the only platform where this helped was s390. Okay, the ARM builds would
be a
good candidate maybe too.
bye adrian
Michal wrote:
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in a ram disk. The build root creation should be speeded up using similar
technique
as Yast uses - installation from images. Regards Michal Vyskocil
How about this: - If you use existing main board, add 1 or 2 extra memory modules to that. - Create a ramdisk using that ram. - Build the smaller small packages on that. Cheers Glenn
Am Dienstag, 4. August 2009 06:13:29 schrieb doiggl@velocitynet.com.au:
Adrian wrote:
Exactly. And you need mainboards which can take this amount of memory
at
all (32GB * 16 = 0.5 TB RAM) So this is too expensive, but the systems should use the existing ram for caching as much as possible in our configuration.
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in
a
ram disk. The build root creation should be speeded up using similar technique as Yast uses - installation from images.
Where is the advantage ? We use the hdd, but if there is enough memory, the kernel will not write out anything. Or at a point of time which is best for it. We played with images based setup, but given the speed of current CPUs, the only platform where this helped was s390. Okay, the ARM builds would
be a
good candidate maybe too.
bye adrian
Michal wrote:
What about use tmpfs for /usr/src/packages/BUILD (at least for small packages)? It will not need a huge amount of RAM as whole build root in a ram disk. The build root creation should be speeded up using similar
technique
as Yast uses - installation from images. Regards Michal Vyskocil
How about this:
- If you use existing main board, add 1 or 2 extra memory modules to that.
- Create a ramdisk using that ram.
- Build the smaller small packages on that.
Why should I configure a ramdisk at all ? The kernel is using already all memory for caching. Creating a ramdisk means that I take away a partition of it and the kernel can not decide on its own for what to use the memory for caching.
buildservice@lists.opensuse.org
-
Adrian Schröter -
doiggl＠velocitynet.com.au -
Michal Vyskocil