[opensuse-buildservice] OBS aggregate performance
Hi, I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes. I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated. So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 3. April 2013, 11:45:13 wrote UkJung Kim:
Hi,
I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes.
I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated.
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
No, because it needs also to be re-signed, so it is not the same binary. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Apr 3, 2013 at 4:58 AM, Adrian Schröter
I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes.
I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated.
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
No, because it needs also to be re-signed, so it is not the same binary.
Still, it must be faster than linking packages (and thus rebuilding them) mustn't it? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 3. April 2013, 10:54:17 wrote Claudio Freire:
On Wed, Apr 3, 2013 at 4:58 AM, Adrian Schröter
wrote: I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes.
I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated.
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
No, because it needs also to be re-signed, so it is not the same binary.
Still, it must be faster than linking packages (and thus rebuilding them) mustn't it?
I do not get you here, what do you compare? -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wed, Apr 3, 2013 at 11:33 AM, Adrian Schröter
On Mittwoch, 3. April 2013, 10:54:17 wrote Claudio Freire:
On Wed, Apr 3, 2013 at 4:58 AM, Adrian Schröter
wrote: I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes.
I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated.
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
No, because it needs also to be re-signed, so it is not the same binary.
Still, it must be faster than linking packages (and thus rebuilding them) mustn't it?
I do not get you here, what do you compare?
Aggregate vs link. I see the possible confusion with symlinks. I mean package links. Not sure why I brought links up. Anyway, the point was that aggregates copy build results, so they skip the building phase. So it must be faster, even though it has to re-sign, than linking, which involves re-building (and re-signing). So, whereas the performance of the signer could be improved perhaps, aggregating itself doesn't seem like an optimization candidate. The fact that it takes minutes for hundreds of packages might not be bad per-se. Depends on how many minutes and how many hundreds, and whether it's performing more poorly than the corresponding signing phase of linked packages (it shouldn't, that would be a bug). -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 3. April 2013, 11:37:54 wrote Claudio Freire:
On Wed, Apr 3, 2013 at 11:33 AM, Adrian Schröter
wrote: On Mittwoch, 3. April 2013, 10:54:17 wrote Claudio Freire:
On Wed, Apr 3, 2013 at 4:58 AM, Adrian Schröter
wrote: I'd like to use 'aggregate' for hundreds of packages, but the performance of aggregating seems poor. scheduler seems stopped working for a minutes.
I found that the 'aggregate' actually copies all the packages built from aggregated packages to new package repo. It took a time if hundreds of packages to be aggregated.
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
No, because it needs also to be re-signed, so it is not the same binary.
Still, it must be faster than linking packages (and thus rebuilding them) mustn't it?
I do not get you here, what do you compare?
Aggregate vs link. I see the possible confusion with symlinks. I mean package links. Not sure why I brought links up.
Anyway, the point was that aggregates copy build results, so they skip the building phase. So it must be faster, even though it has to re-sign, than linking, which involves re-building (and re-signing).
He had a hard link on disk in mind when he spoke about linking, not about OBS source links ...
So, whereas the performance of the signer could be improved perhaps, aggregating itself doesn't seem like an optimization candidate. The fact that it takes minutes for hundreds of packages might not be bad per-se. Depends on how many minutes and how many hundreds, and whether it's performing more poorly than the corresponding signing phase of linked packages (it shouldn't, that would be a bug).
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Apr 2, 2013 at 11:45 PM, UkJung Kim
So, my question is that how I can improve the performance of aggregating, if it is possible to replace with link instead of copy?
Checking the source, I noticed bs_sched uses the API to get the packages, with view=cpio. Checking BSHTTP.pm to see how this all worked, there are parts that sound quite sub-optimal. Not sure it's the bottleneck, but for large files or large number of them it might. cpio_sender could be improved by use of sendfile, to avoid creating so many small (8k) chunks and instead creating one big chunk per file (HTTP supports it just fine), or many big ones, but also avoiding a lot of data copying. cpio_receiver could be similarly optimized, but that would require kernels newer than 2.6.33 (version at which to-file sendfile support was added). This could be especially efficient, if done both in sender and receiver, for local (loopback) connections, almost equal in performance to a "cp". Not sure if that is the case for b.o.o, but it sure is for appliances. How significant a win that would be, would require some profiling I'd imagine. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter
-
Claudio Freire
-
UkJung Kim