[opensuse-factory] Reducing YaST rebuild time by 30%
The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers. https://lizards.opensuse.org/2016/10/11/reducing-yast-rebuild-time-by-30/ Enjoy. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Ancor Gonzalez Sosa schrieb:
The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers.
Thanks for taking that into consideration :-) YaST usually doesn't leave a negative impression when waiting for rebuilds though. There are worse components still. What's really key for rebuild times of the distribution is to take full advantage of the massively parallel build worker installation we have in OBS. This simulation of the rebuild time for example shows that there's no significant improvement from increasing the number of workers from 40 to 100 (rebuild time in practice is even longer): http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100) Shades of blue represent building, scheduled and blocked there. Ideally the bright 'building' line should not have interruptions with enough scheduled packages to pick from. So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism. And of course libreoffice forms the long tail :-) cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Ludwig Nussel wrote:
And of course libreoffice forms the long tail :-)
Heh - well the code itself can be built massively-parallel (100-fold or so, then the effects taper off). It's possible that help & translations add some extra delays towards the end, but in principle that could be split out to a different builder. :) Just sayin', -- Thorsten
On Wed, 2016-10-12 at 12:32 +0200, Thorsten Behrens wrote:
Ludwig Nussel wrote:
And of course libreoffice forms the long tail :-)
Heh - well the code itself can be built massively-parallel (100-fold or so, then the effects taper off). It's possible that help & translations add some extra delays towards the end, but in principle that could be split out to a different builder. :)
Just sayin',
The problem here is that LO requires one massive builder that can spawn 100 parallel processes to build - whereas we can easier provide 100 VMs that can build 'small pieces in parallel' (usually with up to -j8) If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful. Your suggestion about splitting translations and docs might be worthy to be explored already. Cheers, Dominique
Dominique Leuenberger / DimStar wrote:
If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful.
Tough - I see little way to have that supported upstream, so it likely will be one massive hack. Would icecream/distcc be an option? With that, one can have a smallish builder (might need some extra mem), but running it with -j100 & icecream to fan-out the load to idle builders. Cheers, -- Thorsten
On Wed, 2016-10-12 at 15:40 +0200, Thorsten Behrens wrote:
Dominique Leuenberger / DimStar wrote:
If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful.
Tough - I see little way to have that supported upstream, so it likely will be one massive hack. Would icecream/distcc be an option? With that, one can have a smallish builder (might need some extra mem), but running it with -j100 & icecream to fan-out the load to idle builders.
At least with the current setup of OBS I don't see a way to achieve this. A worker (an independent VM, no network connection) gets a job assigned to perform a build of a package. I don't know enough about the LO build system / source structure to come up with any reasonable suggestion unfortunately. I'm only bitten by the effects it has :) Anothe rissue we see compared to what 'normal devs' see: we always start from a fresh VM, so there is no 'cache' from earlier builds. The entire code base needs to be done all the time (I think currently it takes between 4 and 6 hours, depending on the worker it ends up on) Dominique
On Wednesday 2016-10-12 16:14, Dominique Leuenberger / DimStar wrote:
On Wed, 2016-10-12 at 15:40 +0200, Thorsten Behrens wrote:
Dominique Leuenberger / DimStar wrote:
If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful.
Tough - I see little way to have that supported upstream, so it likely will be one massive hack. Would icecream/distcc be an option? With that, one can have a smallish builder (might need some extra mem), but running it with -j100 & icecream to fan-out the load to idle builders.
Negative. You can think of OBS as a BOINC cluster: distribution is managed not at the user level, but way way higher up. I also would wonder about the reverse scenario - if OBS supported distributing processes, then LO and chromium would perhaps monopolize the entire cluster when their 10000 C++ fileset gets built for 10+ targets. Splitting up huge projects also has the benefit, so I claim, that drive-by contributions may become more plentiful, because you could then reuse prebuilt components from the distro and just link. I don't remember how many CPU hours got wasted just by having to recompile boringssl everytime chromium is triggered. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Engelhardt wrote:
Splitting up huge projects also has the benefit, so I claim, that drive-by contributions may become more plentiful, because you could then reuse prebuilt components from the distro and just link.
That already happens - e.g. with the Document Liberation libs. Didn't check now, but am pretty sure close-to-100% of the LibreOffice 3rd party libs are distro-packaged. There's still some 5M LOC of core code left to build each time ... Cheers, -- Thorsten
On Wednesday 2016-10-12 18:31, Thorsten Behrens wrote:
Jan Engelhardt wrote:
Splitting up huge projects also has the benefit, so I claim, that drive-by contributions may become more plentiful, because you could then reuse prebuilt components from the distro and just link.
That already happens - e.g. with the Document Liberation libs.
Yes, and the DLLs (ha!) are what I consider the ultimate pot of gold. "That's how it's done". I concur it is not easy to find more equally standalone parts in the LO Core. But, here are some other suggestions to reduce build time: - make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time. - drop excessively *small* C++ files, such as bpixel.cxx, where basically _all_ the time is spent JUST for header code. This 1-function-per-c-file (and therefore 1 function per object file) "rule" can be seen in glibc, but let glibc be excused because it makes a "essential" static library. LO does not have that kind of target. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Engelhardt wrote:
- make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time.
Worth a try - needs configuring with --enable-pch. Largely untested upstream though, since ccache delivers much more bang (and is mutually exclusive). _Is_ used for Windows builds upstream, so chances are maintaining a working setup is not too hard.
- drop excessively *small* C++ files, such as bpixel.cxx, where basically _all_ the time is spent JUST for header code.
That's prolly a non-starter upstream - LibreOffice suffers more from the opposite, too large functions / classes, and therefore cxx files. And at the end of the day, an accessible, nicely-structured codebase is what the upstream project will favour. Just to gauge the effects, there's some 200-300 _very_ small cxx files, of >10,000 in total. Likely not worth the hassle. But yeah, c++ is a mess there. And since LibreOffice uses boost pretty much everywhere, we pay a multi-megabyte price for every cxx file we add... Cheers, -- Thorsten
Thorsten Behrens schrieb:
Jan Engelhardt wrote:
- make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time.
Worth a try - needs configuring with --enable-pch. Largely untested upstream though, since ccache delivers much more bang (and is mutually exclusive). _Is_ used for Windows builds upstream, so chances are maintaining a working setup is not too hard.
Maybe using ccache is an option then. How much does it gain? OBS has no support for ccache itself but in theory the spec file could package the ccache cache as rpm subpackage. On next build the package could pull it's own cache from the previous run. Bootstrapping would be an issue but should be solvable via prjconf settings. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Ludwig Nussel wrote:
Maybe using ccache is an option then. How much does it gain?
Hard to predict for realistic cases - if you have enough IO bandwidth, order of magnitude I'd say, for a build that gets close to 100% cache hits. Problem is, when OBS rebuilds, likely some of the dependencies have changed. ;) Cheers, -- Thorsten
Ludwig Nussel píše v Čt 13. 10. 2016 v 09:35 +0200:
Thorsten Behrens schrieb:
Jan Engelhardt wrote:
- make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time.
Worth a try - needs configuring with --enable-pch. Largely untested upstream though, since ccache delivers much more bang (and is mutually exclusive). _Is_ used for Windows builds upstream, so chances are maintaining a working setup is not too hard.
Maybe using ccache is an option then. How much does it gain? OBS has no support for ccache itself but in theory the spec file could package the ccache cache as rpm subpackage. On next build the package could pull it's own cache from the previous run. Bootstrapping would be an issue but should be solvable via prjconf settings.
cu Ludwig
Hello guys, Just few notes about the beast. The build itself is actually pretty fast if we roll in RAM only environment and use multiple threads. BUT we in OBS need to be able to build so I have code that limits threads based on the amount of memory the virtual has. Checking the stats on the SLE12 build it is around 2 to tops 3 hours there... Also the other issue is that ~1 hour is spent splitting the debuginfo and other magic from libraries. This brp fun is an action for the EMU team to handle in following months :) Cheers Tom
On Friday 2016-10-14 12:41, Tomas Chvatal wrote:
Also the other issue is that ~1 hour is spent splitting the debuginfo and other magic from libraries. This brp fun is an action for the EMU team to handle in following months :)
yes, the recap for those who missed that thread: https://lists.opensuse.org/opensuse-factory/2016-05/msg00250.html So, it is only going to take 3 years until it is solved, that is of course, unless the aforementioned team steps up ;) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dominique Leuenberger / DimStar wrote:
Anothe rissue we see compared to what 'normal devs' see: we always start from a fresh VM, so there is no 'cache' from earlier builds. The entire code base needs to be done all the time (I think currently it takes between 4 and 6 hours, depending on the worker it ends up on)
Urk. So for the other exteme - on our CI (32 cores, hot ccache, release build setup), that's usually below 10 minutes (e.g. http://ci.libreoffice.org/job/lo_gerrit/Config=linux_gcc_release_64/1180/). IIWY - since this seems to be so excessively bad - I'd probably setup a dedicated builder, perhaps seed ccache from outside every few days with a fresh build, and pin that one for the worst offenders like FF, Chromium, LibreOffice. We did some splitting up of the sources into different packages in the past, but that turned out to be so against the grain of how the code is structured, it was considered unmaintainable after a while. My 2 cents, -- Thorsten
Hello, Le 12/10/2016 à 18:48, Thorsten Behrens a écrit :
Dominique Leuenberger / DimStar wrote:
Anothe rissue we see compared to what 'normal devs' see: we always start from a fresh VM, so there is no 'cache' from earlier builds. The entire code base needs to be done all the time (I think currently it takes between 4 and 6 hours, depending on the worker it ends up on)
Urk. So for the other exteme - on our CI (32 cores, hot ccache, release build setup), that's usually below 10 minutes (e.g. http://ci.libreoffice.org/job/lo_gerrit/Config=linux_gcc_release_64/1180/).
IIWY - since this seems to be so excessively bad - I'd probably setup a dedicated builder, perhaps seed ccache from outside every few days with a fresh build, and pin that one for the worst offenders like FF, Chromium, LibreOffice.
We did some splitting up of the sources into different packages in the past, but that turned out to be so against the grain of how the code is structured, it was considered unmaintainable after a while.
My 2 cents,
-- Thorsten
I'm also adding my two cents, but could we also use on LO the gold linker to accelerate the build on the openSUSE farm ? For example, openMandriva already did it. Arnaud -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed 2016-10-12 18:48:29, Thorsten Behrens wrote:
We did some splitting up of the sources into different packages in the past, but that turned out to be so against the grain of how the code is structured, it was considered unmaintainable after a while.
Yup, the build had been split into about 10 "independent" source packages many years ago. It took us few months to get it working. And it ended in the rubbish bin because upstream did not like it and it was hard to maintain. Instead upstream converted the build from "dmake" to GNU make. It massively increased possible parallelism. I mean that the build got much faster on machines with many CPUs. By other words, the build was optimized for single powerful machine instead of building pieces on different machines. This might also be a hint for OBS. LibreOffice would need a powerful host with many CPUs. Then we could call make -jN with reasonable big N and profit from the upstream build optimization. Best Regards, Petr -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2016-10-13 13:16, Petr Mladek wrote:
Yup, the build had been split into about 10 "independent" source packages many years ago. It took us few months to get it working. And it ended in the rubbish bin because upstream did not like it and it was hard to maintain.
Well, now that Sun/Oracle is out of the picture and LO is its own upstream, surely that past-time hurdle is gone.
This might also be a hint for OBS. LibreOffice would need a powerful host with many CPUs. Then we could call make -jN with reasonable big N
The thing is, it's not economic. The price for a machine with big N is, for growing N, exponentially higher than the price for M machines with N in summation. (There is a market gap here though, since there seems to be no CPU which is both small like an Atom yet supports sibling packages, so as to create a 1000-CPU Atom-class SGI-style rack computer. You're basically forced to conjoin already-pricey Xeons.) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu 2016-10-13 14:55:04, Jan Engelhardt wrote:
On Thursday 2016-10-13 13:16, Petr Mladek wrote:
Yup, the build had been split into about 10 "independent" source packages many years ago. It took us few months to get it working. And it ended in the rubbish bin because upstream did not like it and it was hard to maintain.
Well, now that Sun/Oracle is out of the picture and LO is its own upstream, surely that past-time hurdle is gone.
I am afraid that this does not help. The split build was discussed with LibreOffice comminity and rejected. Note that it was LibreOffice community who did the conversion to GNU make.
This might also be a hint for OBS. LibreOffice would need a powerful host with many CPUs. Then we could call make -jN with reasonable big N
The thing is, it's not economic. The price for a machine with big N is, for growing N, exponentially higher than the price for M machines with N in summation. (There is a market gap here though, since there seems to be no CPU which is both small like an Atom yet supports sibling packages, so as to create a 1000-CPU Atom-class SGI-style rack computer. You're basically forced to conjoin already-pricey Xeons.)
It sounds reasonable but we probably speak in different orders. To be honest, I did not build LibreOffice last 3 years, so my experience might be outdated. LibreOffice developers used to have workstations with 8-16 cores and buildservice used 4 CPUs when I was lucky. I guess that the BS virtual machines are running on machines with >=16 CPUs these days but I am not sure how much is available for the particular builders. I doubt that 1000 CPUs vs 64 CPUs would make any big difference. There are I/O limits. Also another reply in this thread mentioned 10,000 source files. I am not sure if the build is paralleled enough for 1000 parallel tasks. But it might be interesting to try it on host with 32 or 64 CPUs. Anyway, you would need to find a volunteer who would work on LibreOffice build system. We do not longer have LibreOffice team and Tomas Chvatal has hard times to keep the packages up-to-date and buildable. Also you would need to make it sexy enough so that it is accepted upstream. LO is a really big beast. You would need a lot of resources to maintain any custom solution. Best Regards, Petr -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/12/2016 04:14 PM, Dominique Leuenberger / DimStar wrote:
[...] The entire code base needs to be done all the time (I think currently it takes between 4 and 6 hours, depending on the worker it ends up on)
looking at the latest LO build log [1], it seems that e.g. the pkg-diff.sh alone already eats 2000s (14243s..16143s). Is the output useful anyway? [1] https://build.opensuse.org/build/openSUSE:Factory/standard/x86_64/libreoffic... Have a nice day, Berny -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJYAUFxAAoJEEZQLveWkXGVuC0H/i7WTsTqXkVwprxj8cc5hde2 hutFe+MRGPtFWkGLVlpnKePg9ZVoYKTyW4vhnY5TqZC8b3W9yjUlQMpTQ/y8RmHk TCwjtxoRN78ye+RYnPAJH1gzmtxCcaCh0lygkWmliOjeimX8lOtNn6tODSam8aTi bxaxtaMSI/RsHET+3QOHDYSbor+XxMgnCVKCp+z8gGPDNKnIZs/e9XUv+ECwd64k OsvSu0vMDMmYdNgiTFeHpG440q7OJOZZNlyhnv4/X2kYsVnNRnZ73yprf01zvPhI 7yybrJMF/tFAf8cVf0IhQ1xJHotMbi3ewLMITi/N51zl6OMB5cBc4QaMnkZeR5Y= =Xdsc -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Oct 12, 2016 at 5:58 AM, Ludwig Nussel
Ancor Gonzalez Sosa schrieb:
The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers.
Thanks for taking that into consideration :-) YaST usually doesn't leave a negative impression when waiting for rebuilds though. There are worse components still.
What's really key for rebuild times of the distribution is to take full advantage of the massively parallel build worker installation we have in OBS. This simulation of the rebuild time for example shows that there's no significant improvement from increasing the number of workers from 40 to 100 (rebuild time in practice is even longer): http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)
Shades of blue represent building, scheduled and blocked there. Ideally the bright 'building' line should not have interruptions with enough scheduled packages to pick from.
So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)
Where do these graphs come from?
So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.
The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles. How does the OBS scheduler deal with build cycles? We'd like to fix it the same way in the graph tool. -- Martin Vidner, YaST Team http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
On 13.10.2016 12:45, Martin Vidner wrote:
http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)
Where do these graphs come from?
From within the build service. You can click on the repository to see build cycles and build graphs.
So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.
The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies
Hope you also take substitutes and ignores into account ;)
but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles.
How does the OBS scheduler deal with build cycles?
The simulation is here: https://github.com/openSUSE/open-build-service/blob/master/src/api/vendor/di...
We'd like to fix it the same way in the graph tool.
I'm happy you have time for that. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Oct 13, 2016 at 6:45 AM, Martin Vidner
So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.
The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies
but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles.
How does the OBS scheduler deal with build cycles? We'd like to fix it the same way in the graph tool.
I used it on devel:languages:python3, but that repo has a lot more packages, and the resulting graph is so enormous with so many crossing and overlapping lines I can't make much sense of it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 13 Oct 2016 07:41:35 -0400
Todd Rme
On Thu, Oct 13, 2016 at 6:45 AM, Martin Vidner
wrote:m? So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.
The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies
but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles.
How does the OBS scheduler deal with build cycles? We'd like to fix it the same way in the graph tool.
I used it on devel:languages:python3, but that repo has a lot more packages, and the resulting graph is so enormous with so many crossing and overlapping lines I can't make much sense of it.
Then try second script, that generate yaml with critical path ( critical path is explained in README what it is ). So if you want to speed up, then focus on critical path and on layers which have only few packages, as it is bottle neck. for paralellism as it waits for lower layer and block upper layers. Josef -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
I used it on devel:languages:python3, but that repo has a lot more packages, and the resulting graph is so enormous with so many crossing and overlapping lines I can't make much sense of it.
Have you applied the transitive reduction filter (tred) from graphviz? That trick is quite hidden in our script: https://github.com/mvidner/rpm-build-dependencies/blob/780456852e76e54d5c9e5... -- Martin Vidner, YaST Team http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
On Wed, 12 Oct 2016 10:47:43 -0400
Todd Rme
On Wed, Oct 12, 2016 at 5:58 AM, Ludwig Nussel
wrote: Ancor Gonzalez Sosa schrieb:
The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers.
Thanks for taking that into consideration :-) YaST usually doesn't leave a negative impression when waiting for rebuilds though. There are worse components still.
What's really key for rebuild times of the distribution is to take full advantage of the massively parallel build worker installation we have in OBS. This simulation of the rebuild time for example shows that there's no significant improvement from increasing the number of workers from 40 to 100 (rebuild time in practice is even longer): http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)
Shades of blue represent building, scheduled and blocked there. Ideally the bright 'building' line should not have interruptions with enough scheduled packages to pick from.
So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.
Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.
well, YaST team as part of that work already create such tool. See blog post and especially https://github.com/mvidner/rpm-build-dependencies where is two scirpts 1) dependson_to_graph which create graph of dependencies 2) print_deps.rb which generates yaml file with informations like build time of each package, critical path ( so list of dependencies that took majority of time ) and also number of layers, where layer is set of projects that can be build together, so any layer that have small number of packages are bottle neck. for example output see https://github.com/mvidner/rpm-build-dependencies/blob/master/yast_deps.yaml which is done for YaST:Head and builds there. BTW time -1 means, that package is not building for specified target. So for openSUSE:Factory yaml command will look like this: ruby print_deps.rb openSUSE:Factory standard x86_64 /tmp/factory.yaml But as martin Vidner write above there is a problem with cycles, but in branch "factory_adaptation" I try initial implementation that deal with it and also optimize code. So in around 3 hours you should get such yaml for factory. which contain critical path. Regarding discussion how to speed libraoffice build, it is same as for YaST2: 1) speed up build itself by making it more parallel, etc. 2) remove dependencies that are in one layer lower, so it do not need to wait for it. Josef -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (13)
-
Ancor Gonzalez Sosa
-
Arnaud Versini
-
Bernhard Voelker
-
Dominique Leuenberger / DimStar
-
Jan Engelhardt
-
Josef Reidinger
-
Ludwig Nussel
-
Martin Vidner
-
Petr Mladek
-
Stephan Kulow
-
Thorsten Behrens
-
Todd Rme
-
Tomas Chvatal