Ludwig Nussel wrote:

And of course libreoffice forms the long tail :-)

Heh - well the code itself can be built massively-parallel (100-fold or so, then the effects taper off). It's possible that help & translations add some extra delays towards the end, but in principle that could be split out to a different builder. :) Just sayin', -- Thorsten

Dominique Leuenberger / DimStar wrote:

If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful.

Tough - I see little way to have that supported upstream, so it likely will be one massive hack. Would icecream/distcc be an option? With that, one can have a smallish builder (might need some extra mem), but running it with -j100 & icecream to fan-out the load to idle builders. Cheers, -- Thorsten

On Wednesday 2016-10-12 16:14, Dominique Leuenberger / DimStar wrote:

On Wed, 2016-10-12 at 15:40 +0200, Thorsten Behrens wrote:

...

Dominique Leuenberger / DimStar wrote:

...

If LO can be split on the build level to a degree where it is not one massive build call, then that would be helpful.

Tough - I see little way to have that supported upstream, so it likely will be one massive hack. Would icecream/distcc be an option? With that, one can have a smallish builder (might need some extra mem), but running it with -j100 & icecream to fan-out the load to idle builders.

Negative. You can think of OBS as a BOINC cluster: distribution is managed not at the user level, but way way higher up. I also would wonder about the reverse scenario - if OBS supported distributing processes, then LO and chromium would perhaps monopolize the entire cluster when their 10000 C++ fileset gets built for 10+ targets. Splitting up huge projects also has the benefit, so I claim, that drive-by contributions may become more plentiful, because you could then reuse prebuilt components from the distro and just link. I don't remember how many CPU hours got wasted just by having to recompile boringssl everytime chromium is triggered. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wednesday 2016-10-12 18:31, Thorsten Behrens wrote:

Jan Engelhardt wrote:

...

Splitting up huge projects also has the benefit, so I claim, that drive-by contributions may become more plentiful, because you could then reuse prebuilt components from the distro and just link.

That already happens - e.g. with the Document Liberation libs.

Yes, and the DLLs (ha!) are what I consider the ultimate pot of gold. "That's how it's done". I concur it is not easy to find more equally standalone parts in the LO Core. But, here are some other suggestions to reduce build time: - make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time. - drop excessively *small* C++ files, such as bpixel.cxx, where basically _all_ the time is spent JUST for header code. This 1-function-per-c-file (and therefore 1 function per object file) "rule" can be seen in glibc, but let glibc be excused because it makes a "essential" static library. LO does not have that kind of target. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Thorsten Behrens schrieb:

Jan Engelhardt wrote:

...

- make use of a precompiled header. In a 60KLOC project, this cuts about 30% of build time.

Worth a try - needs configuring with --enable-pch. Largely untested upstream though, since ccache delivers much more bang (and is mutually exclusive). _Is_ used for Windows builds upstream, so chances are maintaining a working setup is not too hard.

Maybe using ccache is an option then. How much does it gain? OBS has no support for ccache itself but in theory the spec file could package the ccache cache as rpm subpackage. On next build the package could pull it's own cache from the previous run. Bootstrapping would be an issue but should be solvable via prjconf settings. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Ludwig Nussel píše v Čt 13. 10. 2016 v 09:35 +0200:

Thorsten Behrens schrieb:

...

Jan Engelhardt wrote:

...

 - make use of a precompiled header. In a 60KLOC project, this cuts    about 30% of build time.

Worth a try - needs configuring with --enable-pch. Largely untested upstream though, since ccache delivers much more bang (and is mutually exclusive). _Is_ used for Windows builds upstream, so chances are maintaining a working setup is not too hard.

Maybe using ccache is an option then. How much does it gain? OBS has no support for ccache itself but in theory the spec file could package the ccache cache as rpm subpackage. On next build the package could pull it's own cache from the previous run. Bootstrapping would be an issue but should be solvable via prjconf settings.

cu Ludwig

Hello guys, Just few notes about the beast. The build itself is actually pretty fast if we roll in RAM only environment and use multiple threads. BUT we in OBS need to be able to build so I have code that limits threads based on the amount of memory the virtual has. Checking the stats on the SLE12 build it is around 2 to tops 3 hours there... Also the other issue is that ~1 hour is spent splitting the debuginfo and other magic from libraries. This brp fun is an action for the EMU team to handle in following months :) Cheers Tom

Dominique Leuenberger / DimStar wrote:

Anothe rissue we see compared to what 'normal devs' see: we always start from a fresh VM, so there is no 'cache' from earlier builds. The entire code base needs to be done all the time (I think currently it takes between 4 and 6 hours, depending on the worker it ends up on)

Urk. So for the other exteme - on our CI (32 cores, hot ccache, release build setup), that's usually below 10 minutes (e.g. http://ci.libreoffice.org/job/lo_gerrit/Config=linux_gcc_release_64/1180/). IIWY - since this seems to be so excessively bad - I'd probably setup a dedicated builder, perhaps seed ccache from outside every few days with a fresh build, and pin that one for the worst offenders like FF, Chromium, LibreOffice. We did some splitting up of the sources into different packages in the past, but that turned out to be so against the grain of how the code is structured, it was considered unmaintainable after a while. My 2 cents, -- Thorsten

On Wed 2016-10-12 18:48:29, Thorsten Behrens wrote:

We did some splitting up of the sources into different packages in the past, but that turned out to be so against the grain of how the code is structured, it was considered unmaintainable after a while.

Yup, the build had been split into about 10 "independent" source packages many years ago. It took us few months to get it working. And it ended in the rubbish bin because upstream did not like it and it was hard to maintain. Instead upstream converted the build from "dmake" to GNU make. It massively increased possible parallelism. I mean that the build got much faster on machines with many CPUs. By other words, the build was optimized for single powerful machine instead of building pieces on different machines. This might also be a hint for OBS. LibreOffice would need a powerful host with many CPUs. Then we could call make -jN with reasonable big N and profit from the upstream build optimization. Best Regards, Petr -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu 2016-10-13 14:55:04, Jan Engelhardt wrote:

On Thursday 2016-10-13 13:16, Petr Mladek wrote:

...

Yup, the build had been split into about 10 "independent" source packages many years ago. It took us few months to get it working. And it ended in the rubbish bin because upstream did not like it and it was hard to maintain.

Well, now that Sun/Oracle is out of the picture and LO is its own upstream, surely that past-time hurdle is gone.

I am afraid that this does not help. The split build was discussed with LibreOffice comminity and rejected. Note that it was LibreOffice community who did the conversion to GNU make.

...

This might also be a hint for OBS. LibreOffice would need a powerful host with many CPUs. Then we could call make -jN with reasonable big N

The thing is, it's not economic. The price for a machine with big N is, for growing N, exponentially higher than the price for M machines with N in summation. (There is a market gap here though, since there seems to be no CPU which is both small like an Atom yet supports sibling packages, so as to create a 1000-CPU Atom-class SGI-style rack computer. You're basically forced to conjoin already-pricey Xeons.)

It sounds reasonable but we probably speak in different orders. To be honest, I did not build LibreOffice last 3 years, so my experience might be outdated. LibreOffice developers used to have workstations with 8-16 cores and buildservice used 4 CPUs when I was lucky. I guess that the BS virtual machines are running on machines with >=16 CPUs these days but I am not sure how much is available for the particular builders. I doubt that 1000 CPUs vs 64 CPUs would make any big difference. There are I/O limits. Also another reply in this thread mentioned 10,000 source files. I am not sure if the build is paralleled enough for 1000 parallel tasks. But it might be interesting to try it on host with 32 or 64 CPUs. Anyway, you would need to find a volunteer who would work on LibreOffice build system. We do not longer have LibreOffice team and Tomas Chvatal has hard times to keep the packages up-to-date and buildable. Also you would need to make it sexy enough so that it is accepted upstream. LO is a really big beast. You would need a lot of resources to maintain any custom solution. Best Regards, Petr -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, Oct 12, 2016 at 5:58 AM, Ludwig Nussel wrote:

Ancor Gonzalez Sosa schrieb:

...

The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers.

Thanks for taking that into consideration :-) YaST usually doesn't leave a negative impression when waiting for rebuilds though. There are worse components still.

What's really key for rebuild times of the distribution is to take full advantage of the massively parallel build worker installation we have in OBS. This simulation of the rebuild time for example shows that there's no significant improvement from increasing the number of workers from 40 to 100 (rebuild time in practice is even longer): http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)

Shades of blue represent building, scheduled and blocked there. Ideally the bright 'building' line should not have interruptions with enough scheduled packages to pick from.

So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.

Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 13.10.2016 12:45, Martin Vidner wrote:

...

...

http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)

Where do these graphs come from?

From within the build service. You can click on the repository to see build cycles and build graphs.

...

...

So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.

Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.

The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies

Hope you also take substitutes and ignores into account ;)

but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles.

How does the OBS scheduler deal with build cycles?

The simulation is here: https://github.com/openSUSE/open-build-service/blob/master/src/api/vendor/di...

We'd like to fix it the same way in the graph tool.

I'm happy you have time for that. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 13 Oct 2016 07:41:35 -0400 Todd Rme wrote:

On Thu, Oct 13, 2016 at 6:45 AM, Martin Vidner wrote:m?

...

...

...

So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.

Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.

The blog post links to the tool Josef and I made: https://github.com/mvidner/rpm-build-dependencies

but we have so far only run it on the YaST:Head project. If you try to run it on Factory, it will go into an endless loop because it does not deal with build cycles.

How does the OBS scheduler deal with build cycles? We'd like to fix it the same way in the graph tool.

I used it on devel:languages:python3, but that repo has a lot more packages, and the resulting graph is so enormous with so many crossing and overlapping lines I can't make much sense of it.

Then try second script, that generate yaml with critical path ( critical path is explained in README what it is ). So if you want to speed up, then focus on critical path and on layers which have only few packages, as it is bottle neck. for paralellism as it waits for lower layer and block upper layers. Josef -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 12 Oct 2016 10:47:43 -0400 Todd Rme wrote:

On Wed, Oct 12, 2016 at 5:58 AM, Ludwig Nussel wrote:

...

Ancor Gonzalez Sosa schrieb:

...

The YaST team has just published a new blog post: this time about speeding build time of YaST to ease life of release managers.

Thanks for taking that into consideration :-) YaST usually doesn't leave a negative impression when waiting for rebuilds though. There are worse components still.

What's really key for rebuild times of the distribution is to take full advantage of the massively parallel build worker installation we have in OBS. This simulation of the rebuild time for example shows that there's no significant improvement from increasing the number of workers from 40 to 100 (rebuild time in practice is even longer): http://paste.opensuse.org/28399811 (40) http://paste.opensuse.org/72537244 (100)

Shades of blue represent building, scheduled and blocked there. Ideally the bright 'building' line should not have interruptions with enough scheduled packages to pick from.

So in the middle there are some packages that block so many other packages that workers may idle while waiting for those few to build. The long build time of llvm is very visible in the middle but also in the later phase package build dependencies don't allow for enough parallelism.

Is there a tool we can use to figure out which packages those are? I have been trying to figure out how to improve the build time of the python/python3 packages but figuring out where the real bottlenecks are is difficult.

well, YaST team as part of that work already create such tool. See blog post and especially https://github.com/mvidner/rpm-build-dependencies where is two scirpts 1) dependson_to_graph which create graph of dependencies 2) print_deps.rb which generates yaml file with informations like build time of each package, critical path ( so list of dependencies that took majority of time ) and also number of layers, where layer is set of projects that can be build together, so any layer that have small number of packages are bottle neck. for example output see https://github.com/mvidner/rpm-build-dependencies/blob/master/yast_deps.yaml which is done for YaST:Head and builds there. BTW time -1 means, that package is not building for specified target. So for openSUSE:Factory yaml command will look like this: ruby print_deps.rb openSUSE:Factory standard x86_64 /tmp/factory.yaml But as martin Vidner write above there is a problem with cycles, but in branch "factory_adaptation" I try initial implementation that deal with it and also optimize code. So in around 3 hours you should get such yaml for factory. which contain critical path. Regarding discussion how to speed libraoffice build, it is same as for YaST2: 1) speed up build itself by making it more parallel, etc. 2) remove dependencies that are in one layer lower, so it do not need to wait for it. Josef -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org