Mailinglist Archive: opensuse-factory (602 mails)

< Previous Next >
Re: [opensuse-factory] Can we assume that /bin/sh is bash?


On 11/02/2019 21:48, Jan Engelhardt wrote:

On Monday 2019-02-11 12:07, Michal Suchánek wrote:

Looking slightly more broadly if changing #!/bin/sh to something lighter
lead to a 10-20% reduction in build times for some of our slower
building packages, that is something that would interest my team and I
guess should also interest the obs team and anyone who cares about our
infrastructure usage.

Many packages that build slow spend a lot of time actually *compiling*
stuff.

Yes and no.
Build times follow the usual long-tail distribution (Pareto or something).
Hugeass packages like chromium of course spent 99% their time in g++,
while things like python-* are probably spending most of their time in
/usr/bin/build and /usr/bin/rpm and disk IO. And we don't exactly have
few packages in the {python, perl, ruby} camp, mind you.

Many packages that build slow spend a lot of time in rpm checkers.

But for different reasons.
* The compression time scales linearly with the size of the contents.
* absolute executable size roughly scales linearly with package size
(something in the ballpark of 60% of a package are executables normally;
ignoring noarch packages)
* ELF analysis (dwz et al) scales linearly with executable size
* the size of executable scales linearly with project size


Yeah for many larger packages up to 1/3rd of the build time is spent
after make install is finished, but for larger slower building packages
as mentioned above the issue is not the rpm checkers, for slow, large
packages the rpm checkers only make up a very small amount of the build
time whereas for small quick packages the rpm checkers take a larger
percentage of time which is why we already invested a bit of time in
speeding them up. For large packages the time is mostly spent doing
compression and extracting the debug symbols, this is currently not done
in parallel so we waste huge amounts of CPU time here currently.
Performing this step in parallel by processing multiple subpackages at
once does provide a significant improvement, however given rpmbuild was
never designed to do this in parallel I am yet to be able to do it
reliably without crashes due to race conditions accessing memory or
deadlocks so i'm not actively working on it atm.

Before this thread we hadn't considered looking at whether using a
different #!/bin/sh would have a impact on builds particularly in the
time it takes to execute configure or process makefiles under different
shells, at some point in the next couple of months its now on my list to
look at. Particularly in regards to packages like the kernel,
libreoffice and chromium. If we can get to a point where we are speeding
up these kind of packages by some amount maybe it makes sense to add an
update alternatives setup for #!/bin/sh even if its mostly so we can use
it in the build root of certain packages to speed them up. At the moment
our team is still looking into a bunch of smaller ways to speed up the
build in the hope if we find enough it will eventually get to something
more substantial, its an area people haven't spent a lot of time looking
at so we are still finding some low hanging fruit. From our point of
view we are trying to reduce the time it takes to release security
updates and one of the ways to do that is obviously speeding up build times.

--

Simon Lees (Simotek) http://simotek.net

Emergency Update Team keybase.io/simotek
SUSE Linux Adelaide Australia, UTC+10:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

< Previous Next >