[opensuse-factory] Replacing -j%jobs with %_smp_mflags
Hi, Most of SUSE specfiles include %{?jobs:-j%jobs} for parallel execution. It is also a well-known fact that quite some time is spent with setting up roots, etc., and compiling projects that can only do -j1. To keep the machine busy, I would increase the number of parallel build instances running. Furthermore, to not let parallel building go out of hand, I would switch from make -jX to make -lX. This would be straightforward to change in .rpmmacros (and/or build's equivalents, respectively): %_smp_mflags -l24 SUSE's specfiles however use a hardcoded %{?jobs:-j%jobs} (historic reasons?) which makes it impossible to easily change to other make flags. I could try touching the $MAKEFLAGS environment variable, but that may as well be reset, ignored, or whatever. In other words, I would prefer -l24 on the command-line. To that end, I am suggesting to change all %{?jobs:-j%jobs} found in specfiles by %_smp_mflags. It should be possible to do this in an automated fashion — projects which are not using -j%jobs in the first place (e.g. because they just are not capable) will not be affected anyway. N.B.: watch out for scons %{?jobs:--jobs=%jobs} which uses --jobs instead of -j. Thoughts? Jan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Sun, Dec 20, 2009 at 06:40:41PM +0100, Jan Engelhardt wrote:
Hi,
Most of SUSE specfiles include %{?jobs:-j%jobs} for parallel execution. It is also a well-known fact that quite some time is spent with setting up roots, etc., and compiling projects that can only do -j1.
To keep the machine busy, I would increase the number of parallel build instances running. Furthermore, to not let parallel building go out of hand, I would switch from make -jX to make -lX.
This would be straightforward to change in .rpmmacros (and/or build's equivalents, respectively):
%_smp_mflags -l24
SUSE's specfiles however use a hardcoded %{?jobs:-j%jobs} (historic reasons?) which makes it impossible to easily change to other make flags. I could try touching the $MAKEFLAGS environment variable, but that may as well be reset, ignored, or whatever. In other words, I would prefer -l24 on the command-line.
To that end, I am suggesting to change all %{?jobs:-j%jobs} found in specfiles by %_smp_mflags. It should be possible to do this in an automated fashion — projects which are not using -j%jobs in the first place (e.g. because they just are not capable) will not be affected anyway.
N.B.: watch out for scons %{?jobs:--jobs=%jobs} which uses --jobs instead of -j.
Thoughts?
What do the Packaging Guidelines say about it? And btw, this is after you submitted 100+ packages already with the old style? :(( It will be quite a time waster to do it again. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Monday 2009-12-21 17:26, Marcus Meissner wrote:
On Sun, Dec 20, 2009 at 06:40:41PM +0100, Jan Engelhardt wrote:
This would be straightforward to change in .rpmmacros (and/or build's equivalents, respectively):
%_smp_mflags -l24
SUSE's specfiles however use a hardcoded %{?jobs:-j%jobs} (historic reasons?) which makes it impossible to easily change to other make flags. To that end, I am suggesting to change all %{?jobs:-j%jobs} found in specfiles by %_smp_mflags. It should be possible to do this in an automated fashion[...] Thoughts?
What do the Packaging Guidelines say about it?
"This text was imported from [[Fedora Guidelines]] and should not be used for openSUSE packaging before review!" "Please follow the [[SUSE Packaging rules]] for now." (which only link back to [[Packaging/Guidelines]]) "Whenever possible, invocations of make should be done as -j%jobs". I have no idea who came up with the idea of imposing a limitation like -j%jobs onto builders.
And btw, this is after you submitted 100+ packages already with the old style? :((
I was only going with what your packaging guidelines set forth. Consider that what I did is marking packages that are "good" for parallel builds.
It will be quite a time waster to do it again.
As I mentioned, I believe that the resubstitution can be automated. Something along the lines of for_all_projects; osc co $i for j in *.spec *.spec.in; do perl -i -pe 's/\%{\?jobs:-j\%jobs}/\%_smp_mflags/g' $j; done; echo "* ..." >>x.changes osc ci done; Now I can either do that, generating a lot of SR noise, or a powerful maintainer (global write permission) -coolo?- directly commits this to the packages without the intermittent SR step. Jan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 12/21/2009 07:57 PM, Jan Engelhardt wrote:
On Monday 2009-12-21 17:26, Marcus Meissner wrote:
On Sun, Dec 20, 2009 at 06:40:41PM +0100, Jan Engelhardt wrote:
This would be straightforward to change in .rpmmacros (and/or build's equivalents, respectively):
%_smp_mflags -l24
SUSE's specfiles however use a hardcoded %{?jobs:-j%jobs} (historic reasons?) which makes it impossible to easily change to other make flags. To that end, I am suggesting to change all %{?jobs:-j%jobs} found in specfiles by %_smp_mflags. It should be possible to do this in an automated fashion[...] Thoughts?
What do the Packaging Guidelines say about it?
"This text was imported from [[Fedora Guidelines]] and should not be used for openSUSE packaging before review!"
"Please follow the [[SUSE Packaging rules]] for now." (which only link back to [[Packaging/Guidelines]])
"Whenever possible, invocations of make should be done as -j%jobs".
I have no idea who came up with the idea of imposing a limitation like -j%jobs onto builders.
The part was indeed rewritten from Fedora's policy. They had make %{?_smp_mflags} I changed it into make %{?jobs:-j%jobs} to match our _OLD_ policy. The wording does not mean that you can't use anything else than -jXX, but that we should try to use parallel build. We can change the way we do it if we find a better solution. See [1] - section "parallel make". [1] http://rpm.org/wiki/Problems/Distributions
Now I can either do that, generating a lot of SR noise, or a powerful maintainer (global write permission) -coolo?- directly commits this to the packages without the intermittent SR step.
This is just a technical issue, we'll sort it out somehow. The bigger problem is the `political` part. -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o openSUSE Boosters Team Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
2009/12/21 Jan Engelhardt <jengelh@medozas.de>:
As I mentioned, I believe that the resubstitution can be automated. Something along the lines of
for_all_projects; osc co $i for j in *.spec *.spec.in; do perl -i -pe 's/\%{\?jobs:-j\%jobs}/\%_smp_mflags/g' $j; done; echo "* ..." >>x.changes osc ci done;
Now I can either do that, generating a lot of SR noise, or a powerful maintainer (global write permission) -coolo?- directly commits this to the packages without the intermittent SR step.
Before you do a global text substitution, why not put a small hack into make(1) so that any -j or -l invocation, is actually using an option configured by the servers sysadmin. In fact, once the package is parallel build capable, surely it's the sysadmin of the machine, not the package maintainers who will know what option makes sense. Rob -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Tuesday 2009-12-22 14:32, Rob OpenSuSE wrote:
for_all_projects; osc co $i for j in *.spec *.spec.in; do perl -i -pe 's/\%{\?jobs:-j\%jobs}/\%_smp_mflags/g' $j; done; echo "* ..." >>x.changes osc ci done;
Now I can either do that, generating a lot of SR noise, or a powerful maintainer (global write permission) -coolo?- directly commits this to the packages without the intermittent SR step.
Before you do a global text substitution, why not put a small hack into make(1) so that any -j or -l invocation, is actually using an option configured by the servers sysadmin.
Of course I considered that. What you see on the mailing list is just the longer-term solution.
In fact, once the package is parallel build capable, surely it's the sysadmin of the machine, not the package maintainers who will know what option makes sense.
Which is why rpm.org ticket 115 proposes %make which is supposed to do just that. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
participants (4)
-
Jan Engelhardt -
Marcus Meissner -
Pavol Rusnak -
Rob OpenSuSE