[opensuse-packaging] Is there a decent explanation of "%__make %{?jobs:-j%{jobs}}" somewhere?
I'm fine tuning the build of lilypond and they (lilypond devs) recommend building the documentation, which takes a long time, using "make -jx CPU_COUNT=x doc" where x is the cpu core count +1. When building online the "%{?jobs:-j%{jobs}}" macro expands to -j 4 but not being sure of these things, when I use just plain "make -j x" on my box it doesn't speed up the documentation build much whereas if I add "-j 3 CPU_COUNT=3" the build time is reduced from one and a half hours to half an hour approximately. Is there a way of duplicating this on line? Another question, using the openSUSE build script, I've tried to pass the option to rpmbuild "-bi --short-circuit" via RPM_BUILD_STAGE environment option and it gets rejected as invalid either because of the other options or because of the '-bi --short-circuit' '' quotes. If I'm not making a silly mistake when executing "export RPM_BUILD_STAGE=-bi\ --short-circuit" then I'll make an enhancement request. Thanks Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Dave Plater wrote:
I'm fine tuning the build of lilypond and they (lilypond devs) recommend building the documentation, which takes a long time, using "make -jx CPU_COUNT=x doc" where x is the cpu core count +1. When building online the "%{?jobs:-j%{jobs}}" macro expands to -j 4 but not being sure of these things, when I use just plain "make -j x" on my box it doesn't speed up the documentation build much whereas if I add "-j 3 CPU_COUNT=3" the build time is reduced from one and a half hours to half an hour approximately. Is there a way of duplicating this on line?
%jobs is set to the number of parallel compile jobs to use. It's independent of the number of actual cpus as %jobs is set when iceream is used too. That CPU_COUNT thing is a special feature of your Makefile I guess. You better don't use %jobs there as icecream is of no use for building the documentation.
Another question, using the openSUSE build script, I've tried to pass the option to rpmbuild "-bi --short-circuit" via RPM_BUILD_STAGE environment option and it gets rejected as invalid either because of the other options or because of the '-bi --short-circuit' '' quotes. If I'm not making a silly mistake when executing "export RPM_BUILD_STAGE=-bi\ --short-circuit" then I'll make an enhancement request.
--short-circuit won't work anyways I guess as the build script wipes %_topdir in each run. You'd have to chroot into the buildroot and call rpmbuild yourself: # vi $BUILD_ROOT/.build.command # chroot $BUILD_ROOT su -c /.build.command - abuild cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 01/08/2010 10:23 AM, Ludwig Nussel wrote:
Dave Plater wrote:
I'm fine tuning the build of lilypond and they (lilypond devs) recommend building the documentation, which takes a long time, using "make -jx CPU_COUNT=x doc" where x is the cpu core count +1. When building online the "%{?jobs:-j%{jobs}}" macro expands to -j 4 but not being sure of these things, when I use just plain "make -j x" on my box it doesn't speed up the documentation build much whereas if I add "-j 3 CPU_COUNT=3" the build time is reduced from one and a half hours to half an hour approximately. Is there a way of duplicating this on line?
%jobs is set to the number of parallel compile jobs to use. It's independent of the number of actual cpus as %jobs is set when iceream is used too. That CPU_COUNT thing is a special feature of your Makefile I guess. You better don't use %jobs there as icecream is of no use for building the documentation.
I'll play with the CPU_COUNT= option, is there a way of getting the jobs= number to use in the spec file, that you know of?
Another question, using the openSUSE build script, I've tried to pass the option to rpmbuild "-bi --short-circuit" via RPM_BUILD_STAGE environment option and it gets rejected as invalid either because of the other options or because of the '-bi --short-circuit' '' quotes. If I'm not making a silly mistake when executing "export RPM_BUILD_STAGE=-bi\ --short-circuit" then I'll make an enhancement request.
--short-circuit won't work anyways I guess as the build script wipes %_topdir in each run. You'd have to chroot into the buildroot and call rpmbuild yourself: # vi $BUILD_ROOT/.build.command # chroot $BUILD_ROOT su -c /.build.command - abuild
cu Ludwig
This calls for an enhancement request, as build is a good way of simulating osc build to fine tune packages, it picks up things that plain rpmbuild doesn't, in fact I can even play with the script myself and submit it. There is nothing better to debug %install and %file section problems than rpmbuild -bx --short-circuit. Thanks Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 01/08/2010 10:58 AM, Dave Plater wrote:
On 01/08/2010 10:23 AM, Ludwig Nussel wrote:
Dave Plater wrote:
I'm fine tuning the build of lilypond and they (lilypond devs) recommend building the documentation, which takes a long time, using "make -jx CPU_COUNT=x doc" where x is the cpu core count +1. When building online the "%{?jobs:-j%{jobs}}" macro expands to -j 4 but not being sure of these things, when I use just plain "make -j x" on my box it doesn't speed up the documentation build much whereas if I add "-j 3 CPU_COUNT=3" the build time is reduced from one and a half hours to half an hour approximately. Is there a way of duplicating this on line?
%jobs is set to the number of parallel compile jobs to use. It's independent of the number of actual cpus as %jobs is set when iceream is used too. That CPU_COUNT thing is a special feature of your Makefile I guess. You better don't use %jobs there as icecream is of no use for building the documentation.
I'll play with the CPU_COUNT= option, is there a way of getting the jobs= number to use in the spec file, that you know of?
I used "%__make %{?jobs:-j%{jobs}} CPU_COUNT=%{jobs} doc" and it works like a charm. Regards Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le vendredi 8 janvier 2010 10:42, Dave Plater a écrit :
I used "%__make %{?jobs:-j%{jobs}} CPU_COUNT=%{jobs} doc" and it works like a charm.
I'm not sure if %{jobs} is guaranteed to be always defined. I suspect not, so the following would probably be safer: %__make %{?jobs:-j%{jobs} CPU_COUNT=%{jobs}} doc That being said, this CPU_COUNT= thing looks like a ugly hack and the upstream developers should be invited to get rid of it if possible. Their build process shouldn't depend on how many jobs there are. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 01/08/2010 11:52 AM, Jean Delvare wrote:
Le vendredi 8 janvier 2010 10:42, Dave Plater a écrit :
I used "%__make %{?jobs:-j%{jobs}} CPU_COUNT=%{jobs} doc" and it works like a charm.
I'm not sure if %{jobs} is guaranteed to be always defined. I suspect not, so the following would probably be safer:
%__make %{?jobs:-j%{jobs} CPU_COUNT=%{jobs}} doc
That being said, this CPU_COUNT= thing looks like a ugly hack and the upstream developers should be invited to get rid of it if possible. Their build process shouldn't depend on how many jobs there are.
The document build consists mostly of ghostscript and tex commands and these jobs are run simultaneously according to the CPU_COUNT= value. It slows everything else down to a crawl on my box but it cuts the build time in half. Regards Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le vendredi 8 janvier 2010 11:02, Dave Plater a écrit :
On 01/08/2010 11:52 AM, Jean Delvare wrote:
Le vendredi 8 janvier 2010 10:42, Dave Plater a écrit :
I used "%__make %{?jobs:-j%{jobs}} CPU_COUNT=%{jobs} doc" and it works like a charm.
I'm not sure if %{jobs} is guaranteed to be always defined. I suspect not, so the following would probably be safer:
%__make %{?jobs:-j%{jobs} CPU_COUNT=%{jobs}} doc
That being said, this CPU_COUNT= thing looks like a ugly hack and the upstream developers should be invited to get rid of it if possible. Their build process shouldn't depend on how many jobs there are.
The document build consists mostly of ghostscript and tex commands and these jobs are run simultaneously according to the CPU_COUNT= value. It slows everything else down to a crawl on my box but it cuts the build time in half.
Running jobs in parallel is exactly what "make -j" is for. I don't get why they would also need CPU_COUNT=... unless there is a single job which can't be split into individual tasks - but then make -j itself is no longer needed nor desirable. Anyway, as I don't have the time to dig further into this, I won't comment more on this. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 01/08/2010 02:12 PM, Jean Delvare wrote:
Le vendredi 8 janvier 2010 11:02, Dave Plater a écrit :
On 01/08/2010 11:52 AM, Jean Delvare wrote:
Le vendredi 8 janvier 2010 10:42, Dave Plater a écrit :
I used "%__make %{?jobs:-j%{jobs}} CPU_COUNT=%{jobs} doc" and it works like a charm.
I'm not sure if %{jobs} is guaranteed to be always defined. I suspect not, so the following would probably be safer:
%__make %{?jobs:-j%{jobs} CPU_COUNT=%{jobs}} doc
That being said, this CPU_COUNT= thing looks like a ugly hack and the upstream developers should be invited to get rid of it if possible. Their build process shouldn't depend on how many jobs there are.
The document build consists mostly of ghostscript and tex commands and these jobs are run simultaneously according to the CPU_COUNT= value. It slows everything else down to a crawl on my box but it cuts the build time in half.
Running jobs in parallel is exactly what "make -j" is for. I don't get why they would also need CPU_COUNT=... unless there is a single job which can't be split into individual tasks - but then make -j itself is no longer needed nor desirable.
Anyway, as I don't have the time to dig further into this, I won't comment more on this.
All I can give is the excerpt from INSTALL.txt : Known issues and warnings ......................... The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running `lilypond-book' instances, so `-j' `make' option does not significantly speed up the build process. To help speed it up, the makefile variable CPU_COUNT may be set in `local.make' or on the command line to the number of `.ly' files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine make -j3 CPU_COUNT=3 doc The recommended value of CPU_COUNT is one plus the number of cores or processors, but it is advisable to set it to a smaller value if your system has not enough RAM to run that many simultaneous LilyPond instances. Regards Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le vendredi 8 janvier 2010 19:07, Dave Plater a écrit :
All I can give is the excerpt from INSTALL.txt : Known issues and warnings .........................
The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running `lilypond-book' instances, so `-j' `make' option does not significantly speed up the build process. To help speed it up, the makefile variable CPU_COUNT may be set in `local.make' or on the command line to the number of `.ly' files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine
make -j3 CPU_COUNT=3 doc
The recommended value of CPU_COUNT is one plus the number of cores or processors, but it is advisable to set it to a smaller value if your system has not enough RAM to run that many simultaneous LilyPond instances.
"there cannot be several simultaneously running `lilypond-book' instances" This is the key problem, that should be addressed by the developers. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, On Fri, 08 Jan 2010, Jean Delvare wrote:
Le vendredi 8 janvier 2010 19:07, Dave Plater a écrit :
All I can give is the excerpt from INSTALL.txt : Known issues and warnings .........................
The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running `lilypond-book' instances, so `-j' `make' option does not significantly speed up the build process. To help speed it up, the makefile variable CPU_COUNT may be set in `local.make' or on the command line to the number of `.ly' files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine
make -j3 CPU_COUNT=3 doc
The recommended value of CPU_COUNT is one plus the number of cores or processors, but it is advisable to set it to a smaller value if your system has not enough RAM to run that many simultaneous LilyPond instances.
"there cannot be several simultaneously running `lilypond-book' instances"
This is the key problem, that should be addressed by the developers.
lilypond-book has an option to run more that one job, which _is_ set to $CPU_COUNT in the Makefile. One could only parse $(MAKEFLAGS) instead, at least I don't find a make-Variable containing the number of jobs. HTH, -dnh -- "MIME might be okay for rec.clowns.silent, but not here." -- Jake Kesinger -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 01/09/2010 12:40 AM, David Haller wrote:
Hello,
On Fri, 08 Jan 2010, Jean Delvare wrote:
Le vendredi 8 janvier 2010 19:07, Dave Plater a écrit :
All I can give is the excerpt from INSTALL.txt : Known issues and warnings .........................
The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running `lilypond-book' instances, so `-j' `make' option does not significantly speed up the build process. To help speed it up, the makefile variable CPU_COUNT may be set in `local.make' or on the command line to the number of `.ly' files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine
make -j3 CPU_COUNT=3 doc
The recommended value of CPU_COUNT is one plus the number of cores or processors, but it is advisable to set it to a smaller value if your system has not enough RAM to run that many simultaneous LilyPond instances.
"there cannot be several simultaneously running `lilypond-book' instances"
This is the key problem, that should be addressed by the developers.
lilypond-book has an option to run more that one job, which _is_ set to $CPU_COUNT in the Makefile. One could only parse $(MAKEFLAGS) instead, at least I don't find a make-Variable containing the number of jobs.
HTH, -dnh
Just for interest, here's a build log section with four gs simultanious jobs :- Initializing FontConfig... adding font directory: /usr/src/packages/BUILD/lilypond-2.12.3/out/share/lilypond/current/fonts/otf adding font directory: /usr/src/packages/BUILD/lilypond-2.12.3/out/share/lilypond/current/fonts/type1 Building font database. [/usr/src/packages/BUILD/lilypond-2.12.3/out/lybook-db/snippet-names--7471402555679770406.ly] Forking into jobs: (14487 14486 14485 14484) GPL Ghostscript 8.64 (2009-02-03) Copyright (C) 2009 Artifex Software, Inc. All rights reserved. This software comes with NO WARRANTY: see the file COPYING for details. GPL Ghostscript 8.64 (2009-02-03) Copyright (C) 2009 Artifex Software, Inc. All rights reserved. This software comes with NO WARRANTY: see the file COPYING for details. GPL Ghostscript 8.64 (2009-02-03) Copyright (C) 2009 Artifex Software, Inc. All rights reserved. This software comes with NO WARRANTY: see the file COPYING for details. Regards Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le vendredi 8 janvier 2010 23:40, David Haller a écrit :
Hello,
On Fri, 08 Jan 2010, Jean Delvare wrote:
Le vendredi 8 janvier 2010 19:07, Dave Plater a écrit :
All I can give is the excerpt from INSTALL.txt : Known issues and warnings .........................
The most time consuming task for building the documentation is running LilyPond to build images of music, and there cannot be several simultaneously running `lilypond-book' instances, so `-j' `make' option does not significantly speed up the build process. To help speed it up, the makefile variable CPU_COUNT may be set in `local.make' or on the command line to the number of `.ly' files that LilyPond should process simultaneously, e.g. on a bi-processor or dual core machine
make -j3 CPU_COUNT=3 doc
The recommended value of CPU_COUNT is one plus the number of cores or processors, but it is advisable to set it to a smaller value if your system has not enough RAM to run that many simultaneous LilyPond instances.
"there cannot be several simultaneously running `lilypond-book' instances"
This is the key problem, that should be addressed by the developers.
lilypond-book has an option to run more that one job, which _is_ set to $CPU_COUNT in the Makefile.
I could imagine that without even looking at the code. But that's the problem, not the solution. Good build systems are when the work is split into small tasks, so that these tasks can be run in parallel, with "make" handling the dependencies so that things are processed in the right order. Here, lilypond-book is moving the task splitting one level deeper, so "make" is basically useless. The proper fix would be to let several _instances_ of lilypond-book run in parallel, with each instance doing a part of the job and possibly one instance merging the results. Similar to gcc being used to build many small object files and then all object files being linked together.
One could only parse $(MAKEFLAGS) instead, at least I don't find a make-Variable containing the number of jobs.
It is indeed in $(MAKEFLAGS), but I wouldn't recommend using it. If you have to pass $CPU_COUNT to lilypond-book, this means that "make" can't do the parallelization. If it really can't then the -j is useless. If there are other tasks to achieve and "make" _can_ do some parallelization, then the number of tasks will be twice as much as what you asked for (if not worse...) and your build system will go on its knees. For the time being, "make CPU_COUNT=3 doc" (without -j3) seems better. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Dave Plater wrote:
On 01/08/2010 10:23 AM, Ludwig Nussel wrote:
--short-circuit won't work anyways I guess as the build script wipes %_topdir in each run. You'd have to chroot into the buildroot and call rpmbuild yourself: # vi $BUILD_ROOT/.build.command # chroot $BUILD_ROOT su -c /.build.command - abuild
This calls for an enhancement request, as build is a good way of simulating osc build to fine tune packages, it picks up things that plain rpmbuild doesn't, in fact I can even play with the script myself and submit it. There is nothing better to debug %install and %file section problems than rpmbuild -bx --short-circuit.
Sure. The source code of build is at http://gitorious.org/opensuse/build cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (4)
-
Dave Plater -
David Haller -
Jean Delvare -
Ludwig Nussel