[opensuse-packaging] Parallel build
Hi there, I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track? Now, if parallel build should indeed be added, I would have expected it to be handled globally rather than on a per-package basis. Can you please clarify? What problem are you trying to solve, where do we come from and where do we expect to go? Thanks, -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Tue, Dec 08, 2009 at 11:36:13AM +0100, Jean Delvare wrote:
Hi there,
I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track?
I dont think the buildservice uses icecream. Please correct me though.
Now, if parallel build should indeed be added, I would have expected it to be handled globally rather than on a per-package basis.
Can you please clarify? What problem are you trying to solve, where do we come from and where do we expect to go?
Well, you need to tell the .spec files that "make -jsomething" is possible instead of just "make". You cannot just replace make blindly by a automatic parallel version. (Of course the question is really if it is necessary for all packages.) Ciao, Marcus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag, 8. Dezember 2009 12:06:42 schrieb Marcus Meissner:
On Tue, Dec 08, 2009 at 11:36:13AM +0100, Jean Delvare wrote:
Hi there,
I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track?
I dont think the buildservice uses icecream. Please correct me though.
our instances do not, but the backend is prepare to use it. However, this would be security problem for the opensuse.org instance. bye adrian
Now, if parallel build should indeed be added, I would have expected it to be handled globally rather than on a per-package basis.
Can you please clarify? What problem are you trying to solve, where do we come from and where do we expect to go?
Well, you need to tell the .spec files that "make -jsomething" is possible instead of just "make".
You cannot just replace make blindly by a automatic parallel version.
(Of course the question is really if it is necessary for all packages.)
Ciao, Marcus
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Tue, Dec 08, 2009 at 11:36:13AM +0100, Jean Delvare wrote:
Hi there,
I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track?
I dont think the buildservice uses icecream. Please correct me though. It does not, but let's assume it does: icecream can only really help if the
Am Dienstag 08 Dezember 2009 schrieb Marcus Meissner: package builds with parallel jobs, so it needs to be enabled in the package.
Now, if parallel build should indeed be added, I would have expected it to be handled globally rather than on a per-package basis.
Can you please clarify? What problem are you trying to solve, where do we come from and where do we expect to go?
Well, you need to tell the .spec files that "make -jsomething" is possible instead of just "make".
You cannot just replace make blindly by a automatic parallel version.
Actually the fact that we build for a long time without %jobs > 1 led to many packagers adding %jobs and having no good way to test this: and now many packages fail because their build systems are just not up to parallel make ;(
(Of course the question is really if it is necessary for all packages.)
I wonder too. Many of the packages patched compile for very short time and spend their time in doing installs and test suits. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le mardi 8 décembre 2009 13:25, Stephan Kulow a écrit :
Am Dienstag 08 Dezember 2009 schrieb Marcus Meissner:
(Of course the question is really if it is necessary for all packages.)
I wonder too. Many of the packages patched compile for very short time and spend their time in doing installs and test suits.
Well, test suites may be candidates to parallelization too. I am precisely working on this for quilt. The quilt test suite takes 10 times longer than it takes to built quilt. But parallel build works while parallel testing does not. Obviously there's room for improvement. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag 08 Dezember 2009 schrieb Jean Delvare:
Le mardi 8 décembre 2009 13:25, Stephan Kulow a écrit :
Am Dienstag 08 Dezember 2009 schrieb Marcus Meissner:
(Of course the question is really if it is necessary for all packages.)
I wonder too. Many of the packages patched compile for very short time and spend their time in doing installs and test suits.
Well, test suites may be candidates to parallelization too. I am precisely working on this for quilt. The quilt test suite takes 10 times longer than it takes to built quilt. But parallel build works while parallel testing does not. Obviously there's room for improvement.
Yes, but simply adding %jobs won't fix it - and the worst thing is: if it worked once, it can still be broken ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag, 8. Dezember 2009 11:36:13 schrieb Jean Delvare:
Hi there,
I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track?
no, icecream is not used due to security reasons (our xen instances have no network interface on purpose and neither icecream nor gcc can be considered to be safe against an attack). However, most systems build with -j4 on their local cpu's.
Now, if parallel build should indeed be added, I would have expected it to be handled globally rather than on a per-package basis.
A package is using it, if you use the %jobs macro behind your make call.
Can you please clarify? What problem are you trying to solve, where do we come from and where do we expect to go?
Thanks,
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 12/08/2009 01:22 PM, Adrian Schröter wrote:
A package is using it, if you use the %jobs macro behind your make call.
While we are discussing it, we can also discuss the actual macros that we use. See http://rpm.org/wiki/Problems/Distributions - section "parallel make". We are using %{?jobs:-j%jobs} while Fedora uses %{?_smp_mflags}. The latter one also works on openSUSE. What do you think about unification? I also proposed macros for RPM upstream (see the link above and corresponding ticket - http://rpm.org/ticket/115). -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o openSUSE Boosters Team Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le mardi 08 décembre 2009, à 13:38 +0100, Pavol Rusnak a écrit :
On 12/08/2009 01:22 PM, Adrian Schröter wrote:
A package is using it, if you use the %jobs macro behind your make call.
While we are discussing it, we can also discuss the actual macros that we use. See http://rpm.org/wiki/Problems/Distributions - section "parallel make". We are using %{?jobs:-j%jobs} while Fedora uses %{?_smp_mflags}. The latter one also works on openSUSE. What do you think about unification? I also proposed macros for RPM upstream (see the link above and corresponding ticket - http://rpm.org/ticket/115).
I guess it's better to wait for the upstream ticket to get fixed (so we don't have to change things twice). FWIW, I think "%make and %make_parallel" is better than "%make_single and %make", especially for compatibility :-) Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Pavol Rusnak wrote:
On 12/08/2009 01:22 PM, Adrian Schröter wrote:
A package is using it, if you use the %jobs macro behind your make call.
While we are discussing it, we can also discuss the actual macros that we use. See http://rpm.org/wiki/Problems/Distributions - section "parallel make". We are using %{?jobs:-j%jobs} while Fedora uses %{?_smp_mflags}. The latter one also works on openSUSE. What do you think about unification?
Both are useful. I guess %_smp_mflags already includes '-j'. So it would be only useful for 'make' itself while %jobs could be used for other tools as well. Therefore unification would mean supporting both settings IMHO. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (7)
-
Adrian Schröter -
Jean Delvare -
Ludwig Nussel -
Marcus Meissner -
Pavol Rusnak -
Stephan Kulow -
Vincent Untz