[opensuse-packaging] Re: Parallel build
![](https://seccdn.libravatar.org/avatar/dcaea5e8a6fad867936a40a9c21e8ff3.jpg?s=120&d=mm&r=g)
Le mardi 8 décembre 2009 12:28, Jan Engelhardt a écrit :
On Tuesday 2009-12-08 11:36, Jean Delvare wrote:
Hi there,
I see that you are submitting many requests to let openSUSE packages build in parallel. I am a little surprised that it is needed, I thought the build service was powered by icecream which was magically dispatching build jobs as needed. Or am I far off track?
Icecream does not magically mean the use of distcc or any other tricks. Nope, inside a Xen instance which OBS seems to use, it just runs "build", the opensuse builder, with plain gcc and make.
Needless to say that I build some packages outside of OBS (but still with build/lbuild), with a kind of different iron that has lots, but slow CPUs.
OK, thanks for the clarification. I'll accept your request for quilt. While we're here, is there a way for Makefiles to know whether the instance of "make" which is processing them was passed -j > 1? I might need this soon. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/fff0f38e92656c8a636916213eb952c4.jpg?s=120&d=mm&r=g)
Hi, On Tue, 8 Dec 2009, Jean Delvare wrote:
Needless to say that I build some packages outside of OBS (but still with build/lbuild), with a kind of different iron that has lots, but slow CPUs.
OK, thanks for the clarification. I'll accept your request for quilt.
While we're here, is there a way for Makefiles to know whether the instance of "make" which is processing them was passed -j > 1? I might need this soon.
$(MAKEFLAGS) But you should ask yourself why you'd want to detect this. If you feel the need to change behaviour depending on if the make is parallel or not, then this must mean that something else is fishy. Perhaps it's better to fix that fishyness instead of working around it :) Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/dcaea5e8a6fad867936a40a9c21e8ff3.jpg?s=120&d=mm&r=g)
Le mardi 8 décembre 2009 17:04, Michael Matz a écrit :
Hi,
On Tue, 8 Dec 2009, Jean Delvare wrote:
Needless to say that I build some packages outside of OBS (but still with build/lbuild), with a kind of different iron that has lots, but slow CPUs.
OK, thanks for the clarification. I'll accept your request for quilt.
While we're here, is there a way for Makefiles to know whether the instance of "make" which is processing them was passed -j > 1? I might need this soon.
$(MAKEFLAGS)
But you should ask yourself why you'd want to detect this.
I knew someone would tell this ;) Fair enough.
If you feel the need to change behaviour depending on if the make is parallel or not, then this must mean that something else is fishy. Perhaps it's better to fix that fishyness instead of working around it :)
The case at hand is quilt's test suite. It's relatively verbose. When tests are run sequentially it's OK. If they are run in parallel then all outputs will interlace and things will become very difficult to read. So I am looking for ways to run the tests in parallel while still obtaining a usable output if one of the tests fails. And I thought it might help if I could pass a flag to the tester to warn it about parallelism so that it can adjust its output. I'm not too sure how, though. Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout? -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/bc496665c59405794cdb5236dd40ad59.jpg?s=120&d=mm&r=g)
2009/12/8 Jean Delvare <jdelvare@suse.de>:
Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout?
Would perl script, slurping the whole file into memory, and then flush the contents not work? Rob -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/dcaea5e8a6fad867936a40a9c21e8ff3.jpg?s=120&d=mm&r=g)
Le mardi 8 décembre 2009 18:07, Rob OpenSuSE a écrit :
2009/12/8 Jean Delvare <jdelvare@suse.de>:
Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout?
Would perl script, slurping the whole file into memory, and then flush the contents not work?
That I can solve any problem using perl isn't exactly new ;) I was just curious if there was already a unix command doing the same, so that I don't reinvent the wheel. -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/bff0c215e01f23fcee6fe49e65fae458.jpg?s=120&d=mm&r=g)
On Tue, Dec 08, 2009 at 05:07:54PM +0000, Rob OpenSuSE wrote:
2009/12/8 Jean Delvare <jdelvare@suse.de>:
Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout?
Would perl script, slurping the whole file into memory, and then flush the contents not work?
Perhaps dd with a very large blocksize. dd bs=10G Ciao, Marcus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/bc496665c59405794cdb5236dd40ad59.jpg?s=120&d=mm&r=g)
2009/12/8 Marcus Meissner <meissner@suse.de>:
On Tue, Dec 08, 2009 at 05:07:54PM +0000, Rob OpenSuSE wrote:
2009/12/8 Jean Delvare <jdelvare@suse.de>:
Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout?
Would perl script, slurping the whole file into memory, and then flush the contents not work?
Perhaps dd with a very large blocksize.
dd bs=10G
perl -e '$/ = undef; $fin = <STDIN>; print STDOUT $fin' is "obvious", but I got this down to a general filter : perl -p00e "" And you can ensure it reads whole file, by testing at terminal with trad. place holder "-" : ~/bin> perl -p00e "" "date |" - "date |" Tue Dec 8 22:36:01 GMT 2009 1 # typing now 2 3 # going to hit CNTRL-D next line 1 # typing now 2 3 # going to hit CNTRL-D next line Tue Dec 8 22:36:05 GMT 2009 Should only allocate the memory you actually use. Rob -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/bc496665c59405794cdb5236dd40ad59.jpg?s=120&d=mm&r=g)
2009/12/8 Rob OpenSuSE <rob.opensuse.linux@googlemail.com>:
~/bin> perl -p00e "" "date |" - "date |"
Whoops! Actually perl -p0777e "" is right, -p0e is OK if you don't have to handle NULL's, had trouble with the -0777 option until I realised I needed -e "". Just had a play with it, when the bs=10G idea arrived *grin* Rob -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/dcaea5e8a6fad867936a40a9c21e8ff3.jpg?s=120&d=mm&r=g)
Hi Rob, Le mardi 8 décembre 2009 23:58, Rob OpenSuSE a écrit :
2009/12/8 Rob OpenSuSE <rob.opensuse.linux@googlemail.com>:
~/bin> perl -p00e "" "date |" - "date |"
Whoops! Actually perl -p0777e "" is right, -p0e is OK if you don't have to handle NULL's, had trouble with the -0777 option until I realised I needed -e "". Just had a play with it, when the bs=10G idea arrived *grin*
Both Marcus' dd trick and your perl one-lines are interesting. But thinking a little more about my problem, I think I'd rather change my script to output everything at once in the first place. And also to be quieter on success. This should solve the problem without having to go through an external filter. Thanks for all the good ideas anyway, I might need them someday :) -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
![](https://seccdn.libravatar.org/avatar/a6ffef5dde34bf02c36fb9fb70f3e397.jpg?s=120&d=mm&r=g)
On Tue, 8 Dec 2009, Jean Delvare wrote:
Le mardi 8 décembre 2009 17:04, Michael Matz a écrit :
Hi,
On Tue, 8 Dec 2009, Jean Delvare wrote:
<snip> The case at hand is quilt's test suite. It's relatively verbose. When tests are run sequentially it's OK. If they are run in parallel then all outputs will interlace and things will become very difficult to read.
So I am looking for ways to run the tests in parallel while still obtaining a usable output if one of the tests fails. And I thought it might help if I could pass a flag to the tester to warn it about parallelism so that it can adjust its output. I'm not too sure how, though.
Out of curiosity, is there a shell command that would simply buffer every input until EOF and then flush everything to stdout?
Well I've had trouble with something similar, my case was distributed over a whole subnet. Here's what I've used: Each test becomes a unique name, with this name the test creates a temp-logfile does it's job including a final state-line with it's name for easy grepping. After finishing the (sub-)test the temp-logfile is appened as whole to the master-logfile (with file-locking to prevent race-conditions). This way the master-logfile stays easyly readable. In my case it was the easiest, best maintainable solution. An other way would be like syslog with a unique test-identifier + test-pid + output-linenumber (for this test). But without grep, one would be lost, trying to read this. Hope this gives some new views on the problem. And it becomes high time to get all of our asses in gear to make parallel jobs of all kind possible. Not just "Compile" but also "Test" and later "Use" of a program. just try starting a program - with logging option on - more than one time from the same user: in most cases: unreadable logs and/or race-conditions. Not nice for the user without any notification of such trouble in the documentation. Questions? - Try reading a .xsession-errors after a well hidden overflow crash of the window-manager, just to try to find out what happend. Cheers, Michael -- "The word 'politics' is derived from the word 'poly', meaning 'many', and the word 'ticks', meaning 'blood sucking parasites'."
participants (5)
-
Jean Delvare
-
Marcus Meissner
-
Michael Foerster
-
Michael Matz
-
Rob OpenSuSE