[opensuse-buildservice] OBS 101: How to threat packages with multiple spec files
Hi guys, if you have an OBS package containing multiple spec files, you may have discovered that OBS only builds the spec file matching the OBS package name. If you want to have the other(s) spec file(s) built, you should use a link, don't use copypac! For example, devel:languages:python / python-nose is a OBS package containing four (!) spec files. In this case, documentation building is separate because doc building dependencies (python-Sphinx) would create a build cycle. A second set of packages are Python3-related, because d:l:p builds both Python and Python3 at the moment. So we end up with the following list of spec files [1]: python-nose/python-nose.spec python-nose/python-nose-doc.spec python-nose/python3-nose.spec python-nose/python3-nose-doc.spec As you can see here [2], only python-nose.spec is build, so we have to do the following (on a command line near you, given you have the rights to do it in the project): $ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some: - To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec) - When building the same thing against a different set of (build) requirements, like: + Different $DYNAMIC_LANGUAGE interpreter versions (usually $INTERPRETER-$PACKAGE.spec) Footnotes: [1] https://build.opensuse.org/package/show?package=python-nose&project=devel%3A... [2] https://build.opensuse.org/package/binaries?package=python-nose&project=deve... -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
Hi, On 2013-01-10 10:22:52 +0100, Sascha Peilicke wrote: <SNIP>
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some:
- To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec)
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance: - the package is pushed to the repo even if the testsuite fails - the package and package-doc might be inconsistent (if the package is pushed to the repo and the doc building fails) IMHO it's probably better to comment out the long running parts (or define some macros for them) when doing local builds.
- When building the same thing against a different set of (build) requirements, like: + Different $DYNAMIC_LANGUAGE interpreter versions (usually $INTERPRETER-$PACKAGE.spec)
Thanks for the pointers! Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Jan 10, 2013 1:42 PM, "Marcus Hüwe" <suse-tux@gmx.de> wrote:
Hi,
On 2013-01-10 10:22:52 +0100, Sascha Peilicke wrote:
<SNIP>
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some:
- To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec)
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance: - the package is pushed to the repo even if the testsuite fails - the package and package-doc might be inconsistent (if the package is pushed to the repo and the doc building fails)
IMHO it's probably better to comment out the long running parts (or define some macros for them) when doing local builds.
There is another common reason for splitting packages, especially doc packages. If a package is used as a dependency for a large number of other packages, splitting the doc package allows the build of the other packages to start much earlier since they don't have to wait for the documentation to finish. This is especially true when the documentation is dynamically generated API documentation and thus takes a long time to build. This is the case with this example, where nose is used as a post-install check for a massive I number of python packages, and many of those that don't use it depend on ones that do. Further, as with most python packages the API documentation is dynamically generated. Trying to build the documentation would be a bottleneck for almost the entire repo, as well as a big chunk of factory. There are several other widely-used python packages that are built the same way for the same reason. And this is by no means unique to python, I see it in other repos where critical components are in the same tarball as not-so-critical ones. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
* Marcus Hüwe (suse-tux@gmx.de) [20130110 13:42]:
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance:
There are some packages like coreutils where the testsuite runs for quite some time and it's a basic package, so it slows down building quite a lot. For such packages you have no choice but to split off the test suite. yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package. Philipp -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Jan 10, 13 20:03:29 +0100, Philipp Thomas wrote:
yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package.
How about this: Make the test suite package contain a logfile of the tests. Require (not BuildRequire) these testresults with exact version and release numbers. Now the testsuite has all the time between building the main package and installing the main package. and if you really want to experiment with a newly built package immediatly, you can instruct zypper to ignore the missing dependency. cheers, JW- -- o \ Juergen Weigert paint it green! __/ _=======.=======_ <V> | jw@suse.de back to ascii! __/ _---|____________\/ \ | 0911 74053-508 say #263A!__/ (____/ /\ (/) | _____________________________/ _/ \_ vim:set sw=2 wm=8 SUSE LINUX Products GmbH, GF: Jeff Hawn, J.Guild, F.Imendoerffer, HRB 16746 (AG Nuernberg), Maxfeldstrasse 5, 90409 Nuernberg, Germany SuSE. Supporting Linux since 1992. ☺ -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Dnia piątek, 11 stycznia 2013 10:30:46 Juergen Weigert pisze:
On Jan 10, 13 20:03:29 +0100, Philipp Thomas wrote:
yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package.
How about this: Make the test suite package contain a logfile of the tests. Require (not BuildRequire) these testresults with exact version and release numbers.
That would also pull the test suite log file to all machines that have the package installed. Do we want that? Chris -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Sascha Peilicke <saschpe@suse.de> writes:
As you can see here [2], only python-nose.spec is build, so we have to do the following (on a command line near you, given you have the rights to do it in the project):
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc
You should add "-C copy" to get matching release numbers. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels... Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2013-01-15 01:54, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels...
Don't "threat" packages with multiple spec files, and they might do that ;-) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 01/15/2013 01:54 AM, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels... You can still do "osc co -u" to checkout the unexpanded sources (i.e. just the _link file). However, due to the generic (and hugely overused) nature of the link feature, it's not always easy to find out programmatically what the packager is about to do with a package containing links.
If it's a link to a distribution (like Factory), you would want to work on the sources or branch from it (and don't care). If you have multiple spec files to work on you have to use the above cludge (and it's sort of an abuse of the link feature actually). But you almost always want the expanded sources. So I'd say the current situation is good enough, but you could do a feature request for the OBS to thread several spec files in one OBS package better ;-) -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
Quoting Sascha Peilicke <saschpe@suse.de>:
On 01/15/2013 01:54 AM, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels... You can still do "osc co -u" to checkout the unexpanded sources (i.e. just the _link file). However, due to the generic (and hugely overused) nature of the link feature, it's not always easy to find out programmatically what the packager is about to do with a package containing links.
Actually, in above example (with multiple spec files in one package) I usually ONLY care for the 'main' one (where all others link to). That one being checked out and not the _link'ed ones is sufficient... all modifications have to happen in this package anyway (not the linked ones). And to do local builds, I just do osc build <which_spec_file_to_build.spec> so, not checking out the kernel 10 times for each branch would be a big bandwidth and time saver :) Dominique -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 01/15/2013 10:41 AM, Dominique Leuenberger a.k.a. Dimstar wrote:
Quoting Sascha Peilicke <saschpe@suse.de>:
On 01/15/2013 01:54 AM, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels... You can still do "osc co -u" to checkout the unexpanded sources (i.e. just the _link file). However, due to the generic (and hugely overused) nature of the link feature, it's not always easy to find out programmatically what the packager is about to do with a package containing links.
Actually, in above example (with multiple spec files in one package) I usually ONLY care for the 'main' one (where all others link to). That one being checked out and not the _link'ed ones is sufficient... all modifications have to happen in this package anyway (not the linked ones).
And to do local builds, I just do osc build <which_spec_file_to_build.spec>
so, not checking out the kernel 10 times for each branch would be a big bandwidth and time saver :)
Sure, I totally agree. So the magic that would have to go into "osc up" is "If _link points to same project than and the OBS package that is currently looked at corresponds to a spec file in the _link'ed OBS package, do an unexpanded checkout else do expanded". Doesn't sound too hard, no? -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
Am Dienstag, 15. Januar 2013, 11:11:36 schrieb Sascha Peilicke:
On 01/15/2013 10:41 AM, Dominique Leuenberger a.k.a. Dimstar wrote:
Quoting Sascha Peilicke <saschpe@suse.de>:
On 01/15/2013 01:54 AM, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \
devel:languages:python python-nose-doc
$ osc linkpac devel:languages:python python-nose \
devel:languages:python python3-nose
$ osc linkpac devel:languages:python python-nose \
devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels...
You can still do "osc co -u" to checkout the unexpanded sources (i.e. just the _link file). However, due to the generic (and hugely overused) nature of the link feature, it's not always easy to find out programmatically what the packager is about to do with a package containing links.
Actually, in above example (with multiple spec files in one package) I usually ONLY care for the 'main' one (where all others link to). That one being checked out and not the _link'ed ones is sufficient... all modifications have to happen in this package anyway (not the linked ones).
But in the light of huge repos, which one is the base and which are the linked ones. Only a logic, like outlined from Sascha below will make this manageable.
And to do local builds, I just do osc build <which_spec_file_to_build.spec>
so, not checking out the kernel 10 times for each branch would be a big bandwidth and time saver :)
Sure, I totally agree. So the magic that would have to go into "osc up" is "If _link points to same project than and the OBS package that is currently looked at corresponds to a spec file in the _link'ed OBS package, do an unexpanded checkout else do expanded".
Doesn't sound too hard, no?
Something along these lines, I had in mind ;-) If a "to be checked out" package links to an existing local repo, simply create a symlink. If that linked package does not exist, I would prefer to have that one checked out, along with the additional symlink. The hard part will be to handle the inherited waste efficiently, but I bet, this logic will show up in the download statistics, once done and used. Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello, On Jan 15 11:11 Sascha Peilicke wrote (excerpt):
So the magic that would have to go into "osc up" is "If _link points to same project than and the OBS package that is currently looked at corresponds to a spec file in the _link'ed OBS package, do an unexpanded checkout else do expanded".
Because the issue becomes more and more complicated, it seems the root cause should be solved instead of trying to make an oversophosticated workaround via link packages. I would prefer if the openSUSE build service software "just works" with source packages that contain multiple spec files. Preferably even spec files with arbitrary names that create binary packages with arbitrary names. Non matching arbitrary names are of course not a good idea in general (cause any kind of confusion) but the openSUSE build service software should not have issues with it. Otherwise every new contributor who creates a source package with multiple spec files gets all those issues again and again. Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH -- Maxfeldstrasse 5 -- 90409 Nuernberg -- Germany HRB 16746 (AG Nuernberg) GF: Jeff Hawn, Jennifer Guild, Felix Imendoerffer -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Dienstag, 15. Januar 2013, 11:59:10 schrieb Johannes Meixner:
Hello,
On Jan 15 11:11 Sascha Peilicke wrote (excerpt):
So the magic that would have to go into "osc up" is "If _link points to same project than and the OBS package that is currently looked at corresponds to a spec file in the _link'ed OBS package, do an unexpanded checkout else do expanded".
Because the issue becomes more and more complicated, it seems the root cause should be solved instead of trying to make an oversophosticated workaround via link packages.
I would prefer if the openSUSE build service software "just works" with source packages that contain multiple spec files.
Preferably even spec files with arbitrary names that create binary packages with arbitrary names. Non matching arbitrary names are of course not a good idea in general (cause any kind of confusion) but the openSUSE build service software should not have issues with it.
Otherwise every new contributor who creates a source package with multiple spec files gets all those issues again and again.
This is planned as an option for the future. However, we will not be enable it for current distros, because it changes the result of their sources. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hans-Peter Jansen <hpj@urpla.net> writes:
If only osc co would deal with those packages correctly (e.g. by using a symlink)
You can get the symlink with "osc co -u". Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (13)
-
Adrian Schröter -
Andreas Schwab -
Andreas Schwab -
Dominique Leuenberger a.k.a. Dimstar -
Hans-Peter Jansen -
Jan Engelhardt -
Johannes Meixner -
Juergen Weigert -
Krzysztof Żelechowski -
Marcus Hüwe -
Philipp Thomas -
Sascha Peilicke -
todd rme