[opensuse-packaging] OBS 101: How to threat packages with multiple spec files
Hi guys, if you have an OBS package containing multiple spec files, you may have discovered that OBS only builds the spec file matching the OBS package name. If you want to have the other(s) spec file(s) built, you should use a link, don't use copypac! For example, devel:languages:python / python-nose is a OBS package containing four (!) spec files. In this case, documentation building is separate because doc building dependencies (python-Sphinx) would create a build cycle. A second set of packages are Python3-related, because d:l:p builds both Python and Python3 at the moment. So we end up with the following list of spec files [1]: python-nose/python-nose.spec python-nose/python-nose-doc.spec python-nose/python3-nose.spec python-nose/python3-nose-doc.spec As you can see here [2], only python-nose.spec is build, so we have to do the following (on a command line near you, given you have the rights to do it in the project): $ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some: - To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec) - When building the same thing against a different set of (build) requirements, like: + Different $DYNAMIC_LANGUAGE interpreter versions (usually $INTERPRETER-$PACKAGE.spec) Footnotes: [1] https://build.opensuse.org/package/show?package=python-nose&project=devel%3Alanguages%3Apython [2] https://build.opensuse.org/package/binaries?package=python-nose&project=devel%3Alanguages%3Apython&repository=openSUSE_Factory -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
Le jeudi 10 janvier 2013, à 10:22 +0100, Sascha Peilicke a écrit :
For example, devel:languages:python / python-nose is a OBS package containing four (!) spec files. In this case, documentation building is separate because doc building dependencies (python-Sphinx) would create a build cycle. A second set of packages are Python3-related, because d:l:p builds both Python and Python3 at the moment. So we end up with the following list of spec files [1]:
python-nose/python-nose.spec python-nose/python-nose-doc.spec python-nose/python3-nose.spec python-nose/python3-nose-doc.spec
As you can see here [2], only python-nose.spec is build, so we have to do the following (on a command line near you, given you have the rights to do it in the project):
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links.
One thing that helps people to only update the right package is to set devel packages: osc changedevelrequest devel:languages:python python-nose-doc \ devel:languages:python python-nose osc changedevelrequest devel:languages:python python3-nose \ devel:languages:python python-nose osc changedevelrequest devel:languages:python python3-nose-doc \ devel:languages:python python3-nose This helps people branching from python-nose-doc (or python3-nose/python3-nose-doc) so they branch from the right place. Cheers, Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On 2013-01-10 10:22:52 +0100, Sascha Peilicke wrote: <SNIP>
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some:
- To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec)
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance: - the package is pushed to the repo even if the testsuite fails - the package and package-doc might be inconsistent (if the package is pushed to the repo and the doc building fails) IMHO it's probably better to comment out the long running parts (or define some macros for them) when doing local builds.
- When building the same thing against a different set of (build) requirements, like: + Different $DYNAMIC_LANGUAGE interpreter versions (usually $INTERPRETER-$PACKAGE.spec)
Thanks for the pointers! Marcus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 01/10/2013 02:41 PM, Marcus Hüwe wrote:
Hi,
On 2013-01-10 10:22:52 +0100, Sascha Peilicke wrote:
<SNIP>
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some:
- To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec)
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance: - the package is pushed to the repo even if the testsuite fails Agreed, that could happen. Still better than not running the testsuite at all :-)
- the package and package-doc might be inconsistent (if the package is pushed to the repo and the doc building fails) I agree, but a failed doc build is usually less of an issue than if the complete package build fails. However, if you want a good example where it makes sense to split, check the build requirements of
https://build.opensuse.org/package/view_file?expand=1&file=openstack-nova.spec&package=openstack-nova&project=Cloud%3AOpenStack%3AMaster versus https://build.opensuse.org/package/view_file?expand=1&file=openstack-nova-doc.spec&package=openstack-nova&project=Cloud%3AOpenStack%3AMaster
IMHO it's probably better to comment out the long running parts (or define some macros for them) when doing local builds.
Running the testsuite only locally doesn't guarantee it's working on the build host either.
- When building the same thing against a different set of (build) requirements, like: + Different $DYNAMIC_LANGUAGE interpreter versions (usually $INTERPRETER-$PACKAGE.spec)
Thanks for the pointers!
Marcus
-- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
On Jan 10, 2013 1:42 PM, "Marcus Hüwe" <suse-tux@gmx.de> wrote:
Hi,
On 2013-01-10 10:22:52 +0100, Sascha Peilicke wrote:
<SNIP>
Even though you end up with four OBS packages, you only have to modify / fix "python-nose" due to the links. This is much better than using copypac (as I've seen recently). Of course you should only split up into several spec files if there's a very good reason for the extra work. Here are some:
- To avoid build cycles - To off-load looong-running parts of a package build, like: + Running a testsuite ($PACKAGE-testsuite.spec) + Building documentation ($PACKAGE-doc.spec)
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance: - the package is pushed to the repo even if the testsuite fails - the package and package-doc might be inconsistent (if the package is pushed to the repo and the doc building fails)
IMHO it's probably better to comment out the long running parts (or define some macros for them) when doing local builds.
There is another common reason for splitting packages, especially doc packages. If a package is used as a dependency for a large number of other packages, splitting the doc package allows the build of the other packages to start much earlier since they don't have to wait for the documentation to finish. This is especially true when the documentation is dynamically generated API documentation and thus takes a long time to build. This is the case with this example, where nose is used as a post-install check for a massive I number of python packages, and many of those that don't use it depend on ones that do. Further, as with most python packages the API documentation is dynamically generated. Trying to build the documentation would be a bottleneck for almost the entire repo, as well as a big chunk of factory. There are several other widely-used python packages that are built the same way for the same reason. And this is by no means unique to python, I see it in other repos where critical components are in the same tarball as not-so-critical ones. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
* Marcus Hüwe (suse-tux@gmx.de) [20130110 13:42]:
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance:
There are some packages like coreutils where the testsuite runs for quite some time and it's a basic package, so it slows down building quite a lot. For such packages you have no choice but to split off the test suite. yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package. Philipp -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Jan 10, 13 20:03:29 +0100, Philipp Thomas wrote:
yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package.
How about this: Make the test suite package contain a logfile of the tests. Require (not BuildRequire) these testresults with exact version and release numbers. Now the testsuite has all the time between building the main package and installing the main package. and if you really want to experiment with a newly built package immediatly, you can instruct zypper to ignore the missing dependency. cheers, JW- -- o \ Juergen Weigert paint it green! __/ _=======.=======_ <V> | jw@suse.de back to ascii! __/ _---|____________\/ \ | 0911 74053-508 say #263A!__/ (____/ /\ (/) | _____________________________/ _/ \_ vim:set sw=2 wm=8 SUSE LINUX Products GmbH, GF: Jeff Hawn, J.Guild, F.Imendoerffer, HRB 16746 (AG Nuernberg), Maxfeldstrasse 5, 90409 Nuernberg, Germany SuSE. Supporting Linux since 1992. ☺ -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Dnia piątek, 11 stycznia 2013 10:30:46 Juergen Weigert pisze:
On Jan 10, 13 20:03:29 +0100, Philipp Thomas wrote:
yWhat is still missing in the OBS is the ability to make have the main package depend on the test suite so that a failing test suite blocks the main package.
How about this: Make the test suite package contain a logfile of the tests. Require (not BuildRequire) these testresults with exact version and release numbers.
That would also pull the test suite log file to all machines that have the package installed. Do we want that? Chris -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am 10.01.2013 20:03, schrieb Philipp Thomas:
* Marcus Hüwe (suse-tux@gmx.de) [20130110 13:42]:
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance:
There are some packages like coreutils where the testsuite runs for quite some time and it's a basic package, so it slows down building quite a lot. For such packages you have no choice but to split off the test suite.
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that? -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tue, Jan 15, 2013 at 11:23 AM, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Am 10.01.2013 20:03, schrieb Philipp Thomas:
* Marcus Hüwe (suse-tux@gmx.de) [20130110 13:42]:
Hmm I don't think it's a good idea to move the run of a testsuite or the doc building to different spec files because it might decrease the "quality" of a package. For instance:
There are some packages like coreutils where the testsuite runs for quite some time and it's a basic package, so it slows down building quite a lot. For such packages you have no choice but to split off the test suite.
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that?
As long as it isn't published, I don't see why that is a major problem. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am 16.01.2013 22:36, schrieb todd rme:
On Tue, Jan 15, 2013 at 11:23 AM, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that?
As long as it isn't published, I don't see why that is a major problem.
Only the testsuite package will not be published, the main package will be AFAIU. -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Wed, Jan 16, 2013 at 4:47 PM, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Am 16.01.2013 22:36, schrieb todd rme:
On Tue, Jan 15, 2013 at 11:23 AM, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that?
As long as it isn't published, I don't see why that is a major problem.
Only the testsuite package will not be published, the main package will be AFAIU.
If you let it run automatically, yes. But for something critical like coreutils I would think that publishing would be disabled until it can be confirmed that everything went smoothly. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello, Am Mittwoch, 16. Januar 2013 schrieb Stefan Seyfried:
Am 16.01.2013 22:36, schrieb todd rme:
On Tue, Jan 15, 2013 at 11:23 AM, Stefan Seyfried
<stefan.seyfried@googlemail.com> wrote:
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that?
As long as it isn't published, I don't see why that is a major problem. Only the testsuite package will not be published, the main package will be AFAIU.
Could this be a starting point? My proposal would be (untested): - add to coreutils-testsuite: Provides: coreutils-tested = %(rpm -q --qf '%{version}-%{release}' coreutils) - add to coreutils: Requires: coreutils-tested = %{version}-%{release} (Requires, not BuildRequires!) Can/will this work, or does it create a circular loop in the post-build checks? As an alternative, change the BuildRequires in lots of packages from coreutils to coreutils-testsuite (maybe using "substitute" in OBS). I know it's not really correct[tm], but I'm quite sure it would work ;-) Gruß Christian Boltz -- <suseROCKs> I hate email <suseROCKs> I hate internet <suseROCKs> I hate computers <suseROCKs> why can't we go back to old fashioned smoke signaling?? [from #opensuse-project] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Jan 17, 2013 at 12:30 PM, Christian Boltz <cb@cboltz.de> wrote:
Hello,
Am Mittwoch, 16. Januar 2013 schrieb Stefan Seyfried:
Am 16.01.2013 22:36, schrieb todd rme:
On Tue, Jan 15, 2013 at 11:23 AM, Stefan Seyfried
<stefan.seyfried@googlemail.com> wrote:
Well, but if the testsuite is split off, a broken (testsuite-failing) version of coreutils might be used to build all the dependencies. Do we want that?
As long as it isn't published, I don't see why that is a major problem. Only the testsuite package will not be published, the main package will be AFAIU.
As an alternative, change the BuildRequires in lots of packages from coreutils to coreutils-testsuite (maybe using "substitute" in OBS). I know it's not really correct[tm], but I'm quite sure it would work ;-)
Gruß
Christian Boltz
The whole point of building the test separately is so other packages will not need to wait for the testsuite before building. Personally I see two approaches: 1. If you are modifying a package with a separate testsuite spec file, disable publishing until you can confirm the testsuite passed. 2. Add a new repository configuration option that allows project maintainers to specify a package as a "blocker" that will prevent the repository from publishing if it doesn't build successfully (I am not sure what this would be called). -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
* todd rme (toddrme2178@gmail.com) [20130117 13:54]:
1. If you are modifying a package with a separate testsuite spec file, disable publishing until you can confirm the testsuite passed.
No use. The testsuite can fail because of changes in other packages like glibc so disabling publishing would have to be an automatic thing in order to be usefull.
2. Add a new repository configuration option that allows project maintainers to specify a package as a "blocker" that will prevent the repository from publishing if it doesn't build successfully (I am not sure what this would be called).
IMO the only viable way if such an approach is used. Philipp -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Sascha Peilicke <saschpe@suse.de> writes:
As you can see here [2], only python-nose.spec is build, so we have to do the following (on a command line near you, given you have the rights to do it in the project):
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc
You should add "-C copy" to get matching release numbers. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels... Pete -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tuesday 2013-01-15 01:54, Hans-Peter Jansen wrote:
Am Donnerstag, 10. Januar 2013, 10:22:52 schrieb Sascha Peilicke:
$ osc linkpac devel:languages:python python-nose \ devel:languages:python python-nose-doc $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose $ osc linkpac devel:languages:python python-nose \ devel:languages:python python3-nose-doc
If only osc co would deal with those packages correctly (e.g. by using a symlink) and not checking out the same package 4 times. This is especially true for things like kernels...
Don't "threat" packages with multiple spec files, and they might do that ;-) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hans-Peter Jansen <hpj@urpla.net> writes:
If only osc co would deal with those packages correctly (e.g. by using a symlink)
You can get the symlink with "osc co -u". Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (13)
-
Andreas Schwab -
Andreas Schwab -
Christian Boltz -
Hans-Peter Jansen -
Jan Engelhardt -
Juergen Weigert -
Krzysztof Żelechowski -
Marcus Hüwe -
Philipp Thomas -
Sascha Peilicke -
Stefan Seyfried -
todd rme -
Vincent Untz