RFC: Factory review policy change regarding multi-specfile packages
Hi all, As part of the setup of ALP products into build service, we would like to change Factory review policy regarding multi-specfile package: Currently, packages containing multiple specfiles require creating links to another package, with the name of secondary specfiles. For instance: on Build service, package foo contains: -foo.spec -bar.spec Only foo.spec will be built. For bar.spec to be built, a linkpac is needed from foo => bar. Creation of this link is handled by OBS when accepting a sr. With ALP, our goal is to migrate source management to git backend in the (hopefully) near future (there won't be any immediate change on how to do package submissions, don't worry ;). To do that and after discussing with Build Service project managers, it would be easier (and cleaner) to add to packages with multiple specfiles a _multibuild file to handle additional specfile, without the need of linkpac. In the example above, package foo would contains: - foo.spec - bar.spec - _multibuild _multibuild content: <multibuild> <flavor>bar.spec</flavor> </multibuild> (there is no need to mention foo.spec). If there is an agreement, all multi-spec Factory packages will need to be migrated, factory-auto bot will need to be adapted to reject submission without multi-spec. Hopefully, this will be a one-time only cleanup. Questions ? Comments ? -- Frederic CROZAT Enterprise Linux OS and Containers Architect SUSE
On Monday 2023-04-24 15:01, Frederic Crozat via openSUSE Factory wrote:
For instance: on Build service, package foo contains: -foo.spec -bar.spec
Only foo.spec will be built. [..] factory-auto bot will need to be adapted to reject submission without multi-spec.
There is a legit case where packages contain multiple .spec but no _multibuild, which is when e.g. package foo contains: - foo-Fedora_38.spec - foo.spec The review bot should not reject what obssched deems acceptable.
On Mon, 2023-04-24 at 16:10 +0200, Jan Engelhardt wrote:
On Monday 2023-04-24 15:01, Frederic Crozat via openSUSE Factory wrote:
For instance: on Build service, package foo contains: -foo.spec -bar.spec
Only foo.spec will be built. [..] factory-auto bot will need to be adapted to reject submission without multi-spec.
There is a legit case where packages contain multiple .spec but no _multibuild, which is when e.g. package foo contains:
- foo-Fedora_38.spec - foo.spec
The review bot should not reject what obssched deems acceptable.
Distro submit policies (ant thus what factory-auto enforces) are by definition stricter than what OBS/RPM can build. Your example does not give proper justfication to NOT do it though: already now, if any package were to include those two spec files, the 2nd one would be auto-linked on accept into Factory (actually not by OBS itself, but by the staging accept tooling) Either don't submit this to Factory or have special tags in the fedora_38.spec %if 0%{?suse_version} BuildArch: do-not-build %endif Defacto, nothing different to what we had so far - the staging tooling would be messing up badly in this case. Cheers Dominique
On Montag, 24. April 2023 15:01:41 CEST Frederic Crozat wrote:
Hi all,
As part of the setup of ALP products into build service, we would like to change Factory review policy regarding multi-specfile package:
Currently, packages containing multiple specfiles require creating links to another package, with the name of secondary specfiles.
For instance: on Build service, package foo contains: -foo.spec -bar.spec
Only foo.spec will be built. For bar.spec to be built, a linkpac is needed from foo => bar.
Creation of this link is handled by OBS when accepting a sr.
With ALP, our goal is to migrate source management to git backend in the (hopefully) near future (there won't be any immediate change on how to do package submissions, don't worry ;).
To do that and after discussing with Build Service project managers, it would be easier (and cleaner) to add to packages with multiple specfiles a _multibuild file to handle additional specfile, without the need of linkpac.
In the example above, package foo would contains: - foo.spec - bar.spec - _multibuild
_multibuild content: <multibuild> <flavor>bar.spec</flavor> </multibuild>
(there is no need to mention foo.spec).
If there is an agreement, all multi-spec Factory packages will need to be migrated, factory-auto bot will need to be adapted to reject submission without multi-spec.
Hopefully, this will be a one-time only cleanup.
Questions ? Comments ?
What I have been wishing for for a long time is a combinatorial expansion for multibuild flavors. E.g. in the science project there are currently several packages which build for different GCC versions and several MPI implementations. Currently, each has to be spelled out explicitly. There is also the case of using different spec files and multibuild at the same time. E.g. building the documentation often uses a vastly different dependency set (texlive, doxygen, graphviz, ...) as the build of the binary package(s). Its of course possible to put all into the same spec file, but the spec file gets more convoluted. It also causes rebuilds of the binary packages when just adjusting the documentation build, and vice versa. How about: <multibuild> <source file="foo.spec"> <flavor>baz</flavor> <flavor>bar</flavor> </source> <source file="bar.spec"/> </multibuild> This syntax would be backwards compatible with the current one. In case no source, but multiple flavors are specified, just keep the current behavior. Multiple source elements replaces package links. Regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen phone: +49 241 53809034 mobile: +49 151 50412019
On Mon, 24 Apr 2023, Frederic Crozat wrote:
Hi all,
As part of the setup of ALP products into build service, we would like to change Factory review policy regarding multi-specfile package:
Currently, packages containing multiple specfiles require creating links to another package, with the name of secondary specfiles.
For instance: on Build service, package foo contains: -foo.spec -bar.spec
Only foo.spec will be built. For bar.spec to be built, a linkpac is needed from foo => bar.
Creation of this link is handled by OBS when accepting a sr.
With ALP, our goal is to migrate source management to git backend in the (hopefully) near future (there won't be any immediate change on how to do package submissions, don't worry ;).
To do that and after discussing with Build Service project managers, it would be easier (and cleaner) to add to packages with multiple specfiles a _multibuild file to handle additional specfile, without the need of linkpac.
In the example above, package foo would contains: - foo.spec - bar.spec - _multibuild
_multibuild content: <multibuild> <flavor>bar.spec</flavor> </multibuild>
(there is no need to mention foo.spec).
If there is an agreement, all multi-spec Factory packages will need to be migrated, factory-auto bot will need to be adapted to reject submission without multi-spec.
Hopefully, this will be a one-time only cleanup.
Questions ? Comments ?
There's a big usability issue when you have many .spec files (for example see gcc13). On osc commit some of the services that are run automatically are quadratic in the number of spec files and multibuild flavors which causes a commit to take many minutes (unless you use --noservice). There's an old github issue about this but nobody cares. So I'd say this particular "mode" of _multibuild isn't really supported first-class. Richard.
1. not sure that using the same element for multiple build descriptions and flavors within a build description is a good idea 2. how would you define something like ``` foo.spec: - flavor: postgresql15 - flavor: postgresql14 bar.spec: - flavor: postgresql15 - flavor: postgresql14 ``` with your new syntax? -- Always remember: Never accept the world as it appears to be. Dare to see it for what it could be. The world can always use more heroes.
Le mardi 25 avril 2023 à 13:42 +0200, Marcus Rückert a écrit :
1. not sure that using the same element for multiple build descriptions and flavors within a build description is a good idea
This is wanted, according to Adrian (see https://github.com/openSUSE/open-build-service/issues/8847#issuecomment-5701... where package xml tag is considered obsolete)
2. how would you define something like
``` foo.spec: - flavor: postgresql15 - flavor: postgresql14
bar.spec: - flavor: postgresql15 - flavor: postgresql14 ```
with your new syntax?
Again, it is not my new syntax, we are only leveraging OBS features (see https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.multibuild.... ) Maybe Adrian or mls could share some lights on how to achieve what you are asking for ? -- Frederic CROZAT Enterprise Linux OS and Containers Architect SUSE
On Tue, 2023-04-25 at 06:05 +0000, Richard Biener wrote:
There's a big usability issue when you have many .spec files (for example see gcc13). On osc commit some of the services that are run automatically are quadratic in the number of spec files and multibuild flavors which causes a commit to take many minutes (unless you use --noservice). There's an old github issue about this but nobody cares.
So I'd say this particular "mode" of _multibuild isn't really supported first-class.
Richard.
Thanks for bumping this. The issue referenced is https://github.com/openSUSE/obs-service-source_validator/issues/103 I think the easiest fix is to 'not do the flavor expansion matrix' when there is more than one spec file found. OBS does not do the FLAVOR expansion / build neither in this case as far as we know: https://github.com/openSUSE/open-build-service/issues/8847 So: the package either uses flavors (one spec file and BUILD_FLAVOR) in which case the validator has to run over all flavors OR the package has more than one spec files, in which case the validator should treat them all as single spec files without flavor expansion. That should be 'simple enough' to fix in the source-validator. Cheers, Dominique
On Mon, Apr 24, 2023 at 01:01:41PM +0000, Frederic Crozat via openSUSE Factory wrote:
_multibuild content: <multibuild> <flavor>bar.spec</flavor> </multibuild>
You're supposed to drop the ".spec" suffix in the flavor. It should be just: <multibuild> <flavor>bar</flavor> </multibuild> (The bar.spec works by accident. The .spec is treated like a maintenance incident suffix.) Cheers, Michael. -- Michael Schroeder SUSE Software Solutions Germany GmbH mls@suse.de GF: Ivo Totev HRB 36809, AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
Le mardi 25 avril 2023 à 12:34 +0000, Michael Schroeder a écrit :
On Mon, Apr 24, 2023 at 01:01:41PM +0000, Frederic Crozat via openSUSE Factory wrote:
_multibuild content: <multibuild> <flavor>bar.spec</flavor> </multibuild>
You're supposed to drop the ".spec" suffix in the flavor. It should be just:
<multibuild> <flavor>bar</flavor> </multibuild>
(The bar.spec works by accident. The .spec is treated like a maintenance incident suffix.)
Thanks for the clarification ! -- Frederic CROZAT Enterprise Linux OS and Containers Architect SUSE
On Dienstag, 25. April 2023 13:42:30 CEST Marcus Rückert wrote:
1. not sure that using the same element for multiple build descriptions and flavors within a build description is a good idea
2. how would you define something like
``` foo.spec: - flavor: postgresql15 - flavor: postgresql14
bar.spec: - flavor: postgresql15 - flavor: postgresql14 ```
with your new syntax?
Multiple expansion is currently only possible with package links. If you wan't 4 builds in total *without* links, you have to specify 4 flavors, and either use exactly one spec file (i.e. merge both, use conditionals inside), or create one for each flavor. Therefor my wish for multiple levels of expansion. Another possible syntax may be using nested flavors in the xml file: <flavor name="foo"> <flavor name="postgresql15"> <flavor name="postgresql14"> </flavor> <flavor name="bar"> <flavor name="postgresql15"> <flavor name="postgresql14"> </flavor> @BUILD_FLAVOR@ would be e.g. "foo_postgresql15" (probably also define @BUILD_FLAVOR_{1,2}@ as "foo" resp. "postgresql15"). Spec file lookup is: 1. foo_postgresql15.spec 2. foo.spec (For backwards compatibility, <flavor>foo</flavor> is the same as <flavor name="foo"/>). Regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen phone: +49 241 53809034 mobile: +49 151 50412019
On Tue, 25 Apr 2023, Dominique Leuenberger wrote:
On Tue, 2023-04-25 at 06:05 +0000, Richard Biener wrote:
There's a big usability issue when you have many .spec files (for example see gcc13). On osc commit some of the services that are run automatically are quadratic in the number of spec files and multibuild flavors which causes a commit to take many minutes (unless you use --noservice). There's an old github issue about this but nobody cares.
So I'd say this particular "mode" of _multibuild isn't really supported first-class.
Richard.
Thanks for bumping this. The issue referenced is https://github.com/openSUSE/obs-service-source_validator/issues/103
I think the easiest fix is to 'not do the flavor expansion matrix' when there is more than one spec file found. OBS does not do the FLAVOR expansion / build neither in this case as far as we know: https://github.com/openSUSE/open-build-service/issues/8847
So: the package either uses flavors (one spec file and BUILD_FLAVOR) in which case the validator has to run over all flavors OR the package has more than one spec files, in which case the validator should treat them all as single spec files without flavor expansion.
That should be 'simple enough' to fix in the source-validator.
source-validator could also simply look for @BUILD_FLAVOR@ in each .spec
and not expand if that isn't even mentioned.
Richard.
--
Richard Biener
On Tue, 2023-04-25 at 13:17 +0000, Richard Biener wrote: > On Tue, 25 Apr 2023, Dominique Leuenberger wrote: > > > On Tue, 2023-04-25 at 06:05 +0000, Richard Biener wrote: > > > There's a big usability issue when you have many .spec files (for > > > example see gcc13). On osc commit some of the services that are > > > run automatically are quadratic in the number of spec files and > > > multibuild flavors which causes a commit to take many minutes > > > (unless you use --noservice). There's an old github issue about > > > this but nobody cares. > > > > > > So I'd say this particular "mode" of _multibuild isn't really > > > supported first-class. > > > > > > Richard. > > > > Thanks for bumping this. The issue referenced is > > > > https://github.com/openSUSE/obs-service-source_validator/issues/103 > > > > I think the easiest fix is to 'not do the flavor expansion matrix' > > when > > there is more than one spec file found. OBS does not do the FLAVOR > > expansion / build neither in this case as far as we know: > > https://github.com/openSUSE/open-build-service/issues/8847 > > > > So: the package either uses flavors (one spec file and > > BUILD_FLAVOR) in > > which case the validator has to run over all flavors > > OR > > the package has more than one spec files, in which case the > > validator > > should treat them all as single spec files without flavor > > expansion. > > > > That should be 'simple enough' to fix in the source-validator. > > source-validator could also simply look for @BUILD_FLAVOR@ in each > .spec > and not expand if that isn't even mentioned. > > Richard. > That's also a valid condition, indeed. Created a quick patch. Result is promising Without patch - on gcc13 sources: > time osc service ra source_validator - package has check-build.sh: (unchanged) - package has gcc13-rpmlintrc: (unchanged) - package has gcc13-testresults-rpmlintrc: (unchanged) real 3m31.572s user 3m0.255s sys 0m32.047s with patch: > time osc service ra source_validator - package has check-build.sh: (unchanged) - package has gcc13-rpmlintrc: (unchanged) - package has gcc13-testresults-rpmlintrc: (unchanged) real 0m9.608s user 0m8.035s sys 0m1.667s I guess that's something we can live with. Will create a PR against source-validator shortly Cheers, Dominique
On Tue, 25 Apr 2023, Dominique Leuenberger wrote: > On Tue, 2023-04-25 at 13:17 +0000, Richard Biener wrote: > > On Tue, 25 Apr 2023, Dominique Leuenberger wrote: > > > > > On Tue, 2023-04-25 at 06:05 +0000, Richard Biener wrote: > > > > There's a big usability issue when you have many .spec files (for > > > > example see gcc13). On osc commit some of the services that are > > > > run automatically are quadratic in the number of spec files and > > > > multibuild flavors which causes a commit to take many minutes > > > > (unless you use --noservice). There's an old github issue about > > > > this but nobody cares. > > > > > > > > So I'd say this particular "mode" of _multibuild isn't really > > > > supported first-class. > > > > > > > > Richard. > > > > > > Thanks for bumping this. The issue referenced is > > > > > > https://github.com/openSUSE/obs-service-source_validator/issues/103 > > > > > > I think the easiest fix is to 'not do the flavor expansion matrix' > > > when > > > there is more than one spec file found. OBS does not do the FLAVOR > > > expansion / build neither in this case as far as we know: > > > https://github.com/openSUSE/open-build-service/issues/8847 > > > > > > So: the package either uses flavors (one spec file and > > > BUILD_FLAVOR) in > > > which case the validator has to run over all flavors > > > OR > > > the package has more than one spec files, in which case the > > > validator > > > should treat them all as single spec files without flavor > > > expansion. > > > > > > That should be 'simple enough' to fix in the source-validator. > > > > source-validator could also simply look for @BUILD_FLAVOR@ in each > > .spec > > and not expand if that isn't even mentioned. > > > > Richard. > > > > That's also a valid condition, indeed. Created a quick patch. Result is > promising > > Without patch - on gcc13 sources: > > > time osc service ra source_validator > - package has check-build.sh: (unchanged) > - package has gcc13-rpmlintrc: (unchanged) > - package has gcc13-testresults-rpmlintrc: (unchanged) > > real 3m31.572s > user 3m0.255s > sys 0m32.047s > > > with patch: > > time osc service ra source_validator > - package has check-build.sh: (unchanged) > - package has gcc13-rpmlintrc: (unchanged) > - package has gcc13-testresults-rpmlintrc: (unchanged) > > real 0m9.608s > user 0m8.035s > sys 0m1.667s > > > I guess that's something we can live with. Nice! > Will create a PR against source-validator shortly Thanks, Richard.
participants (7)
-
Dominique Leuenberger -
Frederic Crozat -
Jan Engelhardt -
Marcus Rückert -
Michael Schroeder -
Richard Biener -
Stefan Brüns