[opensuse-buildservice] multibuild flavor vs package
Hi, Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files. Are there any plans to change that? So one could write something like <multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild> to produce six packages? Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors? cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Dec 9, 2019 at 7:21 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Hi,
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
Why not just define the flavors in the spec itself? It seems crazy to try to use features like multibuild for this. It's not difficult, see: https://src.fedoraproject.org/rpms/mingw-openal-soft/blob/master/f/mingw-ope... Alternatively, something like what we do for Python and Ruby builds could do the trick here (macros that expand to flavors). -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Neal Gompa schrieb:
On Mon, Dec 9, 2019 at 7:21 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
Why not just define the flavors in the spec itself? It seems crazy to try to use features like multibuild for this.
It's not difficult, see: https://src.fedoraproject.org/rpms/mingw-openal-soft/blob/master/f/mingw-ope...
Alternatively, something like what we do for Python and Ruby builds could do the trick here (macros that expand to flavors).
I am aware of those methods. Looks like it is a matter of choosing your favorite way of getting tortured. The price of those is a lot of (black) magic with macros. Tricky to implement and hard to debug. Compared to that, multibuild looks rather straight forward to me. That's why I'm exploring it first. Requires some clarification of where multibuild leads to though. Dominique also filed an issue for that: https://github.com/openSUSE/open-build-service/issues/8847 When thinking about it I wonder if a _multibuild file is needed at all. The spec file selection could be solved with classical links just fine. Maybe with some more code in OBS to create the requires packages automatically on request accept, like external scripts do (or did?) for staging. The flavors with the @BUILD_FLAVOR@ substitution feels like it should be done in a more rpmish way. Thinking of extending bcond, eg assuming we have %bcond_with bootstrap %bcond_with small %bcond_with full [...] %if %{with bootstrap} foo %else bar %endif One could come up with a macro to replace the %bcond_with lines, let's say %{bcond_xor bootstrap small full} So that bcond_xor would have to make sure one of the options is actually selected and fail rpmbuild if not. OBS would then have to parse the %bcond_xor macro in the spec to generate those virtual packages with colon separated flavor we know from multibuild today. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday 2019-12-09 15:08, Ludwig Nussel wrote:
When thinking about it I wonder if a _multibuild file is needed at all. The spec file selection could be solved with classical links just fine.
Links don't play nice (i.e. they get expanded sometimes and sometimes differently) when you are about to copypac or sr a package. And sometimes, the sr gets rejected by the bot because it's a link. Can't happen with multibuild. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Jan Engelhardt schrieb:
On Monday 2019-12-09 15:08, Ludwig Nussel wrote:
When thinking about it I wonder if a _multibuild file is needed at all. The spec file selection could be solved with classical links just fine.
Links don't play nice (i.e. they get expanded sometimes and sometimes differently) when you are about to copypac or sr a package. And sometimes, the sr gets rejected by the bot because it's a link.
Inconsistent handling of local links sound like bugs. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 9. Dezember 2019, 13:21:13 CET Ludwig Nussel wrote:
Hi,
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
hm, not sure if it makes sense to use _multibuild here. Isn't this more like python2 vs python3 stuff? So, one spec, one build enviroment where you could pull in both works and build multiple binary rpm's? -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Adrian Schröter schrieb:
On Montag, 9. Dezember 2019, 13:21:13 CET Ludwig Nussel wrote:
Hi,
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
hm, not sure if it makes sense to use _multibuild here. Isn't this more like python2 vs python3 stuff?
So, one spec, one build enviroment where you could pull in both works and build multiple binary rpm's?
After looking at a more elaborate spec file I'm quite sure that using multibuild would be superior to complex macro magic. Much easier to handle and understand for packagers. Rpm macros like done with python and also how Fedora does mingw have to hide the fact that things are done multiple times. So when calling something like eg. %make that wouldn't actually call "make" but eg cd build_win32 make cd ../build_win64 make cd .. All commands to deal with the sources have to be duplicated that way, either hidden by macros or manually. Look at this for example: https://src.fedoraproject.org/rpms/mingw-qt5-qtbase/blob/master/f/mingw-qt5-... The spec is long enough without the duplication already. Also, by building both flavors in the same spec the build gets serialized whereas multibuild could build on two hosts in parallel. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Dec 16, 2019 at 4:59 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Adrian Schröter schrieb:
On Montag, 9. Dezember 2019, 13:21:13 CET Ludwig Nussel wrote:
Hi,
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild> <flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package> </multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
hm, not sure if it makes sense to use _multibuild here. Isn't this more like python2 vs python3 stuff?
So, one spec, one build enviroment where you could pull in both works and build multiple binary rpm's?
After looking at a more elaborate spec file I'm quite sure that using multibuild would be superior to complex macro magic. Much easier to handle and understand for packagers.
Rpm macros like done with python and also how Fedora does mingw have to hide the fact that things are done multiple times.
So when calling something like eg. %make that wouldn't actually call "make" but eg
cd build_win32 make cd ../build_win64 make cd ..
All commands to deal with the sources have to be duplicated that way, either hidden by macros or manually. Look at this for example:
https://src.fedoraproject.org/rpms/mingw-qt5-qtbase/blob/master/f/mingw-qt5-...
The spec is long enough without the duplication already.
Also, by building both flavors in the same spec the build gets serialized whereas multibuild could build on two hosts in parallel.
There are major downsides too: using multibuild means that local development of such packages is functionally impossible. You are also saying it's okay to occupy more builders at a time (which is probably not actually okay to do at scale). You also lose the guarantee that the binary artifacts are always in sync, since a failure for one flavor doesn't stop another. That is a subtle kind of breakage that's hard to perceive and fix. Duplication in the spec is something that is solvable (cf. SUSE-style single-spec for Python and Ruby). But losing understandability and coherence is something I'd rather not see us do. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Neal Gompa schrieb:
On Mon, Dec 16, 2019 at 4:59 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
[...] After looking at a more elaborate spec file I'm quite sure that using multibuild would be superior to complex macro magic. Much easier to handle and understand for packagers. [...]
There are major downsides too: using multibuild means that local development of such packages is functionally impossible.
You mean because of that nasty @BUILD_FLAVOR@ substitution? I really hope that we can find a sane alternative, like the one I suggest earlier in this thread.
You are also saying it's okay to occupy more builders at a time (which is probably not actually okay to do at scale). You also lose the
OBS is all about parallelization. That's a feature that needs to be leveraged as much as possible :-)
guarantee that the binary artifacts are always in sync, since a failure for one flavor doesn't stop another. That is a subtle kind of breakage that's hard to perceive and fix.
May happen but is a pretty standard problem to have. In fact we have that all the time with different architectures for example. And in the end mingw32 and mingw64 are nothing else than architectures. Just weirdly packaged as there's no general concept of handling cross arch packages. So no nothing special here. Doesn't sound like a big deal to me.
Duplication in the spec is something that is solvable (cf. SUSE-style single-spec for Python and Ruby). But losing understandability and coherence is something I'd rather not see us do.
Like I said, at the expense of more macro magic. I wouldn't consider myself a total novice in rpm macros but what python is doing there is close to witchcraft. That works in a situation where most packages are standard situations, using the same build system etc. Mingw packages are just like any other package in the distro. Ie has to deal with all build systems anyone ever invented means all the quirks and hacks needed to deal with them. On top of that cross compilation hacks. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday, December 17, 2019 12:36:30 AM CET Neal Gompa wrote:
On Mon, Dec 16, 2019 at 4:59 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Adrian Schröter schrieb:
On Montag, 9. Dezember 2019, 13:21:13 CET Ludwig Nussel wrote:
Hi,
Trying to figure out a way to use multibuild for mingw packages. The challenge there is that almost all packages need to be built for 32bit as well as 64bit Windows. So currently we have mingw32-foo and mingw64-foo packages. With exactly identical spec files except for s/mingw64/mingw32/ and vice versa. So sounds like a job for multibuild with flavors mingw32 and mingw64 to produce those packages from a common mingw-foo package. Now there are also packages that have multiple spec files though like gcc and pkgconf. Looks like that case was somehow forseen for multibuild as there are 'flavor' and 'package' tags. In the end both are treated the same though and end up as @BUILD_FLAVOR@ in spec files.
Are there any plans to change that? So one could write something like
<multibuild>
<flavor>mingw64</flavor> <flavor>mingw32</flavor> <package>mingw-gcc</package> <package>mingw-cross-gcc</package> <package>mingw-cross-gcc-bootstrap</package>
</multibuild>
to produce six packages?
Short of that would it work to have both links and multibuild? Ie links according to spec file names and only mingw flavors?
hm, not sure if it makes sense to use _multibuild here. Isn't this more like python2 vs python3 stuff?
So, one spec, one build enviroment where you could pull in both works and build multiple binary rpm's?
After looking at a more elaborate spec file I'm quite sure that using multibuild would be superior to complex macro magic. Much easier to handle and understand for packagers.
Rpm macros like done with python and also how Fedora does mingw have to hide the fact that things are done multiple times.
So when calling something like eg. %make that wouldn't actually call "make" but eg
cd build_win32 make cd ../build_win64 make cd ..
All commands to deal with the sources have to be duplicated that way, either hidden by macros or manually. Look at this for example:
https://src.fedoraproject.org/rpms/mingw-qt5-qtbase/blob/master/f/mingw-qt 5-qtbase.spec
The spec is long enough without the duplication already.
Also, by building both flavors in the same spec the build gets serialized whereas multibuild could build on two hosts in parallel.
There are major downsides too: using multibuild means that local development of such packages is functionally impossible.
Care to elaborate? Whats wrong with "osc build -M <flavor>"?
You are also saying it's okay to occupy more builders at a time (which is probably not actually okay to do at scale).
Often it is possible to reduce the peak resources required for package to build, which improves scaling. Also, when the build is split up, you hold on to these resources for shorter time. At least for the public OBS, scaling horizontally (more builders) is no problem at all, but blocking the few "fat" workers definitely is. Also, often the flavors have vastly different build requirements package wise. These requirements cause build loops, or delay the build of a package so doing a rebuild (which happens on every snapshot) takes much longer.
You also lose the guarantee that the binary artifacts are always in sync, since a failure for one flavor doesn't stop another. That is a subtle kind of breakage that's hard to perceive and fix.
In most cases this is not a problem: - as soon as a package is submitted from one of its development repositories, all flavors should build. - often there is no requirement for the packages to be in sync. Why should e.g. the API documentation depend on a specific rebuilt? - synchronized versions, in case these are necessary, can be enforced by appropriate Requires. Build synchronization does *not* guarantee same versions on the installed system - in case you do e.g. a "zypper in foo", the pulled in dependencies may be newer than already fulfilled (installed) ones.
Duplication in the spec is something that is solvable (cf. SUSE-style single-spec for Python and Ruby). But losing understandability and coherence is something I'd rather not see us do.
IMHO _multibuild packages are much easier to understand than the magic happening in e.g. the python singlespec macros. And why should one rebuild *all* the python3 packages and its dependees when only python2 has changed? What a waste of resources ... Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen phone: +49 241 53809034 mobile: +49 151 50412019 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (5)
-
Adrian Schröter -
Jan Engelhardt -
Ludwig Nussel -
Neal Gompa -
Stefan Brüns