Hi Carsten, Hi Adrian, On Thursday, February 16, 2012 11:01:15 AM Adrian Schröter wrote:
Am Donnerstag, 16. Februar 2012, 10:43:01 schrieb Carsten Munk: [...]
Daniel Gollub suggested that we work with N amount of 'sysroots' and N rpm databases and I tend to agree. Right now in SB2 I distinguish between host (/) and target /target, but it might be more flexible to do with N targets instead.
We started last week finally to do some prototyping of this idea.
Added Christian and Uwe into the loop, since booth of them implemented most of the stuff I'm mentioning here.
This is a "brief" (sorry, at least I tried to keep it brief) what we have already in place and is working to some point (we have no build results yet).
Scheduler and Worker modification by Christian:
* Add new parameter to prjconf for additional architectures
New parameter 'AdditionalArch' added to projectconfiguration (prjconf). This
parameter identifies additional architectures with its sysroot that
should be downloaded during build preparation.
AdditionalArch: <arch> <sysroot>
In the prjconf we put something like this:
---8<--
[...]
Required: gcc gcc%{gcc_version} glibc rpm tar patch
%ifarch armv5el armv7l
Hostarch: x86_64
%obs_crossbuild 1 # not yet used
AdditionalArch: x86_64 / # actually not required, IIR
AdditionalArch: armv7l /sysroot/armv7l
AdditionalArch: armv5el /sysroot/armv5el
Required(target): tar
[...]
--->8----
* determine packages used in hostsystem or in target-sysroot
To determine which package has to be installed where, in the host system
or in a sysroot for a crossbuild. Therefor the XML for the buildinfo got
a new field for packages. 'neededbyhost' has to be set to 1 if a package
has to be installed in the host system. If this parameter is not set or
0 the package will be downloaded for the architecture the buildjob is
for and for all additional architectures into their sysroots.
With the example prjconf from above, the scheduler sends an enhanced buildjob file which looks something like this:
---8<---
<buildinfo project="home:Admin" repository="openSUSE_Factory_ARM_standard" package="foo" srcserver="http://b1-systems.de:5352" reposerver="http://b1-systems.de:5252">
<job>home:Admin::openSUSE_Factory_ARM_standard::foo-6763ac6ae31fbc1a89ed98c53d443ce8</job>
<arch>armv7l</arch>
<hostarch>x86_64</hostarch>
The question is of course, can we effectively work with multiple dependency trees (maybe even across schedulers?) within the current code and how do we do it best?
Our prototype managed to assemble a buildjob which a patched worker understands and retrieves RPMs from different trees. But all this requries that those RPMs from the different architectures are already build and available. We still need to introduce something that the scheduler schedules the cross build job only if all cross-target dependencies are solved. So there is the need that the scheduler check the meta-data/status of the other architectures - if the all cross dependencies are ready. For example: if a pre-install: for the Hostarch x86_64 is failing/missing, the armv7l scheduler would still schedule the buildjob.
Adrian suggested something like Preinstall($hostarch) to indicate that this needs to be preinstalled on host architecture.
I guess introducing "preinstall($hostarch)" is harder then "preinstall(host)".
Working with our current prototype it really looks like you do not really need to care about what your hostarch is.
You just need to distinguish between: is it a package for the host or for one of the targets?
So we currently follow this approach:
Imagine you need have as Hostarch: x86_64 (<hostarch />) and need additionally those Targets: armv7l, i586, armv5el. (
Thinking again about it, specify an arch here makes not much sense.
Better something like this in spec files:
BuildRequire(host): gcc BuildRequire(target): libqt4-devel
Since this is not supported by rpm yet, we would need to have something like
#!BuildRequire(host): gcc #!BuildRequire(target): libqt4-devel
So when a
BuildRequire: gcc
appears later in the spec file (for plain rpm non-cross building), OBS can see that it has it already, but it knows already for which architecture it needs it.
I thought about doing this in another way or even provide something additionally. Instead of putting special crossbuild dependencies in each spec files (by default), use some kind of Substitute(host): / Substitute(target): / Required(target): / Required(host): combination. Something like this: Substitute(target): gcc %{crosstag}gcc -> Drops gcc from the target dependency list and will replace it with a dependency of %{crosstag}gcc in the hostarch tree. Substitute(crossdep): libqt4-devel -> if libqt4-devel shows up in the build-depenency of a target (in a crossbuild), libqt4-devel gets a "cross dependency": it gets additionally a dependency for the hostarch. (e.g. if you want to use pre-processors and run them with native performance, but need target specific headers, static libraries, ...)
The prjconf Preinstall, Support and Require lines could also support the (host) and (target) extensions.
Right. So far we only needed Required(target). But very likely we need more of them while keep working on that.
My suggestion would be perhaps to indicate through configuration where to put things, for example, Roots: host=/ armv7l=/opt/cross/armv7l, etc, so we can do dynamic labeling and telling what location on the build filesystem things should go in.
Yes, similar to Hostarch: directive, we could add a Targetpath: directive...
What do you think about the "AdditionalArch: $ARCH $TARGETPATH" directive? Or should we change this to something else? Btw, our git branches can be found here: https://github.com/b1-systems/obs-build/commits/crossbuild https://github.com/b1-systems/open-build-service/commits/crossbuild Best Regards, Daniel -- Daniel Gollub Linux Consultant & Developer Tel.: +49-160 47 73 970 Mail: gollub@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537