Mailinglist Archive: opensuse-buildservice (214 mails)

< Previous Next >
Re: [opensuse-buildservice] Cross compilation dependency handling
Hi Carsten,

On Monday, February 27, 2012 02:17:34 PM Christian Schneemann wrote:
[...]
Preinstall: and VMinstall: will always go into hostarch (which
makes sense) in order to have a bootable environment

Right.


What scope will BuildRequires: dependencies from the package be
resolved with? Will it look within the <scheduler> that it is
being built within?

Will BuildRequires: be dep resolved on all architectures or just
one?

I just pushed out something which starts addressing this issues:
https://github.com/b1-systems/obs-build/commit/930554fe36992097c2b200e6db74a9ffd107600a
https://github.com/b1-systems/open-build-service/commit/91bece36f12a8095d6cf6f56566b7639a04a863d


Since we want to have _Transpartent_ cross builds (which means not too much
fidlling in each spec file with regards to BuildRequires and other things) I
enabled Build::get_build and Build::get_deps
to:

1. Substitue cross build dependency e.g. toolchain.
Like this (in your prjconf):

Substitute: gcc cross-gcc-armv7l(HOST)


(no modification on Build.pm required for this)

The idea is to assign dependencies to a specific dependency-tree/sysroot
by adding a suffix "package(SYSROOT_LABEL)".

In the above example and this first the build dependency would end up in:
cross-gcc-armv7l(HOST)

2. Additional code directly after the Build::do_subst function get called:
- Build::extract_crossdeps
extracts cross dependencies, which are marked like this
"package(SYSROOT_LABEL)". This function will then return a hash with
the SYSROOT_LABEL as key and an array/list of package name which are
dependencies for this SYSROOT_LABEL.

So this is suitable for N-sysroots.

This hash get stored as seperated seperated cross dependency variable
and returnted later.

3. In the next step Build::drop_crossdeps will drop cross dependecies from
the original dependecy list.

Since the expander function would fail to resolve the dependency of:
"coss-gcc-armv7l(HOST)"

So all those "package(SYSROOT_LABEL)" get dropped from the original list.


4. unmodified Build::expand or BSSolv.xs exapnder get called.
The dependency list will only cover native dependencies - so it should
expand as always.
This requires that the (cross) Substitute from did not break anything.
In some cases you might do something like this to no break the expander:

Substitute: libmy-devel libmy-devel libmy-devel(HOST)


5. Build::get_deps / Build::get_build returns the expanded dependency list
for the target sysroot. The other additional dependencies for the host or
other sysroot got not expanded or resolved by Build::get_deps /
Build::get_build.

I have not tried to resolve them on purpose inside Build:: since it would
not use the expander pool from BSSolv of different scheduler instances.

The return value slightly changed of Build::get_deps / Build::get_build
function:

- return ($eok, @deps)
+ return ($eok, \@deps, \%crossdeps)

(if anyone has an idea how to avoid this - please let us know)

6. All the bs_sched (one per architecture) instances call
Build::get_deps / Build::get_build once a build need or so get kicked off.

Now the scheduler retrives multiple dependency lists:
- one resolved for the native architecture for the sysroot target
-> @bdeps (like it was before)
- one or more unresolved for other sysroots (incl. the host sysroot)
-> %crossdeps

TODO: find a way how to handle the dependencies for other sysroots
Some rough ideas:
- dependencies of sysroots with the same architecture could be resolved
in the same scheduler instance and block the build or let it fail due
unresolved package
- dependencies of sysroots with other architecture:
* probe/call the other scheduler instances and ask to resolve the
dependency?
* check if we can get this information from the meta data on the disk?


Not yet solved: blocking builds if a dependency from a other
architectures is not ready/available.

7. set_building (from bs_sched) get called with an additional parameter:
%crossdeps

(Since %crossbuild is not yet expanded/resolved correctly - your cross
dependecies might not prevent the scheduler to kick off a build which
will fail to due missing cross dependencies)

Christian is working right now on checking if there %crossdeps is filled
with additional dependencies and add those as dependecy into the buildjob
file:

<bdep name="cross-gcc-armv7l" sysroot="HOST" />


8. bs_worker retrieves the job and assembles for each sysroot a differnet
package list and stores the package also in seperated directories.

9. build script will get called with additional parameters to setup different
sysroots. IIRC something like this:

build [..] --rpmlist /path/to/worker/package/host/packagelist
--sysroot /path/to/sysroot:/path/to/worker/target/pkglist \
--sysroot /path/sysroot2:/path/to/worker/target2/pkglist \
--sysroot /path/sysrootN:/path/to/worker/targetN/pkglist \
--target-sysroot /path/to/sysroot \
--target armv7l

10. init_buildsystem (WIP) setups for each sysroot a seperated package
database. Installs package into each sysroot.

11. build kicksoff rpmbuild [...] --target armv7l --root $TARGET_SYSROOT
(WIP)

...



Have you considered how to deal with x86_64 cross compiling to
x86_64 target as well? (or i586->i586) - that would be a typical
use case for 'daily development' kind of things, demo embedded
system in a X86 VM for example. That's why I was thinking of labels
instead for scheduler.

Not considered yet, but that should be possible. We are using now
labels too :)

Right. Since we moved now from AdditoinalArch: to Sysroot: it should be
possible to have whatever archiecture you want as additional sysroot.

I guess you could even map the Host-Sysroot to something else then /, but then
you still need to make sure that your BUILD_ROOT is still works with the
buildscript somehow.


Won't you be having an inconsistent RPM database in the target, as
in, 'gcc' package name is not represented in package names when
doing substitutions?


If we subsitute, lets say "gcc", so it results in cross-gcc-armv7l or something
like that - then there would be no BuildRequires/Dependency anylonger for the
target once the the job file get
assemabled by the scheduler.

But as already said, you could also try do some substitution tricks like:

Substitute: gcc cross-gcc-armv7l(HOSTLABEL) gcc

Which would add cross-gcc-armv7l as dependency for the sysroot with the label
HOSTLABLEL and it would be still a regular buildrequires for the target.


This way could populated the other sysroots as well by doing this:

Subsitute: libfoo libfoo libfoo(petproject_armv7l) libfoo(petproject_i586)

or even do this directly in your spec file:

BuildRequires: libfoo(petproject_armv7l) libfoo(petproject_i586)

[..]

Best Regards,
Daniel

--
Daniel Gollub
Linux Consultant & Developer
Tel.: +49-160 47 73 970
Mail: gollub@xxxxxxxxxxxxx

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
< Previous Next >
Follow Ups