Mailinglist Archive: opensuse-buildservice (214 mails)

< Previous Next >
[opensuse-buildservice] RFC: patches for SB2 integration into OBS for better cross compilation,
  • From: Carsten Munk <carsten.munk@xxxxxxxxx>
  • Date: Fri, 10 Feb 2012 09:33:02 +0100
  • Message-id: <CAK=iLr=OPzvS-7CyrjxLCARSSQBFLWSmonarq0REfJ3pUMVj4w@mail.gmail.com>
Hi all,

I'd like to request comments and review on my upcoming patches which
makes it possible to integrate SB2 (Scratchbox2, not related to
Scratchbox1 in usage or code) into OBS for much cleaner, flexible and
faster cross compilation approaches that can be utilized in multiple
distributions.

I'd like to emphasise that this approach is currently being used to
cross compile entire Mer core in production
(http://www.merproject.org), 300+ packages across X86, several ARM
architectures and MIPS as well, with no visible drawbacks[1]. I've
done private experiments that show it is entirely possible to
accelerate OpenSUSE ARM with the same methods and the approach has
been done in such a way it can be utilized in other OS'es besides Mer
with ease as well.

We currently accelerate cross builds as deepily as being able to do
autoconf/automake/autoreconf accelerated without ill effects for
normal perl scripts (30+ seconds down to 2 really helps), so a sample
'acl' for ARM build is down to 65 seconds accelerated with chroot
setup, rpmbuild and rpmlint (19 native x86).

What is Scratchbox2 [2] ? (from man page:), Scratchbox2 creates an
environment with essential differences to the host OS environment:

First, a virtual file system view is presented to the applications. In
practise that means that the file names (paths) which an application
uses may be modified before the OS does its operations; applications
observe a different file system that what the real file system is.
For example, common directories like /bin, /usr/lib and /lib can be
changed to point to target-specific directories, which contain files
for a different CPU architecture than
what the host has. Second, scratchbox2 makes execution of foreign
binaries completely transparent. Target's binaries can be executed on
the host just as if they were host's binaries; scratchbox2 will start
a CPU transparency method automatically whenever needed (Qemu is
typically used for this). This is being done without the use or need
for binfmt_misc setup. [see [3] for more technical info]

SB2 is in use in many different projects such as Maemo SDK+, Tizen
platform SDK and many other places and is solid and works.

There has been developed specific 'modes' for OBS build usage, you can
see some of these at
http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/modes/obs-rpm-build/fs_rules.lua
as an example of such and how easy it is to add additional
acceleration.

Now, on to the patches -- I'm touching three components, osc,
obs-build, open-build-service, with the most invasive patches in
'obs-build'.

Please review the following commits:

https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd
https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b53c397acd7f3d72d86a
https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5457

To understand the logic/approach in these patches:

* It has been a priority to not alter behavior of normal builds when
SB2 is not in usage, as to not be too invasive a patch

* SB2 deals with a host/tools directory and a target directory in it's
mappings and the patches seek to make this division when SB2 is
enabled. When running with SB2, the primary file system seen is the
target, with accelerated tools from / used.

* Instead of utilizing chroot command everywhere, a wrapper has been
made, called enter_target, see
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd#L3R126
- It classifies chroot calls into three kinds, needroot, needuser and
needscript.

needroot implies that root is needed to run the command, needuser
implies that we should 'su' to the user to run the command, and
needscript is needed to run su to the user with '-s' parameter.

- Most of the changes within the patch has been rewrites in order to
switch from chroot calls to enter_target. When SB2 is not enabled,
these are translated straight to chroot / su / su -s calls.

* There has been introduced the concept of a 'target', and we split
BUILD_ROOT contents into being under BUILD_ROOT and BUILD_TARGET. When
SB2 is not enabled, BUILD_ROOT=BUILD_TARGET=BUILD_ROOT, in order to
not change behavior. Most of the patches are related to putting some
things under BUILD_TARGET instead of BUILD_ROOT

* There has been a prjconf item "SB2install:" added for SB2-needed
dependencies (like Preinstall)

* There has been added a prjconf item "SB2flags:", which, if exists,
will activate SB2 enablement in the 'build' script, and 'getsb2flags'
script to extract this item from prjconf,
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd#L2L1049

* If SB2 is enabled, the following behaviour change will happen:
- If running in chroot mode, it will switch to 'sb2' VM type, and run
a specific init script that sets up SB2, in order to share code
better.
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd#L2R1054

- enter_target will be utilizing SB2 in order to run commands instead
of 'chroot'. At first initialization a SB2 target will be set up and
sessions related to faster cross compilation

- BUILD_TARGET will change to /target and /target will be made

- Packages will get preinstalled/installed into /target, except
SB2install: packages which will get preinstalled into / -- this will
typically be X86 binaries

- A SB2 'target' will be set up, with / as the host and /target as the
sysroot/target,
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd#L3R152

- SB2flags setting will determine a number of things related to SB2
target configuration:

https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919c6d6388bd#L3L55

--toolchain -- the cross compiler from / which should be used for this build
--installmode -- the SB2 mode that should be used to install RPMs with
--defaultmode -- the SB2 mode that should be default (read: to do builds with)
--mappingmethod -- we currently have three mapping methods, C, Lua and
Both, this will get removed eventually and become C by standard
--bootstrap -- don't pass sysroot=/ to the cross compiler, useful for
bootstrapping a target
--qemu -- what QEMU binary should be in use to run target binaries
(qemu-arm, qemu-mips), MUST be a dynamic executable, not static
--infologs -- request that info logs are being made, can be fetched
from /home/abuild/sb2_logs afterwards. Intended to be able to be
picked up like rpmlint.log is right now. These can be used to generate
graphs like http://releases.merproject.org/~carsten/rpm.pdf and with
process accounting, be able to say how much time was wasted running in
QEMU, etc
--debug -- full debug logs

Known issues:

* There is an issue with shell quoting in the chroot wrapper so
RPMIDFMT handling won't work, this is being worked on, but it doesn't
affect general building
* We should be utilizing multiple dependancy trees as proposed by
Daniel instead of SB2install, in order to have a sane way of cross
installing packages
* We can possibly move to a scenario where we have multiple
/opt/cross/ targets with multiple dependancy trees, instead of one
/target
* Work to get SB2 working for Debian packages should be done

Current implementation in Mer:

* We have a qemu-usermode package built
(http://gitweb.merproject.org/gitweb?p=mer-core/qemu-usermode.git;a=tree
) which is dynamically linked, not statically, binfmt_misc not enabled
* We have scratchbox2 package built,
http://gitweb.merproject.org/gitweb?p=mer-core/scratchbox2.git;a=tree
* We have a standard cross compiler built,
http://gitweb.merproject.org/gitweb?p=mer-core/gcc.git;a=blob;f=cross-armv6l-gcc.spec;h=0062e25b188d514afa8987c4e6db1daed4c2f0ba;hb=HEAD
* We have an injection package, sb2-tools which gets exported with
baselibs.conf into target scheduler,
http://gitweb.merproject.org/gitweb?p=mer-crosshelpers/sb2-tools-template.git;a=tree
-- this practically is an RPM that contains a X86 root file system
with qemu, cross compiler, sb2, autoconf, perl, etc.
- Should be done in a cleaner way (is quite ugly at the moment) with
multiple dependancy trees. These x86 binaries are -unmodified-, no
patchelf involved.

Enabling SB2 is as easy as:

%if %{_cross_sb2_enabled}
SB2install: sb2-tools-%{_my_port_arch}-inject
SB2flags: --toolchain %{_my_port_crosscompiler} --qemu
%{_my_port_qemu} --mappingmethod C --defaultmode obs-rpm-build+pp
%endif

Any reviews, questions, constructive criticism more than welcome. I
want to do some more testing on top of my 'master' rebase, but my aim
is to submit this merge request in middle of next week.

[1] http://webui-ci.tspre.org , make account and peek at Core:armv6l,
Core:armv7hl, Core:mipsel, Core:i586
[2] http://maemo.gitorious.org/scratchbox
[3]
http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/docs/sb2.1

BR
Carsten Munk
Imogen Software
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >