[opensuse-buildservice] RFC: patches for SB2 integration into OBS for better cross compilation,
Hi all, I'd like to request comments and review on my upcoming patches which makes it possible to integrate SB2 (Scratchbox2, not related to Scratchbox1 in usage or code) into OBS for much cleaner, flexible and faster cross compilation approaches that can be utilized in multiple distributions. I'd like to emphasise that this approach is currently being used to cross compile entire Mer core in production (http://www.merproject.org), 300+ packages across X86, several ARM architectures and MIPS as well, with no visible drawbacks[1]. I've done private experiments that show it is entirely possible to accelerate OpenSUSE ARM with the same methods and the approach has been done in such a way it can be utilized in other OS'es besides Mer with ease as well. We currently accelerate cross builds as deepily as being able to do autoconf/automake/autoreconf accelerated without ill effects for normal perl scripts (30+ seconds down to 2 really helps), so a sample 'acl' for ARM build is down to 65 seconds accelerated with chroot setup, rpmbuild and rpmlint (19 native x86). What is Scratchbox2 [2] ? (from man page:), Scratchbox2 creates an environment with essential differences to the host OS environment: First, a virtual file system view is presented to the applications. In practise that means that the file names (paths) which an application uses may be modified before the OS does its operations; applications observe a different file system that what the real file system is. For example, common directories like /bin, /usr/lib and /lib can be changed to point to target-specific directories, which contain files for a different CPU architecture than what the host has. Second, scratchbox2 makes execution of foreign binaries completely transparent. Target's binaries can be executed on the host just as if they were host's binaries; scratchbox2 will start a CPU transparency method automatically whenever needed (Qemu is typically used for this). This is being done without the use or need for binfmt_misc setup. [see [3] for more technical info] SB2 is in use in many different projects such as Maemo SDK+, Tizen platform SDK and many other places and is solid and works. There has been developed specific 'modes' for OBS build usage, you can see some of these at http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/mod... as an example of such and how easy it is to add additional acceleration. Now, on to the patches -- I'm touching three components, osc, obs-build, open-build-service, with the most invasive patches in 'obs-build'. Please review the following commits: https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b5... https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5... To understand the logic/approach in these patches: * It has been a priority to not alter behavior of normal builds when SB2 is not in usage, as to not be too invasive a patch * SB2 deals with a host/tools directory and a target directory in it's mappings and the patches seek to make this division when SB2 is enabled. When running with SB2, the primary file system seen is the target, with accelerated tools from / used. * Instead of utilizing chroot command everywhere, a wrapper has been made, called enter_target, see https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - It classifies chroot calls into three kinds, needroot, needuser and needscript. needroot implies that root is needed to run the command, needuser implies that we should 'su' to the user to run the command, and needscript is needed to run su to the user with '-s' parameter. - Most of the changes within the patch has been rewrites in order to switch from chroot calls to enter_target. When SB2 is not enabled, these are translated straight to chroot / su / su -s calls. * There has been introduced the concept of a 'target', and we split BUILD_ROOT contents into being under BUILD_ROOT and BUILD_TARGET. When SB2 is not enabled, BUILD_ROOT=BUILD_TARGET=BUILD_ROOT, in order to not change behavior. Most of the patches are related to putting some things under BUILD_TARGET instead of BUILD_ROOT * There has been a prjconf item "SB2install:" added for SB2-needed dependencies (like Preinstall) * There has been added a prjconf item "SB2flags:", which, if exists, will activate SB2 enablement in the 'build' script, and 'getsb2flags' script to extract this item from prjconf, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... * If SB2 is enabled, the following behaviour change will happen: - If running in chroot mode, it will switch to 'sb2' VM type, and run a specific init script that sets up SB2, in order to share code better. https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - enter_target will be utilizing SB2 in order to run commands instead of 'chroot'. At first initialization a SB2 target will be set up and sessions related to faster cross compilation - BUILD_TARGET will change to /target and /target will be made - Packages will get preinstalled/installed into /target, except SB2install: packages which will get preinstalled into / -- this will typically be X86 binaries - A SB2 'target' will be set up, with / as the host and /target as the sysroot/target, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - SB2flags setting will determine a number of things related to SB2 target configuration: https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... --toolchain -- the cross compiler from / which should be used for this build --installmode -- the SB2 mode that should be used to install RPMs with --defaultmode -- the SB2 mode that should be default (read: to do builds with) --mappingmethod -- we currently have three mapping methods, C, Lua and Both, this will get removed eventually and become C by standard --bootstrap -- don't pass sysroot=/ to the cross compiler, useful for bootstrapping a target --qemu -- what QEMU binary should be in use to run target binaries (qemu-arm, qemu-mips), MUST be a dynamic executable, not static --infologs -- request that info logs are being made, can be fetched from /home/abuild/sb2_logs afterwards. Intended to be able to be picked up like rpmlint.log is right now. These can be used to generate graphs like http://releases.merproject.org/~carsten/rpm.pdf and with process accounting, be able to say how much time was wasted running in QEMU, etc --debug -- full debug logs Known issues: * There is an issue with shell quoting in the chroot wrapper so RPMIDFMT handling won't work, this is being worked on, but it doesn't affect general building * We should be utilizing multiple dependancy trees as proposed by Daniel instead of SB2install, in order to have a sane way of cross installing packages * We can possibly move to a scenario where we have multiple /opt/cross/ targets with multiple dependancy trees, instead of one /target * Work to get SB2 working for Debian packages should be done Current implementation in Mer: * We have a qemu-usermode package built (http://gitweb.merproject.org/gitweb?p=mer-core/qemu-usermode.git;a=tree ) which is dynamically linked, not statically, binfmt_misc not enabled * We have scratchbox2 package built, http://gitweb.merproject.org/gitweb?p=mer-core/scratchbox2.git;a=tree * We have a standard cross compiler built, http://gitweb.merproject.org/gitweb?p=mer-core/gcc.git;a=blob;f=cross-armv6l... * We have an injection package, sb2-tools which gets exported with baselibs.conf into target scheduler, http://gitweb.merproject.org/gitweb?p=mer-crosshelpers/sb2-tools-template.gi... -- this practically is an RPM that contains a X86 root file system with qemu, cross compiler, sb2, autoconf, perl, etc. - Should be done in a cleaner way (is quite ugly at the moment) with multiple dependancy trees. These x86 binaries are -unmodified-, no patchelf involved. Enabling SB2 is as easy as: %if %{_cross_sb2_enabled} SB2install: sb2-tools-%{_my_port_arch}-inject SB2flags: --toolchain %{_my_port_crosscompiler} --qemu %{_my_port_qemu} --mappingmethod C --defaultmode obs-rpm-build+pp %endif Any reviews, questions, constructive criticism more than welcome. I want to do some more testing on top of my 'master' rebase, but my aim is to submit this merge request in middle of next week. [1] http://webui-ci.tspre.org , make account and peek at Core:armv6l, Core:armv7hl, Core:mipsel, Core:i586 [2] http://maemo.gitorious.org/scratchbox [3] http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/doc... BR Carsten Munk Imogen Software -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Crasten, Hi Uwe, Uwe, Carsten is addressing similar issue as we do in the obs-build scripts - you might want to have a look on them so we can collaborate with Carsten. On Friday, February 10, 2012 09:33:02 AM Carsten Munk wrote:
Hi all,
I'd like to request comments and review on my upcoming patches which makes it possible to integrate SB2 (Scratchbox2, not related to Scratchbox1 in usage or code) into OBS for much cleaner, flexible and faster cross compilation approaches that can be utilized in multiple distributions. [...] Now, on to the patches -- I'm touching three components, osc, obs-build, open-build-service, with the most invasive patches in 'obs-build'.
Please review the following commits:
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b5... https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5...
To understand the logic/approach in these patches: [....] * Instead of utilizing chroot command everywhere, a wrapper has been made, called enter_target, see https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - It classifies chroot calls into three kinds, needroot, needuser and needscript.
needroot implies that root is needed to run the command, needuser implies that we should 'su' to the user to run the command, and needscript is needed to run su to the user with '-s' parameter.
- Most of the changes within the patch has been rewrites in order to switch from chroot calls to enter_target. When SB2 is not enabled, these are translated straight to chroot / su / su -s calls.
Here we hit the same challenge. Inside init_buildsystem and other places where a lot of chroot are done. Which makes it hard to use some parts of the scripts for "classic" crossbuilds. In our case we call multiple times init_buildsystem - to setup the actual BUILD_ROOT + for each target sysroot we want to setup. But for the init_buildsystem call for the target sysroots we want to avoid the chroot calls. In some cases some people might want to have very limited sysroots which are not suiteable for chroots at all. For instance all the rpm calls (intalling packages, init-db, quries, ...) could be done by using the host rpm binary and pointing with "rpm --dbpath" to target rpm database. E.g. rpm --dbpath $SYSROOTPATH/var/lib/rpm ... or even use that rpm --root parameter (I'm still not quite sure if rpm really makes use of chroot on its own then). With SB2 are you calling the target rpm binary (and it deps) to do this initial setup? Or do you call somehow the native rpm? (Analog to dpkg) Maybe additionally to a generic wrapper (which should also support plain calls and maybe NOOP) we should write for more frequent calls, specific wrapper - e.g. for rpm / dpkg. So the rpm / dpkg specific wrapper use the hostarch binaries and point to the sysroot package database. obs-build> git show 99417ba6c225c58dbc240720da4b6a245f753123 | grep enter_target | egrep -v '(rpm |dpkg)' | wc -l 26 projects/obs-build> git show 99417ba6c225c58dbc240720da4b6a245f753123 | grep enter_target | wc -l 44
* There has been introduced the concept of a 'target', and we split BUILD_ROOT contents into being under BUILD_ROOT and BUILD_TARGET. When SB2 is not enabled, BUILD_ROOT=BUILD_TARGET=BUILD_ROOT, in order to not change behavior. Most of the patches are related to putting some things under BUILD_TARGET instead of BUILD_ROOT
[...] For the init_buildsystem script we just introduce a --root parameter to overwrite the BUILD_ROOT variable just for the init_buildsystem script (not for the buildscript). So we can call init_buildsystem additionally for each target. But we are still in middle of adapting the init_buildsystem to setup a seperated sysroots. So not quite sure yet if this solves all our problems with BUILD_TARGET/BUILD_ROOT. Have you tried to just overwrite the BUILD_ROOT in the init_buildsystem to some "target" direcotry. This diff hunk is very useful: https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... Without that using init_buildsystem to setup a seperated target sysroot fails to detect the right package-system (deb || rpm). The target architectures would require the rpm or the deb package as native dependency to indicating which kind of disto-package-system should be setup: deb or rpm distro-package-system. You might want to isolate this diff hunk as seperated patch and and ask to get this one pulled. Maybe isolating those new features (e.g. chroot wrapper, BUILD_ROOT->BUILD_TARGET migration, SB2 introduction, ...) might speed up getting this pulled. I guess it would make review easier - since you have smallers diffs which introduce one feature. With regards to your open-build-service change: https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b5... Instead of introducing "SB2install" I guess a generic "Required(host):" might also work for you - right? Required(host): (which is not yet implemented) would add dependencies for the host "target" only. Same for your osc change: https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5... If "Required(host):" would work for you - we could go with that in the osc as well. Best Regards, Daniel -- Daniel Gollub Linux Consultant & Developer Tel.: +49-160 47 73 970 Mail: gollub@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
(accidentially sent this directly to Daniel, re-sending..) 24. feb. 2012 09.29 skrev Daniel Gollub <gollub@b1-systems.de>:
Hi Crasten, Hi Uwe, Hi, some quick comments while I parse through your other mails on OBS dependencies.
Uwe, Carsten is addressing similar issue as we do in the obs-build scripts - you might want to have a look on them so we can collaborate with Carsten.
My goal is that we can have a system that uses similar cross compile dependency solution as yours (generic is always better), but be able to differ in code paths on how exactly things get executed - there's merits to both approaches. Because of SB2 I can do even more transparent cross compile, including running target binaries, so. Just to fit this in with what I'm working on: I have a proof of concept running with SB2 that allows me to have an no-tools-available target (ie, only glibc, glibc-devel, mostly unmodified rpmrc/macros) that can build libtool, acl, attr, busybox.. and run ARM binaries during build too. It's a bit of the same thing you want to do, but without the need to modify packages/specs or do voodoo rpm config changes. But I'd want a cleaner solution when it comes to "host" dependencies, instead of RPM package Provides: magic.
In our case we call multiple times init_buildsystem - to setup the actual BUILD_ROOT + for each target sysroot we want to setup. But for the init_buildsystem call for the target sysroots we want to avoid the chroot calls. In some cases some people might want to have very limited sysroots which are not suiteable for chroots at all. Agreed, that seems like a suitable direction and might make the code significantly cleaner.
For instance all the rpm calls (intalling packages, init-db, quries, ...) could be done by using the host rpm binary and pointing with "rpm --dbpath" to target rpm database. E.g. rpm --dbpath $SYSROOTPATH/var/lib/rpm ... or even use that rpm --root parameter (I'm still not quite sure if rpm really makes use of chroot on its own then).
rpm --root=/foo -i foo.rpm will for postscripts chroot in (not the command, the syscall) and run the script and chroot out again I think we can possibly do something like enter_target rpm / enter_target dpkg so we can vary on approaches to running RPM. You'd want to prefix with --rpmdb / --root for your approach, i'd want to run sb2 + rpm.
With SB2 are you calling the target rpm binary (and it deps) to do this initial setup? Or do you call somehow the native rpm? In SB2 it's possible to do both. I set up a SB2 target in the target, which may or may not have /bin/rpm in it. In practice, when the target rpm binary is executed (or just /bin/rpm attempted to be executed), it will run the host side 'rpm' with the perception that it is within the target file system.
(Analog to dpkg)
Maybe additionally to a generic wrapper (which should also support plain calls and maybe NOOP) we should write for more frequent calls, specific wrapper - e.g. for rpm / dpkg. So the rpm / dpkg specific wrapper use the hostarch binaries and point to the sysroot package database.
Yup, should be trivial to do. needroot/needuser may have to vary though for the package manager.
Have you tried to just overwrite the BUILD_ROOT in the init_buildsystem to some "target" direcotry. No, but it sounds worthwhile to try.
This diff hunk is very useful:
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919...
Without that using init_buildsystem to setup a seperated target sysroot fails to detect the right package-system (deb || rpm). The target architectures would require the rpm or the deb package as native dependency to indicating which kind of disto-package-system should be setup: deb or rpm distro-package-system.
You might want to isolate this diff hunk as seperated patch and and ask to get this one pulled. Noted, I'll take out this patch and send it separately - it's very useful.
With regards to your open-build-service change: https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b5...
Instead of introducing "SB2install" I guess a generic "Required(host):" might also work for you - right?
Required(host): (which is not yet implemented) would add dependencies for the host "target" only. Yes, if we go for your cross dependency approach we wouldn't need SB2install.
Same for your osc change: https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5...
If "Required(host):" would work for you - we could go with that in the osc as well. The less patches to add the better, much easier to deploy then.
BR Carsten Munk -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
24. feb. 2012 10.00 skrev Carsten Munk <carsten.munk@gmail.com>:
This diff hunk is very useful:
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919...
Without that using init_buildsystem to setup a seperated target sysroot fails to detect the right package-system (deb || rpm). The target architectures would require the rpm or the deb package as native dependency to indicating which kind of disto-package-system should be setup: deb or rpm distro-package-system.
You might want to isolate this diff hunk as seperated patch and and ask to get this one pulled. Noted, I'll take out this patch and send it separately - it's very useful.
Submitted as https://github.com/openSUSE/obs-build/pull/3 BR Carsten Munk -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
[..]
* There has been a prjconf item "SB2install:" added for SB2-needed dependencies (like Preinstall)
* There has been added a prjconf item "SB2flags:", which, if exists, will activate SB2 enablement in the 'build' script, and 'getsb2flags' script to extract this item from prjconf, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b6928991 9c6d6388bd#L2L1049
We should find common/generic flags here. Otherwise prjconf's will be unreadable and cluttered with SB*/Preinstall/foobar ... lets keep it simple and generic.
* If SB2 is enabled, the following behaviour change will happen: - If running in chroot mode, it will switch to 'sb2' VM type, and run a specific init script that sets up SB2, in order to share code better. https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b692899 19c6d6388bd#L2R1054
- enter_target will be utilizing SB2 in order to run commands instead of 'chroot'. At first initialization a SB2 target will be set up and sessions related to faster cross compilation
- BUILD_TARGET will change to /target and /target will be made
- Packages will get preinstalled/installed into /target, except SB2install: packages which will get preinstalled into / -- this will typically be X86 binaries
- A SB2 'target' will be set up, with / as the host and /target as the sysroot/target, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b6928991 9c6d6388bd#L3R152
- SB2flags setting will determine a number of things related to SB2 target configuration:
https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b6928991 9c6d6388bd#L3L55
--toolchain -- the cross compiler from / which should be used for this build --installmode -- the SB2 mode that should be used to install RPMs with --defaultmode -- the SB2 mode that should be default (read: to do builds with) --mappingmethod -- we currently have three mapping methods, C, Lua and Both, this will get removed eventually and become C by standard --bootstrap -- don't pass sysroot=/ to the cross compiler, useful for bootstrapping a target --qemu -- what QEMU binary should be in use to run target binaries (qemu-arm, qemu-mips), MUST be a dynamic executable, not static --infologs -- request that info logs are being made, can be fetched from /home/abuild/sb2_logs afterwards. Intended to be able to be picked up like rpmlint.log is right now. These can be used to generate graphs like http://releases.merproject.org/~carsten/rpm.pdf and with process accounting, be able to say how much time was wasted running in QEMU, etc --debug -- full debug logs
Going away from static and patchelf is good. I'm looking forward to the actual implementation of multilibs as discussed in latest talks (e.g. wookey). Adopting this might simplify things even more.
Known issues:
* There is an issue with shell quoting in the chroot wrapper so RPMIDFMT handling won't work, this is being worked on, but it doesn't affect general building * We should be utilizing multiple dependancy trees as proposed by Daniel instead of SB2install, in order to have a sane way of cross installing packages * We can possibly move to a scenario where we have multiple /opt/cross/ targets with multiple dependancy trees, instead of one /target * Work to get SB2 working for Debian packages should be done
Current implementation in Mer:
* We have a qemu-usermode package built (http://gitweb.merproject.org/gitweb?p=mer-core/qemu-usermode.git;a=tree ) which is dynamically linked, not statically, binfmt_misc not enabled * We have scratchbox2 package built, http://gitweb.merproject.org/gitweb?p=mer-core/scratchbox2.git;a=tree * We have a standard cross compiler built, http://gitweb.merproject.org/gitweb?p=mer-core/gcc.git;a=blob;f=cross-armv6 l-gcc.spec;h=0062e25b188d514afa8987c4e6db1daed4c2f0ba;hb=HEAD * We have an injection package, sb2-tools which gets exported with baselibs.conf into target scheduler, http://gitweb.merproject.org/gitweb?p=mer-crosshelpers/sb2-tools-template.g it;a=tree -- this practically is an RPM that contains a X86 root file system with qemu, cross compiler, sb2, autoconf, perl, etc. - Should be done in a cleaner way (is quite ugly at the moment) with multiple dependancy trees. These x86 binaries are -unmodified-, no patchelf involved.
Good, but no big blob should be used in the long run.
Enabling SB2 is as easy as:
Best regards, JS -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Carsten Munk -
Daniel Gollub -
Jan-Simon Möller