Hi all, I'd like to request comments and review on my upcoming patches which makes it possible to integrate SB2 (Scratchbox2, not related to Scratchbox1 in usage or code) into OBS for much cleaner, flexible and faster cross compilation approaches that can be utilized in multiple distributions. I'd like to emphasise that this approach is currently being used to cross compile entire Mer core in production (http://www.merproject.org), 300+ packages across X86, several ARM architectures and MIPS as well, with no visible drawbacks[1]. I've done private experiments that show it is entirely possible to accelerate OpenSUSE ARM with the same methods and the approach has been done in such a way it can be utilized in other OS'es besides Mer with ease as well. We currently accelerate cross builds as deepily as being able to do autoconf/automake/autoreconf accelerated without ill effects for normal perl scripts (30+ seconds down to 2 really helps), so a sample 'acl' for ARM build is down to 65 seconds accelerated with chroot setup, rpmbuild and rpmlint (19 native x86). What is Scratchbox2 [2] ? (from man page:), Scratchbox2 creates an environment with essential differences to the host OS environment: First, a virtual file system view is presented to the applications. In practise that means that the file names (paths) which an application uses may be modified before the OS does its operations; applications observe a different file system that what the real file system is. For example, common directories like /bin, /usr/lib and /lib can be changed to point to target-specific directories, which contain files for a different CPU architecture than what the host has. Second, scratchbox2 makes execution of foreign binaries completely transparent. Target's binaries can be executed on the host just as if they were host's binaries; scratchbox2 will start a CPU transparency method automatically whenever needed (Qemu is typically used for this). This is being done without the use or need for binfmt_misc setup. [see [3] for more technical info] SB2 is in use in many different projects such as Maemo SDK+, Tizen platform SDK and many other places and is solid and works. There has been developed specific 'modes' for OBS build usage, you can see some of these at http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/mod... as an example of such and how easy it is to add additional acceleration. Now, on to the patches -- I'm touching three components, osc, obs-build, open-build-service, with the most invasive patches in 'obs-build'. Please review the following commits: https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... https://github.com/stskeeps/open-build-service/commit/7fced08b7c2eecf93e82b5... https://github.com/stskeeps/osc/commit/f0ba509e308d630345d0e064d7aa3856c5ae5... To understand the logic/approach in these patches: * It has been a priority to not alter behavior of normal builds when SB2 is not in usage, as to not be too invasive a patch * SB2 deals with a host/tools directory and a target directory in it's mappings and the patches seek to make this division when SB2 is enabled. When running with SB2, the primary file system seen is the target, with accelerated tools from / used. * Instead of utilizing chroot command everywhere, a wrapper has been made, called enter_target, see https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - It classifies chroot calls into three kinds, needroot, needuser and needscript. needroot implies that root is needed to run the command, needuser implies that we should 'su' to the user to run the command, and needscript is needed to run su to the user with '-s' parameter. - Most of the changes within the patch has been rewrites in order to switch from chroot calls to enter_target. When SB2 is not enabled, these are translated straight to chroot / su / su -s calls. * There has been introduced the concept of a 'target', and we split BUILD_ROOT contents into being under BUILD_ROOT and BUILD_TARGET. When SB2 is not enabled, BUILD_ROOT=BUILD_TARGET=BUILD_ROOT, in order to not change behavior. Most of the patches are related to putting some things under BUILD_TARGET instead of BUILD_ROOT * There has been a prjconf item "SB2install:" added for SB2-needed dependencies (like Preinstall) * There has been added a prjconf item "SB2flags:", which, if exists, will activate SB2 enablement in the 'build' script, and 'getsb2flags' script to extract this item from prjconf, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... * If SB2 is enabled, the following behaviour change will happen: - If running in chroot mode, it will switch to 'sb2' VM type, and run a specific init script that sets up SB2, in order to share code better. https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - enter_target will be utilizing SB2 in order to run commands instead of 'chroot'. At first initialization a SB2 target will be set up and sessions related to faster cross compilation - BUILD_TARGET will change to /target and /target will be made - Packages will get preinstalled/installed into /target, except SB2install: packages which will get preinstalled into / -- this will typically be X86 binaries - A SB2 'target' will be set up, with / as the host and /target as the sysroot/target, https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... - SB2flags setting will determine a number of things related to SB2 target configuration: https://github.com/stskeeps/obs-build/commit/0687c00013901bdba893f2b69289919... --toolchain -- the cross compiler from / which should be used for this build --installmode -- the SB2 mode that should be used to install RPMs with --defaultmode -- the SB2 mode that should be default (read: to do builds with) --mappingmethod -- we currently have three mapping methods, C, Lua and Both, this will get removed eventually and become C by standard --bootstrap -- don't pass sysroot=/ to the cross compiler, useful for bootstrapping a target --qemu -- what QEMU binary should be in use to run target binaries (qemu-arm, qemu-mips), MUST be a dynamic executable, not static --infologs -- request that info logs are being made, can be fetched from /home/abuild/sb2_logs afterwards. Intended to be able to be picked up like rpmlint.log is right now. These can be used to generate graphs like http://releases.merproject.org/~carsten/rpm.pdf and with process accounting, be able to say how much time was wasted running in QEMU, etc --debug -- full debug logs Known issues: * There is an issue with shell quoting in the chroot wrapper so RPMIDFMT handling won't work, this is being worked on, but it doesn't affect general building * We should be utilizing multiple dependancy trees as proposed by Daniel instead of SB2install, in order to have a sane way of cross installing packages * We can possibly move to a scenario where we have multiple /opt/cross/ targets with multiple dependancy trees, instead of one /target * Work to get SB2 working for Debian packages should be done Current implementation in Mer: * We have a qemu-usermode package built (http://gitweb.merproject.org/gitweb?p=mer-core/qemu-usermode.git;a=tree ) which is dynamically linked, not statically, binfmt_misc not enabled * We have scratchbox2 package built, http://gitweb.merproject.org/gitweb?p=mer-core/scratchbox2.git;a=tree * We have a standard cross compiler built, http://gitweb.merproject.org/gitweb?p=mer-core/gcc.git;a=blob;f=cross-armv6l... * We have an injection package, sb2-tools which gets exported with baselibs.conf into target scheduler, http://gitweb.merproject.org/gitweb?p=mer-crosshelpers/sb2-tools-template.gi... -- this practically is an RPM that contains a X86 root file system with qemu, cross compiler, sb2, autoconf, perl, etc. - Should be done in a cleaner way (is quite ugly at the moment) with multiple dependancy trees. These x86 binaries are -unmodified-, no patchelf involved. Enabling SB2 is as easy as: %if %{_cross_sb2_enabled} SB2install: sb2-tools-%{_my_port_arch}-inject SB2flags: --toolchain %{_my_port_crosscompiler} --qemu %{_my_port_qemu} --mappingmethod C --defaultmode obs-rpm-build+pp %endif Any reviews, questions, constructive criticism more than welcome. I want to do some more testing on top of my 'master' rebase, but my aim is to submit this merge request in middle of next week. [1] http://webui-ci.tspre.org , make account and peek at Core:armv6l, Core:armv7hl, Core:mipsel, Core:i586 [2] http://maemo.gitorious.org/scratchbox [3] http://maemo.gitorious.org/scratchbox2/scratchbox2/blobs/2.3_development/doc... BR Carsten Munk Imogen Software -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org