[opensuse-buildservice] HELP: How to setup cross-build on private OBS 2.5.2 using QEMU chroot?
Hi, I'm in the process of migrating an old private OBS instance up to OBS 2.5. "build" package used to have "copy_qemu" function to copy qemu static binary from worker host into build chroot. The project was building all fine with old "copy_qemu" function, and I want to change from "copy_qemu" to "Preinstall" method. For newer opensuse 13.1 with OBS 2.5, the copy function is gone, so I create a pre-built QEMU binary package in my project, and added lines below to the "Project Config": %ifarch armv7el armv8el Hostarch: x86_64 Preinstall: qemu-linux-user-arm %endif To be safe, I still install packages below on both OBS server and OBS worker: build 20140424-1.1 build-initvm-x86_64 20140424-1.1 qemu 2.0.0-236.1 qemu-arm 2.0.0-236.1 qemu-kvm 2.0.0-236.1 qemu-linux-user 2.0.0-234.3 I have tried pre-built QEMU binaries from both "openSUSE:Factory", "openSUSE:Factory:ARM" and "tizen" by extracting files from RPM and tar the binaries: qemu-linux-user-2.0.0-234.1.i586.rpm qemu-linux-user-2.0.0-234.1.x86_64.rpm qemu-linux-user-2.0.0-234.1.armv7hl.rpm qemu-linux-user-cross-arm-1.6.0-2.5.armv7l.rpm (the outcomes are all the same) But the build still failure with errors below: [ 0s] Memory limit set to 35785004KB [ 0s] Using BUILD_ROOT=/local/ramdrive/obs/worker/root_1 [ 0s] Using BUILD_ARCH=armv7el [ 0s] [ 0s] [ 0s] obs-worker-009 started "build hello.spec" at Wed May 21 20:59:22 UTC 2014. [ 0s] [ 0s] Building hello for project 'home:rickliu:trunk_mxc' repository 'Trunk_MxC_standard' arch 'armv7el' srcmd5 '113b5e223b7ed410d13a13aa2fa6449b' [ 0s] [ 0s] processing recipe /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec ... [ 0s] running changelog2spec --target rpm --file /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec [ 0s] init_buildsystem --configdir /var/run/obs/worker/1/build/configs --cachedir /var/cache/build --clean --rpmlist /local/ramdrive/obs/worker/root_1/.build.rpmlist /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec build rpmlint-Factory ... [ 0s] registering binfmt handlers for cross build [ 0s] interpreter for 'aarch64' is -1 [ 0s] write: Invalid argument [ 0s] /proc/sys/fs/binfmt_misc/register: write failed [ 0s] /var/run/obs/worker/1/build/qemu-reg: line 7: write failed. Content: :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-binfmt:P [ 0s] [ 0s] /.build/qemu-reg: No such file or directory [ 0s] /.build/qemu-reg: failed. Trying alternate binfmt file [ 0s] interpreter for 'aarch64' is -1 [ 0s] write: Invalid argument [ 0s] /proc/sys/fs/binfmt_misc/register: write failed [ 0s] /usr/lib/build/qemu-reg: line 7: write failed. Content: :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-binfmt:P [ 0s] [ 0s] /usr/lib/build/qemu-reg: binfmt registration failed [ 1s] cycle: glibc -> glibc-common -> bash [ 1s] breaking dependency glibc-common -> bash [ 1s] cycle: glibc -> glibc-common [ 1s] breaking dependency glibc -> glibc-common [ 1s] cycle: glibc -> nss-softokn-freebl [ 1s] breaking dependency nss-softokn-freebl -> glibc [ 1s] cycle: libgcc -> glibc -> nss-softokn-freebl [ 1s] breaking dependency glibc -> nss-softokn-freebl [ 1s] cycle: libgcc -> glibc [ 1s] breaking dependency libgcc -> glibc [ 1s] cycle: coreutils -> pam [ 1s] breaking dependency coreutils -> pam [ 1s] cycle: rpm-libs -> rpm [ 1s] breaking dependency rpm -> rpm-libs [ 1s] [1/62] preinstalling setup... [ 1s] [2/62] preinstalling filesystem... [ 1s] [3/62] preinstalling libgcc... [ 1s] [4/62] preinstalling nss-softokn-freebl... [ 1s] [5/62] preinstalling glibc... [ 1s] [6/62] preinstalling bzip2-libs... [ 1s] [7/62] preinstalling db4... [ 1s] [8/62] preinstalling elfutils-libelf... [ 1s] [9/62] preinstalling libattr... [ 1s] [10/62] preinstalling liblua... [ 1s] [11/62] preinstalling libstdc++... [ 1s] [12/62] preinstalling ncurses-libs... [ 1s] [13/62] preinstalling nspr... [ 1s] [14/62] preinstalling popt... [ 1s] [15/62] preinstalling sed... [ 1s] [16/62] preinstalling xz-libs... [ 2s] [17/62] preinstalling zlib... [ 2s] [18/62] preinstalling glibc-common... [ 2s] [19/62] preinstalling libcap... [ 2s] [20/62] preinstalling bash... [ 2s] [21/62] preinstalling file-libs... [ 2s] [22/62] preinstalling libacl... [ 2s] [23/62] preinstalling readline... [ 2s] [24/62] preinstalling qemu-linux-user-armv7hl-arm... [ 3s] [25/62] preinstalling diffutils... [ 3s] [26/62] preinstalling tar... [ 3s] [27/62] preinstalling bzip2... [ 3s] [28/62] preinstalling file... [ 3s] [29/62] preinstalling sqlite... [ 3s] [30/62] preinstalling coreutils... [ 3s] [31/62] preinstalling aaa-meego-accelerator... [ 4s] [32/62] preinstalling pam... [ 4s] [33/62] preinstalling nss... [ 4s] [34/62] preinstalling bzip2-libs-x86-arm... [ 4s] [35/62] preinstalling db4-x86-arm... [ 4s] [36/62] preinstalling elfutils-libelf-x86-arm... [ 4s] [37/62] preinstalling file-libs-x86-arm... [ 4s] [38/62] preinstalling glibc-x86-arm... [ 4s] [39/62] preinstalling libacl-x86-arm... [ 4s] [40/62] preinstalling libattr-x86-arm... [ 4s] [41/62] preinstalling libcap-x86-arm... [ 4s] [42/62] preinstalling libgcc-x86-arm... [ 4s] [43/62] preinstalling liblua-x86-arm... [ 4s] [44/62] preinstalling ncurses-libs-x86-arm... [ 4s] [45/62] preinstalling nspr-x86-arm... [ 4s] [46/62] preinstalling nss-softokn-freebl-x86-arm... [ 4s] [47/62] preinstalling nss-x86-arm... [ 4s] [48/62] preinstalling popt-x86-arm... [ 4s] [49/62] preinstalling sqlite-x86-arm... [ 4s] [50/62] preinstalling xz-libs-x86-arm... [ 4s] [51/62] preinstalling zlib-x86-arm... [ 4s] [52/62] preinstalling rpm... [ 4s] [53/62] preinstalling sed-x86-arm... [ 4s] [54/62] preinstalling bash-x86-arm... [ 4s] [55/62] preinstalling bzip2-x86-arm... [ 5s] [56/62] preinstalling diffutils-x86-arm... [ 5s] [57/62] preinstalling coreutils-x86-arm... [ 5s] [58/62] preinstalling file-x86-arm... [ 5s] [59/62] preinstalling tar-x86-arm... [ 5s] [60/62] preinstalling rpm-libs... [ 5s] [61/62] preinstalling rpm-libs-x86-arm... [ 5s] [62/62] preinstalling rpm-x86-arm... [ 5s] [ 5s] running bzip2-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running tar-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running coreutils-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running nss-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running sed-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running diffutils-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running file-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running rpm-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running aaa-meego-accelerator postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] running bash-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': No such file or directory [ 5s] initializing rpm db... [ 5s] chroot: failed to run command '/usr/bin/rpmdb': No such file or directory Thanks, Rick -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 21. Mai 2014, 21:47:48 wrote Rick Liu:
Hi,
I'm in the process of migrating an old private OBS instance up to OBS 2.5. "build" package used to have "copy_qemu" function to copy qemu static binary from worker host into build chroot. The project was building all fine with old "copy_qemu" function, and I want to change from "copy_qemu" to "Preinstall" method.
yes, that is gone, because on an OBS and qemu update you would run into the problem that newer qemu versions suddenly behave different now. So, we support only qemu builds where qemu comes from some project repository so you can have different versions and also reliable builds in future. (since OBS 2.4)
For newer opensuse 13.1 with OBS 2.5, the copy function is gone, so I create a pre-built QEMU binary package in my project, and added lines below to the "Project Config":
%ifarch armv7el armv8el Hostarch: x86_64 Preinstall: qemu-linux-user-arm %endif
To be safe, I still install packages below on both OBS server and OBS worker: build 20140424-1.1 build-initvm-x86_64 20140424-1.1 qemu 2.0.0-236.1 qemu-arm 2.0.0-236.1 qemu-kvm 2.0.0-236.1 qemu-linux-user 2.0.0-234.3
The qemu packages should not be needed, if you use kvm or xen workers. ...
[ 0s] Building hello for project 'home:rickliu:trunk_mxc' repository 'Trunk_MxC_standard' arch 'armv7el' srcmd5 '113b5e223b7ed410d13a13aa2fa6449b' [ 0s] [ 0s] processing recipe /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec ... [ 0s] running changelog2spec --target rpm --file /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec [ 0s] init_buildsystem --configdir /var/run/obs/worker/1/build/configs --cachedir /var/cache/build --clean --rpmlist /local/ramdrive/obs/worker/root_1/.build.rpmlist /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec build rpmlint-Factory ... [ 0s] registering binfmt handlers for cross build [ 0s] interpreter for 'aarch64' is -1 [ 0s] write: Invalid argument [ 0s] /proc/sys/fs/binfmt_misc/register: write failed [ 0s] /var/run/obs/worker/1/build/qemu-reg: line 7: write failed. Content: :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-binfmt:P
IIRC it should not abort here, but continue to register the other emulators. Can you check if the registration did work? ls /proc/sys/fs/binfmt_misc/arm* -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany
On Donnerstag, 22. Mai 2014, 09:43:55 wrote Adrian Schröter:
On Mittwoch, 21. Mai 2014, 21:47:48 wrote Rick Liu: ...
[ 0s] Building hello for project 'home:rickliu:trunk_mxc' repository 'Trunk_MxC_standard' arch 'armv7el' srcmd5 '113b5e223b7ed410d13a13aa2fa6449b' [ 0s] [ 0s] processing recipe /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec ... [ 0s] running changelog2spec --target rpm --file /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec [ 0s] init_buildsystem --configdir /var/run/obs/worker/1/build/configs --cachedir /var/cache/build --clean --rpmlist /local/ramdrive/obs/worker/root_1/.build.rpmlist /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec build rpmlint-Factory ... [ 0s] registering binfmt handlers for cross build [ 0s] interpreter for 'aarch64' is -1 [ 0s] write: Invalid argument [ 0s] /proc/sys/fs/binfmt_misc/register: write failed [ 0s] /var/run/obs/worker/1/build/qemu-reg: line 7: write failed. Content: :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-binfmt:P
IIRC it should not abort here, but continue to register the other emulators.
Can you check if the registration did work?
ls /proc/sys/fs/binfmt_misc/arm*
Another guess, doess your qemu-linux-user package which you preinstalled containing the -binfmt wrappers? We do register them by default, because otherwise argv[0] gets lost which breaks quite some builds. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany
IIRC it should not abort here, but continue to register the other emulators. Can you check if the registration did work? ls /proc/sys/fs/binfmt_misc/arm*
Another guess, doess your qemu-linux-user package which you preinstalled containing the -binfmt wrappers?
We do register them by default, because otherwise argv[0] gets lost which breaks quite some builds.
[Rick Liu] Yes, my pre-built QEMU binary package contains -binfmt wrappers. How to register them? One question here: Since I'm using pre-built qemu-linux-user binary from opensuse, which RPM package I should use to extract the binary from? (even though right now the outcomes are all the same) qemu-linux-user-2.0.0-234.1.i586.rpm qemu-linux-user-2.0.0-234.1.x86_64.rpm qemu-linux-user-2.0.0-234.1.armv7hl.rpm The OBS worker is an OpenSUSE 13.1 x86_64 host trying to build armv7el arch under chroot. What confuses me is that in old days with copy_qemu(), the qemu used under armv7el chroot is from worker host and is x86_64 binary, but with "Preinstall" method (qemu from project repos), I can't install x86_64 RPM package into armv7el chroot. Does that mean I should extract x86_64 qemu-linux-user binary and package them into armv7el RPM package? or it doesn't really matter? Rick -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
To be safe, I still install packages below on both OBS server and OBS worker: build 20140424-1.1 build-initvm-x86_64 20140424-1.1 qemu 2.0.0-236.1 qemu-arm 2.0.0-236.1 qemu-kvm 2.0.0-236.1 qemu-linux-user 2.0.0-234.3
The qemu packages should not be needed, if you use kvm or xen workers.
[Rick Liu] In my old obs (2.3.1), it doesn't use kvm or xen workers. It uses plain chroot environment with copying QEMU static binaries from host into chroot. Is it possible to do the same when QEMU comes from project repository?
[ 0s] Building hello for project 'home:rickliu:trunk_mxc' repository 'Trunk_MxC_standard' arch 'armv7el' srcmd5 '113b5e223b7ed410d13a13aa2fa6449b' [ 0s] [ 0s] processing recipe /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec ... [ 0s] running changelog2spec --target rpm --file /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec [ 0s] init_buildsystem --configdir /var/run/obs/worker/1/build/configs -- cachedir /var/cache/build --clean --rpmlist /local/ramdrive/obs/worker/root_1/.build.rpmlist /local/ramdrive/obs/worker/root_1/.build-srcdir/hello.spec build rpmlint-Factory ... [ 0s] registering binfmt handlers for cross build [ 0s] interpreter for 'aarch64' is -1 [ 0s] write: Invalid argument [ 0s] /proc/sys/fs/binfmt_misc/register: write failed [ 0s] /var/run/obs/worker/1/build/qemu-reg: line 7: write failed. Content: :aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64-binfmt:P
IIRC it should not abort here, but continue to register the other emulators. Can you check if the registration did work? ls /proc/sys/fs/binfmt_misc/arm*
[Rick Liu] Which environment do you want check this " ls /proc/sys/fs/binfmt_misc/arm*"? obs server, obs worker, or obs worker's chroot? I'll check all of them for now: obs server: obs-server-005:/proc/sys/fs/binfmt_misc # ls -l /proc/sys/fs/binfmt_misc --w------- 1 root root 0 May 22 10:52 register -rw-r--r-- 1 root root 0 May 22 10:52 status obs worker: obs-worker-009:/proc/sys/fs/binfmt_misc # ls -l /proc/sys/fs/binfmt_misc/arm* -rw-r--r-- 1 root root 0 May 21 13:44 /proc/sys/fs/binfmt_misc/arm -rw-r--r-- 1 root root 0 May 21 13:44 /proc/sys/fs/binfmt_misc/armeb obs worker armv7el chroot: (EMPTY) obs-worker-009:/local/ramdrive/obs/worker/root_1/proc # ls -al drwxr-xr-x 2 root root 4096 May 22 10:53 . dr-xr-xr-x 24 root root 4096 May 22 10:54 .. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
So, we support only qemu builds where qemu comes from some project repository so you can have different versions and also reliable builds in future. (since OBS 2.4)
For newer opensuse 13.1 with OBS 2.5, the copy function is gone, so I create a pre-built QEMU binary package in my project, and added lines below to the "Project Config":
%ifarch armv7el armv8el Hostarch: x86_64 Preinstall: qemu-linux-user-arm %endif
To be safe, I still install packages below on both OBS server and OBS worker: build 20140424-1.1 build-initvm-x86_64 20140424-1.1 qemu 2.0.0-236.1 qemu-arm 2.0.0-236.1 qemu-kvm 2.0.0-236.1 qemu-linux-user 2.0.0-234.3
The qemu packages should not be needed, if you use kvm or xen workers.
[Rick Liu] Instead of using pre-built qemu-linux-user binaries from opensuse and package these binary into RPM in my OBS project's package, I rebuilt it from the source. With the installation of "qemu-linux-user" RPM (directly from opensuse) for each OBS worker machine, now I'm able to do QEMU chroot cross-build in OBS 2.5.2 and OBS 2.5.3. PS. Without the installation of "qemu-linux-user" RPM on each OBS worker machine, I'll still get the build error below: [ 0s] Memory limit set to 35790464KB [ 0s] Using BUILD_ROOT=/local/ramdrive/obs/worker/root_1 [ 0s] Using BUILD_ARCH=armv7el [ 0s] [ 0s] [ 0s] obs-worker-008 started "build perl.spec" at Mon May 26 17:59:27 UTC 2014. [ 0s] [ 0s] Building perl for project 'Trunk:MxC' repository 'standard' arch 'armv7el' srcmd5 'cd27a05addb6451b8124f54e6dc779c8' [ 0s] [ 0s] processing recipe /local/ramdrive/obs/worker/root_1/.build-srcdir/perl.spec ... [ 0s] running changelog2spec --target rpm --file /local/ramdrive/obs/worker/root_1/.build-srcdir/perl.spec [ 0s] init_buildsystem --configdir /var/run/obs/worker/1/build/configs --cachedir /var/cache/build --clean --rpmlist /local/ramdrive/obs/worker/root_1/.build.rpmlist /local/ramdrive/obs/worker/root_1/.build-srcdir/perl.spec build rpmlint-Factory ... [ 0s] registering binfmt handlers for cross build [ 0s] interpreter for 'aarch64' is -1 [ 0s] interpreter for 'aarch64' is -1 [ 0s] interpreter for 'arm' is -1 [ 0s] interpreter for 'armeb' is -1 [ 0s] interpreter for 'ppc' is -1 [ 0s] interpreter for 'mips' is -1 [ 0s] interpreter for 'mipsel' is -1 [ 0s] interpreter for 'sh4' is -1 [ 0s] interpreter for 'sh4eb' is -1 [ 1s] cycle: glibc -> glibc-common -> bash [ 1s] breaking dependency glibc-common -> bash [ 1s] cycle: glibc -> glibc-common [ 1s] breaking dependency glibc -> glibc-common [ 1s] cycle: glibc -> nss-softokn-freebl [ 1s] breaking dependency nss-softokn-freebl -> glibc [ 1s] cycle: libgcc -> glibc -> nss-softokn-freebl [ 1s] breaking dependency glibc -> nss-softokn-freebl [ 1s] cycle: libgcc -> glibc [ 1s] breaking dependency libgcc -> glibc [ 1s] cycle: coreutils -> pam [ 1s] breaking dependency coreutils -> pam [ 1s] cycle: rpm-libs -> rpm [ 1s] breaking dependency rpm -> rpm-libs [ 1s] [1/62] preinstalling setup... [ 1s] [2/62] preinstalling filesystem... ... ... ... [ 5s] running bzip2-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running tar-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running coreutils-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running nss-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running sed-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running diffutils-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running file-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running rpm-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running aaa-meego-accelerator postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] running bash-x86-arm postinstall script [ 5s] chroot: failed to run command 'sh': Exec format error [ 5s] initializing rpm db... [ 5s] chroot: failed to run command '/usr/bin/rpmdb': Exec format error Thanks, Rick -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (2)
-
Adrian Schröter -
Rick Liu