[Bug 1143725] New: qemu-aarch64-binfmt, from qemu-linux-user, not working (no output)
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 Bug ID: 1143725 Summary: qemu-aarch64-binfmt, from qemu-linux-user, not working (no output) Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: bnc-team-screening@forge.provo.novell.com Reporter: dfaggioli@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Context: I want to run an ARM64 (aarch64) container on an x86 host. This is possible using qemu Linux user emulation and Linux binfmt. It is supposed to work like this: //Fetching an ARM container on x86 (long story) $ skopeo --override-arch arm64 copy docker://opensuse/leap docker-daemon:opensuse-arm64/leap:latest //Install qemu Linux user, and setup binfmt for aarch64 # zypper install qemu-linux-user $ cp /usr/bin/qemu-aarch64-binfmt . # echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64 # echo ':qemu-aarch64:M:0:\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-binfmt:POCF'
/proc/sys/fs/binfmt_misc/register
//Build our container $ cat << EOF > Dockerfile FROM opensuse-arm64/leap COPY qemu-aarch64-binfmt /usr/bin/qemu-aarch64-binfmt EOF docker build -t qemu-leap . //Now I should be able to run the ARM container, like this: # docker run -it qemu-leap-aarch64-binfmt /bin/sh But, basically, nothing happens. No output at all, and I'm definitely not inside an ARM container. Actually, it appears the container exited: $ docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ea2252caf90a qemu-leap "/bin/sh" 4 seconds ago Exited (255) 3 seconds ago relaxed_goodall Now, if I, instead of qemu-aarch64-binfmt, use a qemu-aarch64-static binary copied from a Debian unstable box I have in my home network, i.e., this one: # ./qemu-aarch64-static -version qemu-aarch64 version 3.1.0 (Debian 1:3.1+dfsg-8) Things work! In fact: $ docker run -it --rm qemu-leap uname -a Linux bf12752f6a51 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux Note the *aarch64*. Of course, we need to change the aarch64 interpreter file too, in /proc/sys/fs/binfmt So, basically, it looks like the procedure is correct, or at least can work. But our qemu-linux-user packages contain binaries that are not adequate for being used for this use case. Things/differences to check: - QEMU version (the debian stolen from Debian, that makes things works is from QEMU 3.1.0, while we ship 4.0.0 - our patch queue - our configuration and compile options -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 Dario Faggioli <dfaggioli@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P4 - Low CC| |brogers@suse.com, | |dfaggioli@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c1 --- Comment #1 from Dario Faggioli <dfaggioli@suse.com> --- Note that is doesn't look like it could be a "container specific" thing. In fact, I tried to setup a "traditional" ARM64 chroot, on the x86 host, again using qemu-linux-user and binfmt (basically, how it is described here: https://en.opensuse.org/HCL:Chroot) and I run into the same issue. BTW, adding brogers@suse.com to Cc, as this is about QEMU. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 Dario Faggioli <dfaggioli@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bnc-team-screening@forge.pr |dfaggioli@suse.com |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c2 --- Comment #2 from Dario Faggioli <dfaggioli@suse.com> --- And I've just checked that building (statically) qemu-aarch64, from the upstream git tree, works as well. $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64-upstream flags: OCF offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff $ /usr/bin/qemu-aarch64-upstream --version qemu-aarch64 version 4.0.93 (v4.1.0-rc3-dirty) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers $ docker run -it --rm qemu-leap /bin/bash :/ # uname -a Linux cf5f2385946e 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux :/ # /usr/bin/qemu-aarch64-upstream --version qemu-aarch64 version 4.0.93 (v4.1.0-rc3-dirty) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers I think this is a step toward ruling out QEMU version issues. Now our patch-queue and our config options remain to be checked. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 Dario Faggioli <dfaggioli@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Basesystem |KVM Found By|--- |Development -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c3 Claudio Fontana <claudio.fontana@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |claudio.fontana@suse.com --- Comment #3 from Claudio Fontana <claudio.fontana@suse.com> --- does running a simple AArch64 binary with qemu user mode work? (no container..?) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c4 --- Comment #4 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Claudio Fontana from comment #3)
does running a simple AArch64 binary with qemu user mode work? (no container..?)
It does. In fact, using the stock qemu-aarch64-binfmt interpreter, as provided by our qemu-linux-user package: $ cat pippo.c #include <stdio.h> int main() { int a=7; printf("a=%d\n", a); return a++; } $ uname -a Linux Palanthas 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) x86_64 x86_64 x86_64 GNU/Linux $ file pippo-static pippo-static: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=f0839e4cd8142d9b85cb41d3cac9afc4b82403ed, for GNU/Linux 3.7.0, with debug_info, not stripped $ ./pippo-static ; echo "pippo-static returned $?" a=7 pippo-static returned 7 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c5 --- Comment #5 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Dario Faggioli from comment #1)
Note that is doesn't look like it could be a "container specific" thing. In fact, I tried to setup a "traditional" ARM64 chroot, on the x86 host, again using qemu-linux-user and binfmt (basically, how it is described here: https://en.opensuse.org/HCL:Chroot) and I run into the same issue.
BTW, adding brogers@suse.com to Cc, as this is about QEMU.
And I was wrong here. A "traditional chroot", works: $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64-binfmt flags: P offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff # chroot src/rootfs/ARM/openSUSE/Tumbleweed/ /bin/bash Palanthas:/ # uname -a Linux Palanthas 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux So, this, together with Comment 4 , seems to suggest that it is something related to the interaction of qemu-aarch64-binfmt (from qemu-linux-user) and docker (or some other container-related component) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c6 Arnav Singh <opensuse@arnavion.dev> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |opensuse@arnavion.dev --- Comment #6 from Arnav Singh <opensuse@arnavion.dev> --- (qemu-linux-user 7.0.0-55.1) I had the same issue with binaries exiting with code 255. I found that using qemu-aarch64 instead works perfectly: $ echo ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:POCF'
/proc/sys/fs/binfmt_misc/register
$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64 flags: POCF offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff Now Docker containers are able to run aarch64 binaries just fine. I did not test with a chroot to see if it behaves differently. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1143725 http://bugzilla.opensuse.org/show_bug.cgi?id=1143725#c9 --- Comment #9 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1143725) was mentioned in https://build.opensuse.org/request/show/1008827 Factory / qemu -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com