Building multi-arch containers using buildah: How to properly setup binfmt?
Good morning, I asked this on the kubic list already, but we found no solution, hence once again in a broader round: How to setup binfmt properly so one can build multi-arch containers with buildah/podman or docker buildx? https://podman.io/blogs/2021/10/11/multiarch.html This is what I found and tested so far: 1. Try to build without any binfmt things:
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a exec /bin/sh: no such file or directory error building at STEP "RUN uname -a": error while running runtime: exit status 1 ERRO[0003] exit status 1
2. Setup binfmt Install qemu-linux-user. Check the proc directory:
# ll /proc/sys/fs/binfmt_misc/ total 0 --w------- 1 root root 0 Mar 10 09:45 register -rw-r--r-- 1 root root 0 Mar 10 09:45 status
Run "/sbin/qemu-binfmt-conf.sh --persistent yes --systemd ALL" followed by "systemctl start systemd-binfmt":
# ll /proc/sys/fs/binfmt_misc/ total 0 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-aarch64 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-aarch64_be -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-alpha -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-arm -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-armeb -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-hexagon -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-hppa -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-m68k -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-microblaze -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-microblazeel -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mips -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mips64 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mips64el -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mipsel -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mipsn32 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-mipsn32el -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-or1k -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-ppc -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-ppc64 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-ppc64le -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-riscv32 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-riscv64 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-s390x -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-sh4 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-sh4eb -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-sparc -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-sparc32plus -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-sparc64 -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-xtensa -rw-r--r-- 1 root root 0 Mar 11 08:42 qemu-xtensaeb --w------- 1 root root 0 Mar 10 09:45 register -rw-r--r-- 1 root root 0 Mar 10 09:45 status
3. Try building again The buildah call still does not work, but now I get another error message...
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a error building at STEP "RUN uname -a": error while running runtime: exit status 255 ERRO[0003] exit status 255
I do not want to run a binfmt-docker-container that "does things" to install it, but I failed to find a nice tutorial on how to set this up (on openSUSE). I found lots of outdated tutorials for Ubuntu, and it seems to boil down to installing the qemu-binfmt package and then enable things. Does anyone have a link at hand that I missed in my searches? Which step did I miss? OS is Tumbleweed 20220308 Thanks in advance! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 11.03.22 at 08:49 Johannes Kastl wrote:
The buildah call still does not work, but now I get another error message...
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a error building at STEP "RUN uname -a": error while running runtime: exit status 255 ERRO[0003] exit status 255
I opened both a bugzilla for qemu as well as an issue with buildah upstream. More information on the binfmt thingy would be nice, and a check to see if it is set up properly would be even better. https://bugzilla.opensuse.org/show_bug.cgi?id=1197298 https://github.com/containers/buildah/issues/3827 Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi everyone, On 18.03.22 at 17:47 Johannes Kastl wrote:
On 11.03.22 at 08:49 Johannes Kastl wrote:
The buildah call still does not work, but now I get another error message...
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a error building at STEP "RUN uname -a": error while running runtime: exit status 255 ERRO[0003] exit status 255
I opened both a bugzilla for qemu as well as an issue with buildah upstream. More information on the binfmt thingy would be nice, and a check to see if it is set up properly would be even better.
https://bugzilla.opensuse.org/show_bug.cgi?id=1197298 https://github.com/containers/buildah/issues/3827
The result from the buildah bugreport is that apparently the binaries in the qemu-linux-user package are dynamically linked and thus not working in this scenario. (I cannot verify if that is true, but I believe it is.) An alternative would be to run a privileged container that then registers the binaries: podman run --rm --privileged docker.io/multiarch/qemu-user-static --reset --persistent yes I just tested this on a Tumbleweed VM and could successfully "build" the examples from the podman example. Question to the ones with more knowledge on qemu and related packages: Could we have a subpackage of qemu containing statically linked binaries for this usecase? Or change the existing package to contain statically linked binaries? Or is that already existing in OBS but not in Tumbleweed? Or it is but I did not find it? I don't like the idea of having to run a container with root permissions on each boot of the OS just to register those binfmt thingies. Thanks in advance and have a nice weekend! Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Fri, Mar 18, 2022 at 6:12 PM Johannes Kastl <kastl@b1-systems.de> wrote:
Could we have a subpackage of qemu containing statically linked binaries for this usecase?
Almost any non-trivial program needs or needed shared glibc at runtime.. That's the reason the docker container you linked to has a :glibc variant which is not fully statucally linked.
Hi Cristian, On 19.03.22 at 14:26 Cristian Rodríguez wrote:
On Fri, Mar 18, 2022 at 6:12 PM Johannes Kastl <kastl@b1-systems.de> wrote:
Could we have a subpackage of qemu containing statically linked binaries for this usecase?
Almost any non-trivial program needs or needed shared glibc at runtime.. That's the reason the docker container you linked to has a :glibc variant which is not fully statucally linked.
Thanks for the reply, but what exactly does that help me? :-) I just used the terms from the buildah bug report. If "statically" linked is not the right term, then please excuse this inaccuracy. Shall we agree on "qemu-linux-user binaries that are compiled in a way that they work also for the use case of building multi-arch containers using binfmt"? SCNR Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi Cristian, hi all, On 19.03.22 at 15:30 Johannes Kastl wrote: [ binaries usable for multi-arch containers] Sorry if my last comment came off rude, that was not intended. And I'll have to eat my words anyway. The binaries are already statically linked. In the meantime the other bugreport, the openSUSE one, got a reply from Fabian stating that while the binaries are already statically linked, there is a downstream patch that introduces a regression. https://bugzilla.opensuse.org/show_bug.cgi?id=1197298 So it looks like we have some progress on this. Have a nice weekend, everyone! Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 3/19/22 14:26, Cristian Rodríguez wrote:
On Fri, Mar 18, 2022 at 6:12 PM Johannes Kastl <kastl@b1-systems.de> wrote:
Could we have a subpackage of qemu containing statically linked binaries for this usecase?
Almost any non-trivial program needs or needed shared glibc at runtime.. That's the reason the docker container you linked to has a :glibc variant which is not fully statucally linked.
Debian has been shipping statically linked qemu-user binaries for quite a while now to be used within a chroot. Adrian
On Sun, Mar 20, 2022 at 4:24 AM John Paul Adrian Glaubitz <adrian.glaubitz@suse.com> wrote:
Debian has been shipping statically linked qemu-user binaries for quite a while now to be used within a chroot.
As long as it doesn't end calling nss then it will work. otherwise bets are off.
On Mär 20 2022, John Paul Adrian Glaubitz wrote:
Debian has been shipping statically linked qemu-user binaries for quite a while now to be used within a chroot.
As does openSUSE. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
On 3/11/22 08:49, Johannes Kastl wrote:
Good morning,
I asked this on the kubic list already, but we found no solution, hence once again in a broader round:
How to setup binfmt properly so one can build multi-arch containers with buildah/podman or docker buildx? https://podman.io/blogs/2021/10/11/multiarch.html
For 'docker buildx' you can use this, prior to running 'docker buildx' docker run --privileged --rm tonistiigi/binfmt --install arm64
Hi Michael, thanks for your answer! On 19.03.22 at 14:38 Michael Pujos wrote:
For 'docker buildx' you can use this, prior to running 'docker buildx'
docker run --privileged --rm tonistiigi/binfmt --install arm64
That is basically the same as the workaround that upstream recommended to work around the limitations that apparently come from the qemu-linux-user binaries in Tumbleweed (that are "dynamically linked" and this breaks the use case). I do not know how you feel, but I would strongly prefer to have a package from openSUSE, installed via RPM, that does the magic for me, rather than having to run a docker container with root permissions that is modifying my system, each time I need to build a multi-arch image... :-) Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Agreed, it would be better for a native solution, with stock TW qemu binfmts playing nice with 'docker buildx'. On 3/19/22 15:34, Johannes Kastl wrote:
For 'docker buildx' you can use this, prior to running 'docker buildx'
docker run --privileged --rm tonistiigi/binfmt --install arm64
That is basically the same as the workaround that upstream recommended to work around the limitations that apparently come from the qemu-linux-user binaries in Tumbleweed (that are "dynamically linked" and this breaks the use case).
I do not know how you feel, but I would strongly prefer to have a package from openSUSE, installed via RPM, that does the magic for me, rather than having to run a docker container with root permissions that is modifying my system, each time I need to build a multi-arch image... :-)
Kind Regards, Johannes
On Sat, 2022-03-19 at 15:34 +0100, Johannes Kastl wrote:
I do not know how you feel, but I would strongly prefer to have a package from openSUSE, installed via RPM, that does the magic for me, rather than having to run a docker container with root permissions that is modifying my system, each time I need to build a multi-arch image... :-)
On Factory or SLE15-SP4 beta, you should be able to obtain the desired effect by running qemu-binfmt-conf.sh with '--qemu-suffix ""': qemu-binfmt-conf.sh --systemd $ARCH --persistent yes --qemu-suffix "" Regards Martin
On 21.03.22 at 11:49 Martin Wilck wrote:
On Factory or SLE15-SP4 beta, you should be able to obtain the desired effect by running qemu-binfmt-conf.sh with '--qemu-suffix ""':
qemu-binfmt-conf.sh --systemd $ARCH --persistent yes --qemu-suffix ""
Thanks Martin, I can confirm that this works, at least the podman examples with just using uname as RUN are working for the few architectures I tried. So I seem to have a working solution now, and the qemu package is checked if those patches can be removed. So I'll set this to solved. Thanks everyone for helping! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
participants (6)
-
Andreas Schwab -
Cristian Rodríguez -
Johannes Kastl -
John Paul Adrian Glaubitz -
Martin Wilck -
Michael Pujos