Building multi-arch containers using buildah: How to properly setup binfmt?
Hi all, can someone untangle this knot for me? I am stuck and am missing an important piece, it seems: I want to build multi-arch containers using buildah. https://podman.io/blogs/2021/10/11/multiarch.html buildah build --jobs=4 --platform=linux/amd64,linux/ppc64le,linux/arm64,linux/s390x --manifest shazam . For this I need to have qemu-emulation available, as far as I understood. So I installed qemu-linux-user and issued the following commands: qemu-binfmt-conf.sh --systemd aarch64 qemu-binfmt-conf.sh --systemd s390x qemu-binfmt-conf.sh --systemd arm I then restarted the systemd-binfmt service and checked in /proc/sys/fs/binfmt_misc: # ls /proc/sys/fs/binfmt_misc qemu-aarch64 qemu-arm qemu-s390x register status # However, building for amd64 works, but fails for both arm64 and s390x.
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a exec /bin/sh: no such file or directory error building at STEP "RUN uname -a": error while running runtime: exit status 1 ERRO[0003] exit status 1
I found multiple examples running "RUN arch" or similar in the Containerfile, and I tried all of them. And each time I get the error on non-amd64. Is there another step I forgot? Is there documentation on enabling the qemu-binfmt-thingy that I did not find? I know that the error says "no such file or directory", I think this just means that the shell could not be executed due to arch-mismatch. I got some "exec format error" messages when trying some golang examples. Thanks for your help! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Moin, Am Samstag, 5. März 2022, 21:39:16 CET schrieb Johannes Kastl:
Hi all,
can someone untangle this knot for me? I am stuck and am missing an important piece, it seems:
I want to build multi-arch containers using buildah. https://podman.io/blogs/2021/10/11/multiarch.html
buildah build --jobs=4 --platform=linux/amd64,linux/ppc64le,linux/arm64,linux/s390x --manifest shazam .
For this I need to have qemu-emulation available, as far as I understood.
So I installed qemu-linux-user and issued the following commands:
qemu-binfmt-conf.sh --systemd aarch64 qemu-binfmt-conf.sh --systemd s390x qemu-binfmt-conf.sh --systemd arm
Try with "--persistent". Otherwise it tries to run the registered handler from inside the container, but e.g. /usr/bin/qemu-aarch64 doesn't exist in that environment. Cheers, Fabian
I then restarted the systemd-binfmt service and checked in /proc/sys/fs/binfmt_misc: # ls /proc/sys/fs/binfmt_misc qemu-aarch64 qemu-arm qemu-s390x register status #
However, building for amd64 works, but fails for both arm64 and s390x.
$ buildah bud --arch arm64 . STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a exec /bin/sh: no such file or directory error building at STEP "RUN uname -a": error while running runtime: exit status 1 ERRO[0003] exit status 1
I found multiple examples running "RUN arch" or similar in the Containerfile, and I tried all of them. And each time I get the error on non-amd64.
Is there another step I forgot? Is there documentation on enabling the qemu-binfmt-thingy that I did not find?
I know that the error says "no such file or directory", I think this just means that the shell could not be executed due to arch-mismatch. I got some "exec format error" messages when trying some golang examples.
Thanks for your help!
Kind Regards, Johannes
-- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de
B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi Fabian, On 05.03.22 at 21:59 Fabian Vogt wrote:
So I installed qemu-linux-user and issued the following commands:
qemu-binfmt-conf.sh --systemd aarch64 qemu-binfmt-conf.sh --systemd s390x qemu-binfmt-conf.sh --systemd arm
Try with "--persistent". Otherwise it tries to run the registered handler from inside the container, but e.g. /usr/bin/qemu-aarch64 doesn't exist in that environment.
First off, thanks for the reply. I stopped the systemd-binfmt service. /proc/sys/fs/binfmt_misc/ was "empty" again after this (i.e. no qemu-things in it). I issued "/sbin/qemu-binfmt-conf.sh --persistent" which spit out an error, but did something nonetheless:
# /sbin/qemu-binfmt-conf.sh --persistent getopt: option '--persistent' requires an argument Setting /usr/bin/qemu-alpha-binfmt as binfmt interpreter for alpha
(I think it should error out and do nothing in case the argument is missing) Buildah did not work. I restarted systemd-binfmt and stopped it again to get back to a clean /proc/sys/fs/binfmt_misc/. I then tried "/sbin/qemu-binfmt-conf.sh --persistent yes" which seems to do something without the error. The buildah call still does not work, but now I get another error message... :-) Previous:
STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a exec /bin/sh: no such file or directory error building at STEP "RUN uname -a": error while running runtime: exit status 1 ERRO[0002] exit status 1
Now:
STEP 1/2: FROM registry.access.redhat.com/ubi8:latest STEP 2/2: RUN uname -a error building at STEP "RUN uname -a": error while running runtime: exit status 255 ERRO[0003] exit status 255
Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi all, On 06.03.22 at 09:23 Johannes Kastl wrote:
I then tried "/sbin/qemu-binfmt-conf.sh --persistent yes" which seems to do something without the error.
The buildah call still does not work, but now I get another error message... :-)
No one able to help out? I think it might be something obvious, but I am missing it and am stuck and this is frustrating... I even tried a reboot. :-) Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
------- Original Message ------- On Tuesday, March 8th, 2022 at 11:54 PM, Johannes Kastl <kastl@b1-systems.de> wrote:
Hi all,
On 06.03.22 at 09:23 Johannes Kastl wrote:
I then tried "/sbin/qemu-binfmt-conf.sh --persistent yes" which seems to do
something without the error.
The buildah call still does not work, but now I get another error message... :-)
No one able to help out?
I think it might be something obvious, but I am missing it and am stuck and this
is frustrating...
I even tried a reboot. :-)
Johannes
--
Johannes Kastl
Linux Consultant & Trainer
Tel.: +49 (0) 151 2372 5802
Mail: kastl@b1-systems.de
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg
GF: Ralph Dehner
Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi Johannes, I had similar experiences as well with multi-arch and couldn't get it sorted yet. `sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process. I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe. -- Br, A.
------- Original Message ------- On Wednesday, March 9th, 2022 at 12:39 PM, Attila Pinter <adathor@protonmail.com> wrote:
Hi Johannes,
I had similar experiences as well with multi-arch and couldn't get it sorted yet.
`sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process.
I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe.
Forget to add the link to the outputs.... https://paste.opensuse.org/93021367
------- Original Message ------- On Wednesday, March 9th, 2022 at 12:46 PM, Attila Pinter <adathor@protonmail.com> wrote:
------- Original Message -------
On Wednesday, March 9th, 2022 at 12:39 PM, Attila Pinter adathor@protonmail.com wrote:
Hi Johannes,
I had similar experiences as well with multi-arch and couldn't get it sorted yet.
`sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process.
I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe.
User error OFC, solved the binfmt issue by issuing the correct commands for qemu-binfmt-conf.sh: sudo /sbin/qemu-binfmt-conf.sh --persistent yes --systemd ALL. Still can't build tho ^_^ -- Br, A.
------- Original Message -------
On Wednesday, March 9th, 2022 at 12:46 PM, Attila Pinter adathor@protonmail.com wrote:
------- Original Message -------
On Wednesday, March 9th, 2022 at 12:39 PM, Attila Pinter adathor@protonmail.com wrote:
Hi Johannes,
I had similar experiences as well with multi-arch and couldn't get it sorted yet.
`sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process.
I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe.
User error OFC, solved the binfmt issue by issuing the correct commands for qemu-binfmt-conf.sh: sudo /sbin/qemu-binfmt-conf.sh --persistent yes --systemd ALL. Still can't build tho ^_^
I'm stuck here: https://paste.opensuse.org/56810307 -- Br, A.
Hi Attila, On 09.03.22 at 06:39 Attila Pinter wrote:
I had similar experiences as well with multi-arch and couldn't get it sorted yet. `sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process.
I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe.
Thanks for sharing your experience, at least I am not the only one where this is not working... Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
------- Original Message ------- On Thursday, March 10th, 2022 at 3:33 PM, Johannes Kastl <kastl@b1-systems.de> wrote:
Hi Attila,
On 09.03.22 at 06:39 Attila Pinter wrote:
I had similar experiences as well with multi-arch and couldn't get it sorted yet.
`sudo /sbin/qemu-binfmt-conf.sh --persistent yes` worked for me, however, systemd-binfmt.service died in the process.
I'm pretty much a newbie building multi-arch containers, but something tells me that the issue might be with binfmt maybe.
Thanks for sharing your experience, at least I am not the only one where this is
not working...
Johannes
Something tells me that the problem is between the keyboard and the chair somewhere ^_^ Plus it is likely a bit poorly documented. -- Br, A.
Hi Attila, On 11.03.22 at 10:26 Attila Pinter wrote:
Something tells me that the problem is between the keyboard and the chair somewhere ^_^
That might well be, yes, and I would be glad if someone could just point me to the error that I made or the step that I missed. :-)
Plus it is likely a bit poorly documented.
I asked this again on the Factory mailinglist, maybe some there has an idea or spots what we have missed. Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
That might well be, yes, and I would be glad if someone could just point me to the error that I made or the step that I missed. :-)
Would love that too ^_^ Got holiday till next Wednesday, will play with it and share my findings if anything changes. Will also try on Fedora to make sure the problem is not on our side. -- Br, A.
Hi all, On 05.03.22 at 21:39 Johannes Kastl wrote:
Hi all,
can someone untangle this knot for me? I am stuck and am missing an important piece, it seems:
I want to build multi-arch containers using buildah. https://podman.io/blogs/2021/10/11/multiarch.html
For the record, on the Factory mailinglist a workaround was proposed. Until the downstream patches that apparently broke the persistent feature in qemu-linux-user and binfmt are removed from the qemu package, the workaround is: qemu-binfmt-conf.sh --systemd $ARCH --persistent yes --qemu-suffix "" I tested this workaround successfully on some of the podman examples. Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
------- Original Message ------- On Monday, March 21st, 2022 at 6:43 PM, Johannes Kastl <kastl@b1-systems.de> wrote:
For the record, on the Factory mailinglist a workaround was proposed.
Until the downstream patches that apparently broke the persistent feature in
qemu-linux-user and binfmt are removed from the qemu package, the workaround is:
qemu-binfmt-conf.sh --systemd $ARCH --persistent yes --qemu-suffix ""
I tested this workaround successfully on some of the podman examples.
This is great, thanks for sharing! I fell ill myself and couldn't continue digging deeper. -- Br, A.
participants (3)
-
Attila Pinter -
Fabian Vogt -
Johannes Kastl