Problems with building multi-arch images
Hey there, I've been trying to get multi-arch images using buildah to work, but I can't seem to figure out how this is supposed to work. And to be honest, the documentation around that topic is a bit lacking too. From reading https://github.com/containers/buildah/issues/1590, I'm fairly certain I've got what I need, but it still doesn't seem to work. First and foremost, a quick recap about my enviroment: - I'm on an x86_64 machine - I'm running this on an up to date install of Void Linux - I've installed buildah from the main Void repos, at version 1.20.1 - I've also installed qemu-user-static, version 5.2.0 Here's my Dockerfile: ```Dockerfile FROM docker.io/alpine RUN arch ``` Now, when I try to build a multi-arch image, I thought I needed to do this: ```sh ➤ buildah bud --pull --manifest test --platform linux/amd64 . ➤ buildah bud --pull --manifest test --platform linux/arm64 . ``` The first image build goes fine, which is expected, as I'm on my native architecture. The second one fails like so: ``` STEP 2: RUN arch standard_init_linux.go:219: exec user process caused: exec format error error building at STEP "RUN arch": error while running runtime: exit status 1 ERRO exit status 1 ``` Am I missing anything obvious? To me it looks like even though I specified the architecture, I'm running this natively without qemu-user-static being involved. I've also tried looking through the `--debug` output, but that's quite verbose and doesn't contain `qemu` anywhere. I'd be very glad to get feedback here, what I'm missing. Thanks in advance :) -- JC
On Tue, Apr 27, 2021 at 06:21:05PM +0200, Jan Christian Grünhage wrote:
Hey there,
I've been trying to get multi-arch images using buildah to work, but I can't seem to figure out how this is supposed to work. And to be honest, the documentation around that topic is a bit lacking too. From reading https://github.com/containers/buildah/issues/1590, I'm fairly certain I've got what I need, but it still doesn't seem to work.
First and foremost, a quick recap about my enviroment: - I'm on an x86_64 machine - I'm running this on an up to date install of Void Linux - I've installed buildah from the main Void repos, at version 1.20.1 - I've also installed qemu-user-static, version 5.2.0
Here's my Dockerfile:
```Dockerfile FROM docker.io/alpine RUN arch ```
Now, when I try to build a multi-arch image, I thought I needed to do this:
```sh ➤ buildah bud --pull --manifest test --platform linux/amd64 . ➤ buildah bud --pull --manifest test --platform linux/arm64 . ```
The first image build goes fine, which is expected, as I'm on my native architecture. The second one fails like so:
``` STEP 2: RUN arch standard_init_linux.go:219: exec user process caused: exec format error error building at STEP "RUN arch": error while running runtime: exit status 1 ERRO exit status 1 ```
Am I missing anything obvious? To me it looks like even though I specified the architecture, I'm running this natively without qemu-user-static being involved.
You're running buildah itself natively, but the "arch" binary in the base image that's being invoked for that RUN instruction is compiled for arm64, and that means you need the help of an emulator to execute it. A good rule of thumb is that RUN instructions (or "buildah run" invocations) when the base image is built for a foreign architecture will need help from an emulator. HTH, Nalin
You're running buildah itself natively, but the "arch" binary in the base image that's being invoked for that RUN instruction is compiled for arm64, and that means you need the help of an emulator to execute it.
Yeah, I understand that so far. The `arch` binary in the container image is not my native architecture, so it needs emulation. But *where* do I insert that emulator? And how? I can't just run `buildah` with `qemu-aarch64-static`, because `buildah` is for my native architecture. I also can't add `qemu-aarch64-static` in the Dockerfile, that'd be pointless (and also wouldn't work, because it'd look for the binary inside the container image). Looking at https://github.com/containers/buildah/issues/1590#issuecomment-771924743 for example, I don't see any manual step for invoking an emulator, leading me to believe that this is something buildah takes care of for me, assuming that `qemu-user-static` is installed (which it is, ftr). Any more hints? -- JC
On 4/27/21 14:59, Jan Christian Grünhage wrote:
You're running buildah itself natively, but the "arch" binary in the base image that's being invoked for that RUN instruction is compiled for arm64, and that means you need the help of an emulator to execute it. Yeah, I understand that so far. The `arch` binary in the container image is not my native architecture, so it needs emulation. But *where* do I insert that emulator? And how? I can't just run `buildah` with `qemu-aarch64-static`, because `buildah` is for my native architecture. I also can't add `qemu-aarch64-static` in the Dockerfile, that'd be pointless (and also wouldn't work, because it'd look for the binary inside the container image).
Looking at https://github.com/containers/buildah/issues/1590#issuecomment-771924743 for example, I don't see any manual step for invoking an emulator, leading me to believe that this is something buildah takes care of for me, assuming that `qemu-user-static` is installed (which it is, ftr).
Any more hints?
-- JC _______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io
With the emulator software installed, the kernel figures out that the arch is different and runs the emulator. I am not quite sure how all this magic happens, but Buildah is not involved, it is just the beneficiary.
With the emulator software installed, the kernel figures out that the arch is different and runs the emulator. I am not quite sure how all this magic happens, but Buildah is not involved, it is just the beneficiary.
So my problem here is most likely to be caused by my distro not hooking qemu-user-static into the kernel properly? Because I do have qemu-user-static installed on my machine.
On Tue, Apr 27, 2021 at 03:37:37PM -0400, Daniel Walsh wrote:
With the emulator software installed, the kernel figures out that the arch is different and runs the emulator. I am not quite sure how all this magic happens, but Buildah is not involved, it is just the beneficiary.
The Fedora version of the package drops configuration files into /usr/lib/binfmt.d/ for each emulator in the package, and systemd's systemd-binfmt.service uses the information they contain to configure the kernel's binfmt_misc module to invoke the right interpreter for the binaries that they can handle, usually at boot-time. That registration is also triggered when the package is installed. Those config files are generated by a script that's part of the qemu source tree [1], and it can configure /proc directly if that's closer to what you need. HTH, Nalin [1] https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh
On 4/27/21 4:20 PM, Nalin Dahyabhai wrote:
With the emulator software installed, the kernel figures out that the arch is different and runs the emulator. I am not quite sure how all this magic happens, but Buildah is not involved, it is just the beneficiary. The Fedora version of the package drops configuration files into /usr/lib/binfmt.d/ for each emulator in the package, and systemd's systemd-binfmt.service uses the information they contain to configure
On Tue, Apr 27, 2021 at 03:37:37PM -0400, Daniel Walsh wrote: the kernel's binfmt_misc module to invoke the right interpreter for the binaries that they can handle, usually at boot-time. That registration is also triggered when the package is installed.
Those config files are generated by a script that's part of the qemu source tree [1], and it can configure /proc directly if that's closer to what you need.
HTH,
Nalin
[1] https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh _______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io
If someone could write up a quick summary of the steps needed in a new Podman Discusssion (https://github.com/containers/podman/discussions) to make this work, that would most likely be helpful to other folks. t
The Fedora version of the package drops configuration files into /usr/lib/binfmt.d/ for each emulator in the package, and systemd's systemd-binfmt.service uses the information they contain to configure the kernel's binfmt_misc module to invoke the right interpreter for the binaries that they can handle, usually at boot-time. That registration is also triggered when the package is installed.
Those config files are generated by a script that's part of the qemu source tree [1], and it can configure /proc directly if that's closer to what you need.
Okay, this was quite helpful, figured out the binfmt error. Void Linux has a similar service to the systemd-binfmt.service, which is in a package that's not depended upon by qemu-user-static. Installed that package and reconfigured qemu-user-static, now I don't get binfmt related errors anymore. That doesn't mean that buildah is happy though: ``` ➤ buildah bud --pull --manifest test --platform linux/arm64 . STEP 1: FROM docker.io/alpine Getting image source signatures Copying blob 595b0fe564bb done Copying config 3fcaaf3dc9 done Writing manifest to image destination Storing signatures STEP 2: RUN arch standard_init_linux.go:219: exec user process caused: no such file or directory error building at STEP "RUN arch": error while running runtime: exit status 1 ERRO exit status 1 ``` That's still not super helpful, not sure how to continue here. I'll try this again in a VM tomorrow, on a Fedora machine or a Debian machine, if that works out I'll go bother the Void people instead. Thanks for all the help so far, learned a lot today :) JC
On Tue, Apr 27, 2021 at 10:41:04PM +0200, Jan Christian Grünhage wrote:
Okay, this was quite helpful, figured out the binfmt error. Void Linux has a similar service to the systemd-binfmt.service, which is in a package that's not depended upon by qemu-user-static. Installed that package and reconfigured qemu-user-static, now I don't get binfmt related errors anymore. That doesn't mean that buildah is happy though:
``` ➤ buildah bud --pull --manifest test --platform linux/arm64 . STEP 1: FROM docker.io/alpine Getting image source signatures Copying blob 595b0fe564bb done Copying config 3fcaaf3dc9 done Writing manifest to image destination Storing signatures STEP 2: RUN arch standard_init_linux.go:219: exec user process caused: no such file or directory error building at STEP "RUN arch": error while running runtime: exit status 1 ERRO exit status 1 ```
That's still not super helpful, not sure how to continue here. I'll try this again in a VM tomorrow, on a Fedora machine or a Debian machine, if that works out I'll go bother the Void people instead.
I took another look at Fedora's packaging, and its qemu-user-static munges the configurations to use copies of the emulators that are statically linked, and registers them with the "F" flag. The "F" flag lets the kernel successfully run binaries under the emulators when the mount namespaces that those binaries are in don't have the emulators in them [1], and I expect (but haven't personally verified by tweaking my own system to check) that the static linking prevents similar problems due to a lack of specific shared libraries that a dynamically linked emulator would have been linked with. HTH, Nalin [1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
On 4/27/21 12:21, Jan Christian Grünhage wrote:
Hey there,
I've been trying to get multi-arch images using buildah to work, but I can't seem to figure out how this is supposed to work. And to be honest, the documentation around that topic is a bit lacking too. From reading https://github.com/containers/buildah/issues/1590, I'm fairly certain I've got what I need, but it still doesn't seem to work.
First and foremost, a quick recap about my enviroment: - I'm on an x86_64 machine - I'm running this on an up to date install of Void Linux - I've installed buildah from the main Void repos, at version 1.20.1 - I've also installed qemu-user-static, version 5.2.0
Here's my Dockerfile:
```Dockerfile FROM docker.io/alpine RUN arch ```
Now, when I try to build a multi-arch image, I thought I needed to do this:
```sh ➤ buildah bud --pull --manifest test --platform linux/amd64 . ➤ buildah bud --pull --manifest test --platform linux/arm64 . ```
The first image build goes fine, which is expected, as I'm on my native architecture. The second one fails like so:
``` STEP 2: RUN arch standard_init_linux.go:219: exec user process caused: exec format error error building at STEP "RUN arch": error while running runtime: exit status 1 ERRO exit status 1 ```
Am I missing anything obvious? To me it looks like even though I specified the architecture, I'm running this natively without qemu-user-static being involved.
I've also tried looking through the `--debug` output, but that's quite verbose and doesn't contain `qemu` anywhere.
I'd be very glad to get feedback here, what I'm missing. Thanks in advance :)
-- JC _______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io
You need to install an emulator. On Fedora the package is qemu-user-static. rpm -q qemu-user-static qemu-user-static-5.2.0-5.fc34.1.x86_64
participants (4)
-
Daniel Walsh -
Jan Christian Grünhage -
Nalin Dahyabhai -
Tom Sweeney