On 16/11/2020 23:23, Nalin Dahyabhai wrote:
On Wed, Nov 11, 2020 at 10:27:49AM +0100, Alexander Wellbrock via
> Hey there, since buildx is not a thing yet to build multi-arch images on one host
I'd like to make use of the multi-arch feature of registries like docker.io and
> I'm using the podman manifest command for this purpose but struggle to get it to
work properly. I hope someone can point me in the right direction.
> I've get it to work (kind-of) by building two images on two different hosts
architecture wise and then used podman manifest create to create a multi-arch manifest of
both images. I then used podman manifest push to get it onto quay.io. This works and
I'm able to use it, but unfortunately there is a catch right now.
> If I'm doing this multiple times, it complains that since a manifest for
example-image:latest is already present. So it only work if the manifest does not already
exist. I can't update it with newer images. Only if I remove the manifest :latest
manually beforehand it obviously works again, but I'd like to avoid that. Apart from
that it makes sense I suppose, because I'd rather like to update the manifest with new
images then to override the old manifest, right?
> I then tried to use podman manifest add, remove etc. Podman then complains about
manifest version not supported. E.g. I tried to update the actual images per architecture
and then add the new ones to the manifest which I couldn't get to work. I'm highly
confused about the podman manifest command man page.
Someone asked a similar question on IRC a few days ago, and I didn't
have a good answer then, so had to poke at it a bit.
Anyway, the sequence of steps for creating a list using images that have
previously been pushed to a registry looks something like this:
on a box of each $arch: podman build -t docker://registry/repo:$arch-tag
on one box: podman manifest create $listname
repeat for each $arch on that one box: podman manifest add $listname
finally, on that one box: podman manifest push [--all] $listname
Now, if you build a new image and want to replace it in the list, you
should be able to start with the list you already have, or you can do
this to start to build a new, updated version of the list:
podman manifest create updated-list
podman manifest add --all updated-list docker://registry/repo:$single-tag
Note: "podman manifest add" (and "buildah manifest add") looks like
just looks at a locally-stored image if it finds one with the same name
as the reference being passed on the command line. That locally-stored
image isn't going to include all of the other entries in the copy of the
list that the registry has, and that can cause some problems. I think
we'll need to fix that.
If it works as intended (or you started with the list you already had),
you should have a copy of the list that's up at the registry, and
"podman manifest inspect" should show you several entries in that list.
The spec doesn't say anything about what to do when there are multiple
entries in a list for the same platform, so we'll want to remove the
entry that the list has for the current machine before we try to add our
newly-built image to the list as its replacement.
arch=$(go env GOARCH)
instance=$(podman manifest inspect updated-list | jq -r \
podman manifest remove updated-list $instance
Now you can go ahead and add your new image to the list and push the
updated list to the registry.
If the new image has already been pushed to the registry:
podman manifest add updated-list docker://registry/repo:$arch-tag
podman manifest push updated-list docker://registry/repo:$single-tag
podman manifest add updated-list local-$arch-image
podman manifest push --all updated-list docker://registry/repo:$single-tag
The main difference between these two cases is that the --all flag tells
podman to push the images that the list references, in addition to the
list itself. That can take a while, since for every image, even though
the pushing logic checks if a part of an image has already been uploaded
to the registry before trying to upload it, it's still a lot of parts,
and that can require a lot of round trips to the registry. But if the
image wasn't pushed first, the registry will probably reject the new
list because it references things that the registry doesn't have, so
I've got a couple of ideas of things we can do to help simplify this
process, so that it requires less typing at the CLI, but none of that's
If you run into trouble, please follow up with the specific errors
you're getting, and we'll see where they point us.
I'm also building multi-arch images with buildah and podman, and I just
would like to add a few remarks.
I use "buildah manifest" instead of "podman manifest". Is there some
behavior differences between these two commands?
If you build your image on an ARM architecture, you may need to add the
--variant option to the "podman manifest add" command:
# example for ARMv7 (like the Raspberry Pi 3)
buildah manifest add --variant 7 image_name:$single-tag
When pushing the manifest to Docker Hub, I had to add the "--format
v2s2" to my "buildah manifest push" command. It seems that Docker Hub
does not support the OCI format and I need to use the v2s2 format (not
100% sure of this):
buildah manifest push --all --format v2s2 image_name:$single-tag
I globally use the same steps as Alexander to push multi-arch images to
- build the image on each architecture (amd64 and armv7 in my case)
- push each image on docker hub with an architecture specific tag
- create and push a manifest as described above
Here is a link to the detailed steps I use to push multi-arch images to