[Podman] Re: Multi-arch with podman manifests

On Wed, Nov 11, 2020 at 10:27:49AM +0100, Alexander Wellbrock via Podman wrote: > Hey there, since buildx is not a thing yet to build multi-arch images on one host I'd like to make use of the multi-arch feature of registries like docker.io and quay.io. > > I'm using the podman manifest command for this purpose but struggle to get it to work properly. I hope someone can point me in the right direction. > > I've get it to work (kind-of) by building two images on two different hosts architecture wise and then used podman manifest create to create a multi-arch manifest of both images. I then used podman manifest push to get it onto quay.io. This works and I'm able to use it, but unfortunately there is a catch right now. > > If I'm doing this multiple times, it complains that since a manifest for example-image:latest is already present. So it only work if the manifest does not already exist. I can't update it with newer images. Only if I remove the manifest :latest manually beforehand it obviously works again, but I'd like to avoid that. Apart from that it makes sense I suppose, because I'd rather like to update the manifest with new images then to override the old manifest, right? > > I then tried to use podman manifest add, remove etc. Podman then complains about manifest version not supported. E.g. I tried to update the actual images per architecture and then add the new ones to the manifest which I couldn't get to work. I'm highly confused about the podman manifest command man page. Someone asked a similar question on IRC a few days ago, and I didn't have a good answer then, so had to poke at it a bit. Anyway, the sequence of steps for creating a list using images that have previously been pushed to a registry looks something like this: on a box of each $arch: podman build -t docker://registry/repo:$arch-tag on one box: podman manifest create $listname repeat for each $arch on that one box: podman manifest add $listname docker://registry/repo:$arch-tag finally, on that one box: podman manifest push [--all] $listname docker://registry/repo:$single-tag Now, if you build a new image and want to replace it in the list, you should be able to start with the list you already have, or you can do this to start to build a new, updated version of the list: podman manifest create updated-list podman manifest add --all updated-list docker://registry/repo:$single-tag Note: "podman manifest add" (and "buildah manifest add") looks like it just looks at a locally-stored image if it finds one with the same name as the reference being passed on the command line. That locally-stored image isn't going to include all of the other entries in the copy of the list that the registry has, and that can cause some problems. I think we'll need to fix that. If it works as intended (or you started with the list you already had), you should have a copy of the list that's up at the registry, and "podman manifest inspect" should show you several entries in that list. The spec doesn't say anything about what to do when there are multiple entries in a list for the same platform, so we'll want to remove the entry that the list has for the current machine before we try to add our newly-built image to the list as its replacement. arch=$(go env GOARCH) instance=$(podman manifest inspect updated-list | jq -r \ '.manifests|map(select(.platform.architecture=="'$arch'"))[].digest') podman manifest remove updated-list $instance Now you can go ahead and add your new image to the list and push the updated list to the registry. If the new image has already been pushed to the registry: podman manifest add updated-list docker://registry/repo:$arch-tag podman manifest push updated-list docker://registry/repo:$single-tag If not: podman manifest add updated-list local-$arch-image podman manifest push --all updated-list docker://registry/repo:$single-tag The main difference between these two cases is that the --all flag tells podman to push the images that the list references, in addition to the list itself. That can take a while, since for every image, even though the pushing logic checks if a part of an image has already been uploaded to the registry before trying to upload it, it's still a lot of parts, and that can require a lot of round trips to the registry. But if the image wasn't pushed first, the registry will probably reject the new list because it references things that the registry doesn't have, so it's required. I've got a couple of ideas of things we can do to help simplify this process, so that it requires less typing at the CLI, but none of that's implemented yet. If you run into trouble, please follow up with the specific errors you're getting, and we'll see where they point us. Cheers, Nalin