[Podman] Re: Multi-arch with podman manifests

Alexander Wellbrock <a.wellbrock(a)mailbox.org&gt; hat am 17.11.2020 12:09 geschrieben: Finally, I eventually got it to work. Still there are some points I want to raise / remark. So first things first thanks a lot to Nalin and Laurent. Your inputs were just the missing bits I needed. Thanks Nalin for that really good summary validating my approach in general. I especially missed the API change at first of the podman manifest create/add command. Laurent, I don't require --variant 7 to push my images / manifests to quay.io or docker.io. However I added the "--format v2s2" flag to the push command as well as "--all" as suggested by Nalin. I also opened a bug report at quay.io and got the response that my manifest was unsupported by quay.io. One of the two images had an implicit "--format oci" which is apparently not officially supported by quay.io. Only "--format docker" is. Changing this alone however did not work. I don't know why. I couldn't get it to work yesterday on my notebook. There the two images had different mediaTypes. One had application/vnd.docker.distribution.manifest.v1+prettyjws the other application/vnd.docker.distribution.manifest.v1+json. Today after updating the CI pipelines both images have "mediaType: application/vnd.docker.distribution.manifest.v1+prettyjws" and it works. The confusing part here is why the buildah command without any --format tag resulted in different mediatypes and especially why one of them as +prettyjws and the other as +json although both build processes ran in the same container (also multi-arch). They run on different CentOS host systems. So maybe because of this. I also updated my buildah image in the process today (was around 3 months old) which might also have fixed this from buildah. Unfortunately I can't tell. In total I'm now doing this: - build the images with "--format docker" on different archs - push arch specific tags - create a manifest containing both images (with "correct" mediaTypes) using first manifest create and then manifest add - push the manifest with "--all" and "--format v2s2" This way I'm able to push and maintain the multiarch images on both docker.io and quay.io. > Laurent Meunier <laurent(a)deltalima.net&gt; hat am 17.11.2020 06:56 geschrieben: > > > On 16/11/2020 23:23, Nalin Dahyabhai wrote: > > On Wed, Nov 11, 2020 at 10:27:49AM +0100, Alexander Wellbrock via Podman wrote: > >> Hey there, since buildx is not a thing yet to build multi-arch images on one host I'd like to make use of the multi-arch feature of registries like docker.io and quay.io. > >> > >> I'm using the podman manifest command for this purpose but struggle to get it to work properly. I hope someone can point me in the right direction. > >> > >> I've get it to work (kind-of) by building two images on two different hosts architecture wise and then used podman manifest create to create a multi-arch manifest of both images. I then used podman manifest push to get it onto quay.io. This works and I'm able to use it, but unfortunately there is a catch right now. > >> > >> If I'm doing this multiple times, it complains that since a manifest for example-image:latest is already present. So it only work if the manifest does not already exist. I can't update it with newer images. Only if I remove the manifest :latest manually beforehand it obviously works again, but I'd like to avoid that. Apart from that it makes sense I suppose, because I'd rather like to update the manifest with new images then to override the old manifest, right? > >> > >> I then tried to use podman manifest add, remove etc. Podman then complains about manifest version not supported. E.g. I tried to update the actual images per architecture and then add the new ones to the manifest which I couldn't get to work. I'm highly confused about the podman manifest command man page. > > > > Someone asked a similar question on IRC a few days ago, and I didn't > > have a good answer then, so had to poke at it a bit. > > > > Anyway, the sequence of steps for creating a list using images that have > > previously been pushed to a registry looks something like this: > > > > on a box of each $arch: podman build -t docker://registry/repo:$arch-tag > > on one box: podman manifest create $listname > > repeat for each $arch on that one box: podman manifest add $listname docker://registry/repo:$arch-tag > > finally, on that one box: podman manifest push [--all] $listname docker://registry/repo:$single-tag > > > > Now, if you build a new image and want to replace it in the list, you > > should be able to start with the list you already have, or you can do > > this to start to build a new, updated version of the list: > > > > podman manifest create updated-list > > podman manifest add --all updated-list docker://registry/repo:$single-tag > > > > Note: "podman manifest add" (and "buildah manifest add") looks like it > > just looks at a locally-stored image if it finds one with the same name > > as the reference being passed on the command line. That locally-stored > > image isn't going to include all of the other entries in the copy of the > > list that the registry has, and that can cause some problems. I think > > we'll need to fix that. > > > > If it works as intended (or you started with the list you already had), > > you should have a copy of the list that's up at the registry, and > > "podman manifest inspect" should show you several entries in that list. > > > > The spec doesn't say anything about what to do when there are multiple > > entries in a list for the same platform, so we'll want to remove the > > entry that the list has for the current machine before we try to add our > > newly-built image to the list as its replacement. > > > > arch=$(go env GOARCH) > > instance=$(podman manifest inspect updated-list | jq -r \ > > '.manifests|map(select(.platform.architecture=="'$arch'"))[].digest') > > podman manifest remove updated-list $instance > > > > Now you can go ahead and add your new image to the list and push the > > updated list to the registry. > > > > If the new image has already been pushed to the registry: > > > > podman manifest add updated-list docker://registry/repo:$arch-tag > > podman manifest push updated-list docker://registry/repo:$single-tag > > > > If not: > > > > podman manifest add updated-list local-$arch-image > > podman manifest push --all updated-list docker://registry/repo:$single-tag > > > > The main difference between these two cases is that the --all flag tells > > podman to push the images that the list references, in addition to the > > list itself. That can take a while, since for every image, even though > > the pushing logic checks if a part of an image has already been uploaded > > to the registry before trying to upload it, it's still a lot of parts, > > and that can require a lot of round trips to the registry. But if the > > image wasn't pushed first, the registry will probably reject the new > > list because it references things that the registry doesn't have, so > > it's required. > > > > I've got a couple of ideas of things we can do to help simplify this > > process, so that it requires less typing at the CLI, but none of that's > > implemented yet. > > > > If you run into trouble, please follow up with the specific errors > > you're getting, and we'll see where they point us. > > > > Cheers, > > > > Nalin > > Hello, > > I'm also building multi-arch images with buildah and podman, and I just > would like to add a few remarks. > > I use "buildah manifest" instead of "podman manifest". Is there some > behavior differences between these two commands? > > If you build your image on an ARM architecture, you may need to add the > --variant option to the "podman manifest add" command: > > # example for ARMv7 (like the Raspberry Pi 3) > buildah manifest add --variant 7 image_name:$single-tag > docker.io/$USERNAME/image_name:$arch-tag > > When pushing the manifest to Docker Hub, I had to add the "--format > v2s2" to my "buildah manifest push" command. It seems that Docker Hub > does not support the OCI format and I need to use the v2s2 format (not > 100% sure of this): > > buildah manifest push --all --format v2s2 image_name:$single-tag > docker://docker.io/$USERNAME/image_name:$single-tag > > I globally use the same steps as Alexander to push multi-arch images to > a registry: > - build the image on each architecture (amd64 and armv7 in my case) > - push each image on docker hub with an architecture specific tag > - create and push a manifest as described above > > Here is a link to the detailed steps I use to push multi-arch images to > Docker Hub: > https://github.com/lmeunier/oci-image-ejabberd#login-to-the-docker-hub-re... > > Best regards, > Laurent > _______________________________________________ > Podman mailing list -- podman(a)lists.podman.io > To unsubscribe send an email to podman-leave(a)lists.podman.io