On Wed, Nov 11, 2020 at 10:27:49AM +0100, Alexander Wellbrock via Podman wrote: > Hey there, since buildx is not a thing yet to build multi-arch images on one host I'd like to make use of the multi-arch feature of registries like docker.io and quay.io. > > I'm using the podman manifest command for this purpose but struggle to get it to work properly. I hope someone can point me in the right direction. > > I've get it to work (kind-of) by building two images on two different hosts architecture wise and then used podman manifest create to create a multi-arch manifest of both images. I then used podman manifest push to get it onto quay.io. This works and I'm able to use it, but unfortunately there is a catch right now. > > If I'm doing this multiple times, it complains that since a manifest for example-image:latest is already present. So it only work if the manifest does not already exist. I can't update it with newer images. Only if I remove the manifest :latest manually beforehand it obviously works again, but I'd like to avoid that. Apart from that it makes sense I suppose, because I'd rather like to update the manifest with new images then to override the old manifest, right? > > I then tried to use podman manifest add, remove etc. Podman then complains about manifest version not supported. E.g. I tried to update the actual images per architecture and then add the new ones to the manifest which I couldn't get to work. I'm highly confused about the podman manifest command man page. Someone asked a similar question on IRC a few days ago, and I didn't have a good answer then, so had to poke at it a bit. Anyway, the sequence of steps for creating a list using images that have previously been pushed to a registry looks something like this: on a box of each $arch: podman build -t docker://registry/repo:$arch-tag on one box: podman manifest create $listname repeat for each $arch on that one box: podman manifest add $listname docker://registry/repo:$arch-tag finally, on that one box: podman manifest push [--all] $listname docker://registry/repo:$single-tag Now, if you build a new image and want to replace it in the list, you should be able to start with the list you already have, or you can do this to start to build a new, updated version of the list: podman manifest create updated-list podman manifest add --all updated-list docker://registry/repo:$single-tag Note: "podman manifest add" (and "buildah manifest add") looks like it just looks at a locally-stored image if it finds one with the same name as the reference being passed on the command line. That locally-stored image isn't going to include all of the other entries in the copy of the list that the registry has, and that can cause some problems. I think we'll need to fix that. If it works as intended (or you started with the list you already had), you should have a copy of the list that's up at the registry, and "podman manifest inspect" should show you several entries in that list. The spec doesn't say anything about what to do when there are multiple entries in a list for the same platform, so we'll want to remove the entry that the list has for the current machine before we try to add our newly-built image to the list as its replacement. arch=$(go env GOARCH) instance=$(podman manifest inspect updated-list | jq -r \ '.manifests|map(select(.platform.architecture=="'$arch'"))[].digest') podman manifest remove updated-list $instance Now you can go ahead and add your new image to the list and push the updated list to the registry. If the new image has already been pushed to the registry: podman manifest add updated-list docker://registry/repo:$arch-tag podman manifest push updated-list docker://registry/repo:$single-tag If not: podman manifest add updated-list local-$arch-image podman manifest push --all updated-list docker://registry/repo:$single-tag The main difference between these two cases is that the --all flag tells podman to push the images that the list references, in addition to the list itself. That can take a while, since for every image, even though the pushing logic checks if a part of an image has already been uploaded to the registry before trying to upload it, it's still a lot of parts, and that can require a lot of round trips to the registry. But if the image wasn't pushed first, the registry will probably reject the new list because it references things that the registry doesn't have, so it's required. I've got a couple of ideas of things we can do to help simplify this process, so that it requires less typing at the CLI, but none of that's implemented yet. If you run into trouble, please follow up with the specific errors you're getting, and we'll see where they point us. Cheers, Nalin

Alexander Wellbrock <a.wellbrock(a)mailbox.org&gt; hat am 17.11.2020 12:09 geschrieben: Finally, I eventually got it to work. Still there are some points I want to raise / remark. So first things first thanks a lot to Nalin and Laurent. Your inputs were just the missing bits I needed. Thanks Nalin for that really good summary validating my approach in general. I especially missed the API change at first of the podman manifest create/add command. Laurent, I don't require --variant 7 to push my images / manifests to quay.io or docker.io. However I added the "--format v2s2" flag to the push command as well as "--all" as suggested by Nalin. I also opened a bug report at quay.io and got the response that my manifest was unsupported by quay.io. One of the two images had an implicit "--format oci" which is apparently not officially supported by quay.io. Only "--format docker" is. Changing this alone however did not work. I don't know why. I couldn't get it to work yesterday on my notebook. There the two images had different mediaTypes. One had application/vnd.docker.distribution.manifest.v1+prettyjws the other application/vnd.docker.distribution.manifest.v1+json. Today after updating the CI pipelines both images have "mediaType: application/vnd.docker.distribution.manifest.v1+prettyjws" and it works. The confusing part here is why the buildah command without any --format tag resulted in different mediatypes and especially why one of them as +prettyjws and the other as +json although both build processes ran in the same container (also multi-arch). They run on different CentOS host systems. So maybe because of this. I also updated my buildah image in the process today (was around 3 months old) which might also have fixed this from buildah. Unfortunately I can't tell. In total I'm now doing this: - build the images with "--format docker" on different archs - push arch specific tags - create a manifest containing both images (with "correct" mediaTypes) using first manifest create and then manifest add - push the manifest with "--all" and "--format v2s2" This way I'm able to push and maintain the multiarch images on both docker.io and quay.io. > Laurent Meunier <laurent(a)deltalima.net&gt; hat am 17.11.2020 06:56 geschrieben: > > > On 16/11/2020 23:23, Nalin Dahyabhai wrote: > > On Wed, Nov 11, 2020 at 10:27:49AM +0100, Alexander Wellbrock via Podman wrote: > >> Hey there, since buildx is not a thing yet to build multi-arch images on one host I'd like to make use of the multi-arch feature of registries like docker.io and quay.io. > >> > >> I'm using the podman manifest command for this purpose but struggle to get it to work properly. I hope someone can point me in the right direction. > >> > >> I've get it to work (kind-of) by building two images on two different hosts architecture wise and then used podman manifest create to create a multi-arch manifest of both images. I then used podman manifest push to get it onto quay.io. This works and I'm able to use it, but unfortunately there is a catch right now. > >> > >> If I'm doing this multiple times, it complains that since a manifest for example-image:latest is already present. So it only work if the manifest does not already exist. I can't update it with newer images. Only if I remove the manifest :latest manually beforehand it obviously works again, but I'd like to avoid that. Apart from that it makes sense I suppose, because I'd rather like to update the manifest with new images then to override the old manifest, right? > >> > >> I then tried to use podman manifest add, remove etc. Podman then complains about manifest version not supported. E.g. I tried to update the actual images per architecture and then add the new ones to the manifest which I couldn't get to work. I'm highly confused about the podman manifest command man page. > > > > Someone asked a similar question on IRC a few days ago, and I didn't > > have a good answer then, so had to poke at it a bit. > > > > Anyway, the sequence of steps for creating a list using images that have > > previously been pushed to a registry looks something like this: > > > > on a box of each $arch: podman build -t docker://registry/repo:$arch-tag > > on one box: podman manifest create $listname > > repeat for each $arch on that one box: podman manifest add $listname docker://registry/repo:$arch-tag > > finally, on that one box: podman manifest push [--all] $listname docker://registry/repo:$single-tag > > > > Now, if you build a new image and want to replace it in the list, you > > should be able to start with the list you already have, or you can do > > this to start to build a new, updated version of the list: > > > > podman manifest create updated-list > > podman manifest add --all updated-list docker://registry/repo:$single-tag > > > > Note: "podman manifest add" (and "buildah manifest add") looks like it > > just looks at a locally-stored image if it finds one with the same name > > as the reference being passed on the command line. That locally-stored > > image isn't going to include all of the other entries in the copy of the > > list that the registry has, and that can cause some problems. I think > > we'll need to fix that. > > > > If it works as intended (or you started with the list you already had), > > you should have a copy of the list that's up at the registry, and > > "podman manifest inspect" should show you several entries in that list. > > > > The spec doesn't say anything about what to do when there are multiple > > entries in a list for the same platform, so we'll want to remove the > > entry that the list has for the current machine before we try to add our > > newly-built image to the list as its replacement. > > > > arch=$(go env GOARCH) > > instance=$(podman manifest inspect updated-list | jq -r \ > > '.manifests|map(select(.platform.architecture=="'$arch'"))[].digest') > > podman manifest remove updated-list $instance > > > > Now you can go ahead and add your new image to the list and push the > > updated list to the registry. > > > > If the new image has already been pushed to the registry: > > > > podman manifest add updated-list docker://registry/repo:$arch-tag > > podman manifest push updated-list docker://registry/repo:$single-tag > > > > If not: > > > > podman manifest add updated-list local-$arch-image > > podman manifest push --all updated-list docker://registry/repo:$single-tag > > > > The main difference between these two cases is that the --all flag tells > > podman to push the images that the list references, in addition to the > > list itself. That can take a while, since for every image, even though > > the pushing logic checks if a part of an image has already been uploaded > > to the registry before trying to upload it, it's still a lot of parts, > > and that can require a lot of round trips to the registry. But if the > > image wasn't pushed first, the registry will probably reject the new > > list because it references things that the registry doesn't have, so > > it's required. > > > > I've got a couple of ideas of things we can do to help simplify this > > process, so that it requires less typing at the CLI, but none of that's > > implemented yet. > > > > If you run into trouble, please follow up with the specific errors > > you're getting, and we'll see where they point us. > > > > Cheers, > > > > Nalin > > Hello, > > I'm also building multi-arch images with buildah and podman, and I just > would like to add a few remarks. > > I use "buildah manifest" instead of "podman manifest". Is there some > behavior differences between these two commands? > > If you build your image on an ARM architecture, you may need to add the > --variant option to the "podman manifest add" command: > > # example for ARMv7 (like the Raspberry Pi 3) > buildah manifest add --variant 7 image_name:$single-tag > docker.io/$USERNAME/image_name:$arch-tag > > When pushing the manifest to Docker Hub, I had to add the "--format > v2s2" to my "buildah manifest push" command. It seems that Docker Hub > does not support the OCI format and I need to use the v2s2 format (not > 100% sure of this): > > buildah manifest push --all --format v2s2 image_name:$single-tag > docker://docker.io/$USERNAME/image_name:$single-tag > > I globally use the same steps as Alexander to push multi-arch images to > a registry: > - build the image on each architecture (amd64 and armv7 in my case) > - push each image on docker hub with an architecture specific tag > - create and push a manifest as described above > > Here is a link to the detailed steps I use to push multi-arch images to > Docker Hub: > https://github.com/lmeunier/oci-image-ejabberd#login-to-the-docker-hub-re... > > Best regards, > Laurent > _______________________________________________ > Podman mailing list -- podman(a)lists.podman.io > To unsubscribe send an email to podman-leave(a)lists.podman.io