rootless podman, docker-credential-gcloud, and snaps
Hi, I'm on Ubuntu, and I've recently encountered an issue when trying to use rootless podman with the docker-credential-gcloud helper installed via snap. This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October. Here's what I see now: ``` $ podman pull gcr.io/private/image Trying to pull gcr.io/private/image... 2021/02/01 13:19:17.474248 cmd_run.go:994: WARNING: cannot create user data directory: cannot create "/root/snap/google-cloud-sdk/166": mkdir /root/snap: permission denied cannot create user data directory: /root/snap/google-cloud-sdk/166: Permission denied error getting credentials - err: exit status 1, out: `` Error: unable to pull gcr.io/private/image: Error initializing source docker://gcr.io/private/image:latest: error getting username and password: error getting credentials - err: exit status 1, out: `` ``` So it looks like the credential helper is being executed as root now. I'm not sure in which component the problem lies, or where I should file an issue. Any pointers would be appreciated. Thanks Ioan Rogers Sent with ProtonMail Secure Email.
Hi Ioan, Thanks for reaching out. On Mon, Feb 1, 2021 at 10:47 PM Ioan Rogers via Podman < podman@lists.podman.io> wrote:
Hi,
I'm on Ubuntu, and I've recently encountered an issue when trying to use
rootless podman with the docker-credential-gcloud helper installed via snap.
This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October.
Do you recall if it broke with an update to Podman?
Here's what I see now: ``` $ podman pull gcr.io/private/image Trying to pull gcr.io/private/image... 2021/02/01 13:19:17.474248 cmd_run.go:994: WARNING: cannot create user data directory: cannot create "/root/snap/google-cloud-sdk/166": mkdir /root/snap: permission denied cannot create user data directory: /root/snap/google-cloud-sdk/166: Permission denied error getting credentials - err: exit status 1, out: `` Error: unable to pull gcr.io/private/image: Error initializing source docker://gcr.io/private/image:latest: error getting username and password: error getting credentials - err: exit status 1, out: `` ```
So it looks like the credential helper is being executed as root now. I'm not sure in which component the problem lies, or where I should file an issue. Any pointers would be appreciated.
I suspect that's due to the user namespace rootless Podman runs in. Kind regards, Valentin
Thanks
Ioan Rogers
Sent with ProtonMail Secure Email._______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Hi,
I'm on Ubuntu, and I've recently encountered an issue when trying to use rootless podman with the docker-credential-gcloud helper installed via snap. This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October.
Do you recall if it broke with an update to Podman?
I forgot to mention that this also happens with buildah, but I guess they share the same code for pulling and pushing? Well, yikes, my shell history shows that this first happened in July, but I decided to just do my builds somewhere else and deal with it later. "Later" didn't come until November, when I next tried to push something, and ended up uninstalling the gcloud snap. So this was when I first encountered it, according to my history: 2020-07-07 strace -ff -o tr buildah push --authfile auth.json gcr.io/private/image:xxx (I do still have those trace files, but I'm not sure if they would have anything private in them, so I won't post them here) The last update before that was on 2020-06-24, buildah from 1.14.9~1 to 1.15.0~1 and podman from 1.9.3~1 to 2.0.0~1. Unfortunately, my shell history doesn't go back before June, but there couple of images which would've only be pushed by me in the registry, which are dated 2020-05-20. At that point, apt history shows buildah 1.14.9~1, so 1.15.0 does seem the likely suspect.
So it looks like the credential helper is being executed as root now. I'm not sure in which component the problem lies, or where I should file an issue. Any pointers would be appreciated.
I suspect that's due to the user namespace rootless Podman runs in.
Is it possible that podman/buildah used to call the credhelper before setting up the user namespace? Thanks Ioan Rogers
On Wed, Feb 3, 2021 at 9:54 PM Ioan Rogers via Podman < podman@lists.podman.io> wrote:
Hi,
I'm on Ubuntu, and I've recently encountered an issue when trying to use rootless podman with the docker-credential-gcloud helper installed via snap. This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October.
Do you recall if it broke with an update to Podman?
I forgot to mention that this also happens with buildah, but I guess they share the same code for pulling and pushing?
Well, yikes, my shell history shows that this first happened in July, but I decided to just do my builds somewhere else and deal with it later. "Later" didn't come until November, when I next tried to push something, and ended up uninstalling the gcloud snap.
So this was when I first encountered it, according to my history: 2020-07-07 strace -ff -o tr buildah push --authfile auth.json gcr.io/private/image: <http://gcr.io/production-176813/mm-perl:5.30.3>xxx (I do still have those trace files, but I'm not sure if they would have anything private in them, so I won't post them here)
The last update before that was on 2020-06-24, buildah from 1.14.9~1 to 1.15.0~1 and podman from 1.9.3~1 to 2.0.0~1.
Unfortunately, my shell history doesn't go back before June, but there couple of images which would've only be pushed by me in the registry, which are dated 2020-05-20. At that point, apt history shows buildah 1.14.9~1, so 1.15.0 does seem the likely suspect.
So it looks like the credential helper is being executed as root now. I'm not sure in which component the problem lies, or where I should file an issue. Any pointers would be appreciated.
I suspect that's due to the user namespace rootless Podman runs in.
Is it possible that podman/buildah used to call the credhelper before setting up the user namespace?
While I'd love this to be possible, it would require major changes in our architecture. All our tools create the user namespace and rexec into it during start up. However, I opened a pull request against the GCR credential helper. If the helper would give precedence to $HOME, it should work well with Podman. Let's see if the maintainers like the change. The only other alternative I can see at the moment is to run as root. Kind regards, Valentin [1] https://github.com/GoogleCloudPlatform/docker-credential-gcr/pull/85
On Fri, Feb 5, 2021 at 1:53 PM Valentin Rothberg <rothberg@redhat.com> wrote:
On Wed, Feb 3, 2021 at 9:54 PM Ioan Rogers via Podman < podman@lists.podman.io> wrote:
Hi,
I'm on Ubuntu, and I've recently encountered an issue when trying to use rootless podman with the docker-credential-gcloud helper installed via snap. This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October.
Do you recall if it broke with an update to Podman?
I forgot to mention that this also happens with buildah, but I guess they share the same code for pulling and pushing?
Well, yikes, my shell history shows that this first happened in July, but I decided to just do my builds somewhere else and deal with it later. "Later" didn't come until November, when I next tried to push something, and ended up uninstalling the gcloud snap.
So this was when I first encountered it, according to my history: 2020-07-07 strace -ff -o tr buildah push --authfile auth.json gcr.io/private/image: <http://gcr.io/production-176813/mm-perl:5.30.3>xxx (I do still have those trace files, but I'm not sure if they would have anything private in them, so I won't post them here)
The last update before that was on 2020-06-24, buildah from 1.14.9~1 to 1.15.0~1 and podman from 1.9.3~1 to 2.0.0~1.
Unfortunately, my shell history doesn't go back before June, but there couple of images which would've only be pushed by me in the registry, which are dated 2020-05-20. At that point, apt history shows buildah 1.14.9~1, so 1.15.0 does seem the likely suspect.
So it looks like the credential helper is being executed as root now. I'm not sure in which component the problem lies, or where I should file an issue. Any pointers would be appreciated.
I suspect that's due to the user namespace rootless Podman runs in.
Is it possible that podman/buildah used to call the credhelper before setting up the user namespace?
While I'd love this to be possible, it would require major changes in our architecture. All our tools create the user namespace and rexec into it during start up.
However, I opened a pull request against the GCR credential helper. If the helper would give precedence to $HOME, it should work well with Podman. Let's see if the maintainers like the change.
The only other alternative I can see at the moment is to run as root.
Kind regards, Valentin
[1] https://github.com/GoogleCloudPlatform/docker-credential-gcr/pull/85
The PR for the docker-credential-gcr helper just merged. Kind regards, Valentin
On 2/1/21 4:37 PM, Ioan Rogers via Podman wrote:
Hi,
I'm on Ubuntu, and I've recently encountered an issue when trying to use rootless podman with the docker-credential-gcloud helper installed via snap. This works fine when using the official google-cloud-sdk apt packages, and it used to work with snap packages until last October.
On the off-chance that tweaking your setup is possible/easier fix: I found it trivial to use rootless-podman in an interactive script with the SDK image `docker.io/google/cloud-sdk:alpine` and simply point it at the executing user's configuration. Something like: GCLOUD="podman run -it --rm --security-opt label=disable -v $HOME/.config/gcloud:/root/.config/gcloud -v $HOME/.config/gcloud/ssh:/root/.ssh $GCLOUD_IMAGE gcloud --configuration=$CFGNAME --project=$PROJECT" Then the user (or a script) can simply do things like: $GCLOUD init --project=$PROJECT --console-only --skip-diagnostics $GCLOUD compute instance create --zone=$ZONE ... --- For non-interactive use, we use built container images containing the installed SDK along with an entrypoint script. From the host-side, we call these containers passing in the GCP service-account JSON within an env. var. For automated use, I deliberately avoid writing and volume-mounting in the credentials. This improves security because it avoids the on-disk credentials being accessable from two places (host and container). The function we call to setup GCP inside the container, is here for reference: https://github.com/containers/automation_images/blob/master/imgts/lib_entryp... -- Chris Evich, RHCA III Senior Quality Assurance Engineer If it ain't broke, yain't tryin' hard nough.
participants (3)
-
Chris Evich -
Ioan Rogers -
Valentin Rothberg