4:08 a.m.
On Fri, Nov 20, 2020 at 12:41 AM Alexander Wellbrock via Podman <
podman(a)lists.podman.io> wrote:
Hey Daniel,
I don't know why podman doesn't seem to find the credentials, let me put a
guess on it in a minute. I've mitigated this however in my setup by putting
an Auth file somewhere in the system and referencing it by --authfile as
part of the podman run command.
My first guess here is you logged in to the registry with a different user
than root and using systemd without specifying the user who's actually
logged in
I echo that thought. Are you running the `podman login` as the same user?
If the systemd service runs on the system level it's running as root. In
that case, we need to `sudo podman login`.
On November 19, 2020 11:39:40 PM GMT+01:00, Tom Sweeney <
tsweeney(a)redhat.com> wrote:
>
> Daniel,
>
> Sorry about not getting back to you sooner. IDK right off the top of my
> head, but I've spun this off to the Podman mailing list, I'm sure folks
> monitoring that will have a thought or three.
>
> t
>
>
> -------- Forwarded Message --------
> Subject: podman question
> Date: Wed, 18 Nov 2020 16:26:32 -0500
> From: Daniel Pivonka <dpivonka(a)redhat.com> <dpivonka(a)redhat.com>
> To: Tom Sweeney <tsweeney(a)redhat.com> <tsweeney(a)redhat.com>
>
> Hi Tom,
>
> One of my coworkers pointed me to you about a podman issue I'm having.
> I'm hoping you can help me or point me in the right direction.
>
> I work on the ceph orchestration team and I'm facing an issue when trying
> to deploy containers from an authenticated registry where podman can't seem
> to access the registry login info.
>
> I'm trying to run containers from systemd in a way similar to this
>
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_at...
>
>
> The image im trying to use comes from registry.redhat.io
>
> so as a test i ran podman login first
>
> then starting my service with this unit file
>
> [Unit]
> Description=Redis container
>
> [Service]
> Restart=always
> ExecStart=/bin/podman run --rm --ipc=host --net=host --name
> ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb-node-exporter.vm-00 --user 65534
> -d --conmon-pidfile
> /run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-pid
> --cidfile
> /run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-cid
> -e CONTAINER_IMAGE=
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 -e
> NODE_NAME=vm-00 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /:/rootfs:ro
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
> --no-collector.timex
> ExecStop=/usr/bin/podman stop -t 2 redis_server
>
> [Install]
> WantedBy=local.target
>
>
> this is similar to the unit.run file that ceph would use for its services.
>
> the service fails though and the journalctl log show that podman was not
> able to pull the image because of a failed authentication
>
>
> [root@vm-00 system]# journalctl -u test.service
> -- Logs begin at Wed 2020-11-18 21:04:45 UTC, end at Wed 2020-11-18
> 21:14:22 UTC. --
> Nov 18 21:14:20 vm-00 systemd[1]: Started Redis container.
> Nov 18 21:14:21 vm-00 podman[9652]: 2020-11-18 21:14:21.066551744 +0000
> UTC m=+0.234565900 system refresh
> Nov 18 21:14:21 vm-00 podman[9652]: Trying to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5...
> Nov 18 21:14:21 vm-00 podman[9652]: unable to retrieve auth token:
> invalid username/password: unauthorized: Please login to the Red Hat
> Registry using your Customer Portal credentials. Further instructions ca>
> Nov 18 21:14:21 vm-00 podman[9652]: Error: unable to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to pull image: Error initializing source docker://
> registry.redhat.io/openshift4/>
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Main process exited,
> code=exited, status=125/n/a
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Failed with result
> 'exit-code'.
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Service RestartSec=100ms
> expired, scheduling restart.
>
>
>
>
>
> i did a little more debugging and it seems that systemd does not know
> where the auth file is
>
>
> Nov 18 21:19:09 vm-00 systemd[1]: Started Redis container.
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Reading configuration file
> \"/usr/share/containers/libpod.conf\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Merged system config
> \"/usr/share/containers/libpod.conf\": &{{false false false false
false
> true} 0 { [] [] []} docker:// runc map[crun:[/usr/bin/crun
> /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun
> /bin/crun /run/current-system/sw/bin/crun] kata-fc:[/usr/bin/kata-fc] kata->
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using conmon: \"/usr/bin/conmon\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Initializing boltdb state at
> /var/lib/containers/storage/libpod/bolt_state.db"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using graph driver overlay"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using graph root /var/lib/containers/storage"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using run root /var/run/containers/storage"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using static dir /var/lib/containers/storage/libpod"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using tmp dir /var/run/libpod"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using volume path /var/lib/containers/storage/volumes"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Set libpod namespace to \"\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="[graphdriver] trying provided driver
\"overlay\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that overlay is supported"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that metacopy is being used"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that native-diff is not being
used"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Not using native diff for overlay, this may cause
> degraded performance for building images: kernel has
> CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="backingFs=extfs, projectQuotaSupported=false,
> useNativeDiff=false, usingMetacopy=true"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Initializing event backend journald"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-qemu: no
> valid executable found for OCI runtime kata-qemu: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-fc: no
> valid executable found for OCI runtime kata-fc: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="using runtime \"/usr/bin/runc\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime crun: no valid
> executable found for OCI runtime crun: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-runtime:
> no valid executable found for OCI runtime kata-runtime: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=info msg="Found CNI network podman (type=bridge) at
> /etc/cni/net.d/87-podman-bridge.conflist"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Default CNI network name podman is unchangeable"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="parsed reference into \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="reference \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
>
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> does not resolve to an image ID"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="parsed reference into \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: Trying to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5...
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="reference rewritten from '
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5' to '
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5'"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Trying to access \"
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Credentials not found"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using registries.d directory /etc/containers/registries.d
> for sigstore configuration"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg=" Using \"default-docker\" configuration"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg=" No signature storage configuration found for
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Looking for TLS certificates and private keys in
> /etc/docker/certs.d/registry.redhat.io"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="GET https://registry.redhat.io/v2/"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="GET
>
https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/aut...
> "
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Server response when trying to obtain an access token:
> \n\"unauthorized: Please login to the Red Hat Registry using your Customer
> Portal credentials. Further instructions can be found here:
> https://access.redhat.com/RegistryAuthentication\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Accessing \"
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
>
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> failed: unable to retrieve auth token: invalid username/password:
> unauthorized: Please login to the Red Hat Registry using your Customer
> Portal credentials. Further instructions can be found here:
> https://access.redhat.c>
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Error pulling image ref //
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: Error
> initializing source docker://
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to retrieve auth token: invalid username/password: unauthorized: Please
> login to the Red Hat Registr>
> Nov 18 21:19:09 vm-00 podman[10481]: unable to retrieve auth token:
> invalid username/password: unauthorized: Please login to the Red Hat
> Registry using your Customer Portal credentials. Further instructions can
> be found here: https://access.redhat.com/RegistryAuthentication
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=error msg="unable to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to pull image: Error initializing source docker://
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to retrieve auth token: invalid username/password: unauthorized: Please
> login to the Red >
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Main process exited,
> code=exited, status=125/n/a
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Failed with result
> 'exit-code'.
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Service RestartSec=100ms
> expired, scheduling restart.
>
>
>
> running 'podman login --get-login registry.redhat.io' always shows im
> logged in though.
>
>
> Are you aware of any reason why it seems like when running a
> container from systemd it cant access the auth file to pull the container
> first?
>
> If you need anymore info or want to see it happen live im more than happy
> to set up a meeting or something just let me know.
>
>
> Thank you!
> -Daniel Pivonka
>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io