5:39 p.m.
Daniel,
Sorry about not getting back to you sooner. IDK right off the top of my
head, but I've spun this off to the Podman mailing list, I'm sure folks
monitoring that will have a thought or three.
t
-------- Forwarded Message --------
Subject: podman question
Date: Wed, 18 Nov 2020 16:26:32 -0500
From: Daniel Pivonka <dpivonka(a)redhat.com>
To: Tom Sweeney <tsweeney(a)redhat.com>
Hi Tom,
One of my coworkers pointed me to you about a podman issue I'm having.
I'm hoping you can help me or point me in the right direction.
I work on the ceph orchestration team and I'm facing an issue when
trying to deploy containers from an authenticated registry where podman
can't seem to access the registry login info.
I'm trying to run containers from systemd in a way similar to this
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_at...
The image im trying to use comes from registry.redhat.io
<http://registry.redhat.io>
so as a test i ran podman login first
then starting my service with this unit file
[Unit]
Description=Redis container
[Service]
Restart=always
ExecStart=/bin/podman run --rm --ipc=host --net=host --name
ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb-node-exporter.vm-00 --user
65534 -d --conmon-pidfile
/run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-pid
--cidfile
/run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-cid
-e
CONTAINER_IMAGE=registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>
-e NODE_NAME=vm-00 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v
/:/rootfs:ro
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>
--no-collector.timex
ExecStop=/usr/bin/podman stop -t 2 redis_server
[Install]
WantedBy=local.target
this is similar to the unit.run file that ceph would use for its services.
the service fails though and the journalctl log show that podman was not
able to pull the image because of a failed authentication
[root@vm-00 system]# journalctl -u test.service
-- Logs begin at Wed 2020-11-18 21:04:45 UTC, end at Wed 2020-11-18
21:14:22 UTC. --
Nov 18 21:14:20 vm-00 systemd[1]: Started Redis container.
Nov 18 21:14:21 vm-00 podman[9652]: 2020-11-18 21:14:21.066551744
+0000 UTC m=+0.234565900 system refresh
Nov 18 21:14:21 vm-00 podman[9652]: Trying to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.>..
Nov 18 21:14:21 vm-00 podman[9652]: unable to retrieve auth token:
invalid username/password: unauthorized: Please login to the Red Hat
Registry using your Customer Portal credentials. Further
instructions ca>
Nov 18 21:14:21 vm-00 podman[9652]: Error: unable to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to pull image: Error initializing source
docker://registry.redhat.io/openshift4/
<http://registry.redhat.io/openshift4/>>
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Main process exited,
code=exited, status=125/n/a
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Failed with result
'exit-code'.
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Service
RestartSec=100ms expired, scheduling restart.
i did a little more debugging and it seems that systemd does not know
where the auth file is
Nov 18 21:19:09 vm-00 systemd[1]: Started Redis container.
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Reading configuration file
\"/usr/share/containers/libpod.conf\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Merged system config
\"/usr/share/containers/libpod.conf\": &{{false false false false
false true} 0 { [] [] []} docker:// runc map[crun:[/usr/bin/crun
/usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun
/bin/crun /run/current-system/sw/bin/crun]
kata-fc:[/usr/bin/kata-fc] kata->
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using conmon: \"/usr/bin/conmon\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Initializing boltdb state at
/var/lib/containers/storage/libpod/bolt_state.db"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using graph driver overlay"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using graph root /var/lib/containers/storage"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using run root /var/run/containers/storage"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using static dir /var/lib/containers/storage/libpod"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using tmp dir /var/run/libpod"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using volume path /var/lib/containers/storage/volumes"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Set libpod namespace to \"\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="[graphdriver] trying provided driver
\"overlay\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that overlay is supported"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that metacopy is being used"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that native-diff is not
being used"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Not using native diff for overlay, this may cause
degraded performance for building images: kernel has
CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="backingFs=extfs, projectQuotaSupported=false,
useNativeDiff=false, usingMetacopy=true"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Initializing event backend journald"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-qemu: no valid executable found for OCI runtime kata-qemu:
invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-fc: no valid executable found for OCI runtime kata-fc: invalid
argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="using runtime \"/usr/bin/runc\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime crun:
no valid executable found for OCI runtime crun: invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-runtime: no valid executable found for OCI runtime
kata-runtime: invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=info msg="Found CNI network podman (type=bridge) at
/etc/cni/net.d/87-podman-bridge.conflist"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Default CNI network name podman is unchangeable"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="parsed reference into
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="reference
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
does not resolve to an image ID"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="parsed reference into
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: Trying to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.>..
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="reference rewritten from
'registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
to 'registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Trying to access
\"registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Credentials not found"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using registries.d directory
/etc/containers/registries.d for sigstore configuration"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg=" Using \"default-docker\" configuration"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg=" No signature storage configuration found for
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Looking for TLS certificates and private keys in
/etc/docker/certs.d/registry.redhat.io <http://registry.redhat.io>"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="GET https://registry.redhat.io/v2/"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="GET
https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/aut...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Server response when trying to obtain an access
token: \n\"unauthorized: Please login to the Red Hat Registry using
your Customer Portal credentials. Further instructions can be found
here: https://access.redhat.com/RegistryAuthentication\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Accessing
\"registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
failed: unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red Hat Registry using your
Customer Portal credentials. Further instructions can be found here:
https://access.redhat.c>
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Error pulling image ref
//registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
Error initializing source
docker://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red Hat Registr>
Nov 18 21:19:09 vm-00 podman[10481]: unable to retrieve auth
token: invalid username/password: unauthorized: Please login to the
Red Hat Registry using your Customer Portal credentials. Further
instructions can be found here:
https://access.redhat.com/RegistryAuthentication
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=error msg="unable to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to pull image: Error initializing source
docker://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red >
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Main process exited,
code=exited, status=125/n/a
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Failed with result
'exit-code'.
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Service
RestartSec=100ms expired, scheduling restart.
running 'podman login --get-login registry.redhat.io
<http://registry.redhat.io>' always shows im logged in though.
Are you aware of any reason why it seems like when running a
container from systemd it cant access the auth file to pull the
container first?
If you need anymore info or want to see it happen live im more than
happy to set up a meeting or something just let me know.
Thank you!
-Daniel Pivonka
6:38 p.m.
Hey Daniel,
I don't know why podman doesn't seem to find the credentials, let me put a guess
on it in a minute. I've mitigated this however in my setup by putting an Auth file
somewhere in the system and referencing it by --authfile as part of the podman run
command.
My first guess here is you logged in to the registry with a different user than root and
using systemd without specifying the user who's actually logged in
Best regards
On November 19, 2020 11:39:40 PM GMT+01:00, Tom Sweeney <tsweeney(a)redhat.com>
wrote:
Daniel,
Sorry about not getting back to you sooner. IDK right off the top of my
head, but I've spun this off to the Podman mailing list, I'm sure folks
monitoring that will have a thought or three.
t
-------- Forwarded Message --------
Subject: podman question
Date: Wed, 18 Nov 2020 16:26:32 -0500
From: Daniel Pivonka <dpivonka(a)redhat.com>
To: Tom Sweeney <tsweeney(a)redhat.com>
Hi Tom,
One of my coworkers pointed me to you about a podman issue I'm having.
I'm hoping you can help me or point me in the right direction.
I work on the ceph orchestration team and I'm facing an issue when
trying to deploy containers from an authenticated registry where podman
can't seem to access the registry login info.
I'm trying to run containers from systemd in a way similar to this
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_at...
The image im trying to use comes from registry.redhat.io
<http://registry.redhat.io>
so as a test i ran podman login first
then starting my service with this unit file
[Unit]
Description=Redis container
[Service]
Restart=always
ExecStart=/bin/podman run --rm --ipc=host --net=host --name
ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb-node-exporter.vm-00 --user
65534 -d --conmon-pidfile
/run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-pid
--cidfile
/run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-cid
-e
CONTAINER_IMAGE=registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>
-e NODE_NAME=vm-00 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v
/:/rootfs:ro
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>
--no-collector.timex
ExecStop=/usr/bin/podman stop -t 2 redis_server
[Install]
WantedBy=local.target
this is similar to the unit.run file that ceph would use for its services.
the service fails though and the journalctl log show that podman was not
able to pull the image because of a failed authentication
[root@vm-00 system]# journalctl -u test.service
-- Logs begin at Wed 2020-11-18 21:04:45 UTC, end at Wed 2020-11-18
21:14:22 UTC. --
Nov 18 21:14:20 vm-00 systemd[1]: Started Redis container.
Nov 18 21:14:21 vm-00 podman[9652]: 2020-11-18 21:14:21.066551744
+0000 UTC m=+0.234565900 system refresh
Nov 18 21:14:21 vm-00 podman[9652]: Trying to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.>..
Nov 18 21:14:21 vm-00 podman[9652]: unable to retrieve auth token:
invalid username/password: unauthorized: Please login to the Red Hat
Registry using your Customer Portal credentials. Further
instructions ca>
Nov 18 21:14:21 vm-00 podman[9652]: Error: unable to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to pull image: Error initializing source
docker://registry.redhat.io/openshift4/
<http://registry.redhat.io/openshift4/>>
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Main process exited,
code=exited, status=125/n/a
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Failed with result
'exit-code'.
Nov 18 21:14:21 vm-00 systemd[1]: test.service: Service
RestartSec=100ms expired, scheduling restart.
i did a little more debugging and it seems that systemd does not know
where the auth file is
Nov 18 21:19:09 vm-00 systemd[1]: Started Redis container.
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Reading configuration file
\"/usr/share/containers/libpod.conf\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Merged system config
\"/usr/share/containers/libpod.conf\": &{{false false false false
false true} 0 { [] [] []} docker:// runc map[crun:[/usr/bin/crun
/usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun
/bin/crun /run/current-system/sw/bin/crun]
kata-fc:[/usr/bin/kata-fc] kata->
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using conmon: \"/usr/bin/conmon\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Initializing boltdb state at
/var/lib/containers/storage/libpod/bolt_state.db"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using graph driver overlay"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using graph root /var/lib/containers/storage"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using run root /var/run/containers/storage"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using static dir /var/lib/containers/storage/libpod"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using tmp dir /var/run/libpod"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using volume path /var/lib/containers/storage/volumes"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Set libpod namespace to \"\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="[graphdriver] trying provided driver
\"overlay\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that overlay is supported"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that metacopy is being used"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="cached value indicated that native-diff is not
being used"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Not using native diff for overlay, this may cause
degraded performance for building images: kernel has
CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="backingFs=extfs, projectQuotaSupported=false,
useNativeDiff=false, usingMetacopy=true"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Initializing event backend journald"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-qemu: no valid executable found for OCI runtime kata-qemu:
invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-fc: no valid executable found for OCI runtime kata-fc: invalid
argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="using runtime \"/usr/bin/runc\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime crun:
no valid executable found for OCI runtime crun: invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Error initializing configured OCI runtime
kata-runtime: no valid executable found for OCI runtime
kata-runtime: invalid argument"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=info msg="Found CNI network podman (type=bridge) at
/etc/cni/net.d/87-podman-bridge.conflist"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=warning msg="Default CNI network name podman is unchangeable"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="parsed reference into
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="reference
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
does not resolve to an image ID"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="parsed reference into
\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: Trying to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5.>..
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="reference rewritten from
'registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
to 'registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Trying to access
\"registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Credentials not found"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Using registries.d directory
/etc/containers/registries.d for sigstore configuration"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg=" Using \"default-docker\" configuration"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg=" No signature storage configuration found for
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Looking for TLS certificates and private keys in
/etc/docker/certs.d/registry.redhat.io <http://registry.redhat.io>"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="GET https://registry.redhat.io/v2/"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="GET
https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/aut...
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Server response when trying to obtain an access
token: \n\"unauthorized: Please login to the Red Hat Registry using
your Customer Portal credentials. Further instructions can be found
here: https://access.redhat.com/RegistryAuthentication\""
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Accessing
\"registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\&g...
failed: unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red Hat Registry using your
Customer Portal credentials. Further instructions can be found here:
https://access.redhat.c>
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=debug msg="Error pulling image ref
//registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
Error initializing source
docker://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red Hat Registr>
Nov 18 21:19:09 vm-00 podman[10481]: unable to retrieve auth
token: invalid username/password: unauthorized: Please login to the
Red Hat Registry using your Customer Portal credentials. Further
instructions can be found here:
https://access.redhat.com/RegistryAuthentication
Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
level=error msg="unable to pull
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to pull image: Error initializing source
docker://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5>;:
unable to retrieve auth token: invalid username/password:
unauthorized: Please login to the Red >
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Main process exited,
code=exited, status=125/n/a
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Failed with result
'exit-code'.
Nov 18 21:19:09 vm-00 systemd[1]: test.service: Service
RestartSec=100ms expired, scheduling restart.
running 'podman login --get-login registry.redhat.io
<http://registry.redhat.io>' always shows im logged in though.
Are you aware of any reason why it seems like when running a
container from systemd it cant access the auth file to pull the
container first?
If you need anymore info or want to see it happen live im more than
happy to set up a meeting or something just let me know.
Thank you!
-Daniel Pivonka
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
4:08 a.m.
On Fri, Nov 20, 2020 at 12:41 AM Alexander Wellbrock via Podman <
podman(a)lists.podman.io> wrote:
Hey Daniel,
I don't know why podman doesn't seem to find the credentials, let me put a
guess on it in a minute. I've mitigated this however in my setup by putting
an Auth file somewhere in the system and referencing it by --authfile as
part of the podman run command.
My first guess here is you logged in to the registry with a different user
than root and using systemd without specifying the user who's actually
logged in
I echo that thought. Are you running the `podman login` as the same user?
If the systemd service runs on the system level it's running as root. In
that case, we need to `sudo podman login`.
On November 19, 2020 11:39:40 PM GMT+01:00, Tom Sweeney <
tsweeney(a)redhat.com> wrote:
>
> Daniel,
>
> Sorry about not getting back to you sooner. IDK right off the top of my
> head, but I've spun this off to the Podman mailing list, I'm sure folks
> monitoring that will have a thought or three.
>
> t
>
>
> -------- Forwarded Message --------
> Subject: podman question
> Date: Wed, 18 Nov 2020 16:26:32 -0500
> From: Daniel Pivonka <dpivonka(a)redhat.com> <dpivonka(a)redhat.com>
> To: Tom Sweeney <tsweeney(a)redhat.com> <tsweeney(a)redhat.com>
>
> Hi Tom,
>
> One of my coworkers pointed me to you about a podman issue I'm having.
> I'm hoping you can help me or point me in the right direction.
>
> I work on the ceph orchestration team and I'm facing an issue when trying
> to deploy containers from an authenticated registry where podman can't seem
> to access the registry login info.
>
> I'm trying to run containers from systemd in a way similar to this
>
<https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_at...
>
>
> The image im trying to use comes from registry.redhat.io
>
> so as a test i ran podman login first
>
> then starting my service with this unit file
>
> [Unit]
> Description=Redis container
>
> [Service]
> Restart=always
> ExecStart=/bin/podman run --rm --ipc=host --net=host --name
> ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb-node-exporter.vm-00 --user 65534
> -d --conmon-pidfile
> /run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-pid
> --cidfile
> /run/ceph-a112bd2e-29d1-11eb-81b2-525400ea3cbb(a)node-exporter.vm-00.service-cid
> -e CONTAINER_IMAGE=
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 -e
> NODE_NAME=vm-00 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /:/rootfs:ro
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5
> --no-collector.timex
> ExecStop=/usr/bin/podman stop -t 2 redis_server
>
> [Install]
> WantedBy=local.target
>
>
> this is similar to the unit.run file that ceph would use for its services.
>
> the service fails though and the journalctl log show that podman was not
> able to pull the image because of a failed authentication
>
>
> [root@vm-00 system]# journalctl -u test.service
> -- Logs begin at Wed 2020-11-18 21:04:45 UTC, end at Wed 2020-11-18
> 21:14:22 UTC. --
> Nov 18 21:14:20 vm-00 systemd[1]: Started Redis container.
> Nov 18 21:14:21 vm-00 podman[9652]: 2020-11-18 21:14:21.066551744 +0000
> UTC m=+0.234565900 system refresh
> Nov 18 21:14:21 vm-00 podman[9652]: Trying to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5...
> Nov 18 21:14:21 vm-00 podman[9652]: unable to retrieve auth token:
> invalid username/password: unauthorized: Please login to the Red Hat
> Registry using your Customer Portal credentials. Further instructions ca>
> Nov 18 21:14:21 vm-00 podman[9652]: Error: unable to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to pull image: Error initializing source docker://
> registry.redhat.io/openshift4/>
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Main process exited,
> code=exited, status=125/n/a
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Failed with result
> 'exit-code'.
> Nov 18 21:14:21 vm-00 systemd[1]: test.service: Service RestartSec=100ms
> expired, scheduling restart.
>
>
>
>
>
> i did a little more debugging and it seems that systemd does not know
> where the auth file is
>
>
> Nov 18 21:19:09 vm-00 systemd[1]: Started Redis container.
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Reading configuration file
> \"/usr/share/containers/libpod.conf\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Merged system config
> \"/usr/share/containers/libpod.conf\": &{{false false false false
false
> true} 0 { [] [] []} docker:// runc map[crun:[/usr/bin/crun
> /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun
> /bin/crun /run/current-system/sw/bin/crun] kata-fc:[/usr/bin/kata-fc] kata->
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using conmon: \"/usr/bin/conmon\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Initializing boltdb state at
> /var/lib/containers/storage/libpod/bolt_state.db"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using graph driver overlay"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using graph root /var/lib/containers/storage"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using run root /var/run/containers/storage"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using static dir /var/lib/containers/storage/libpod"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using tmp dir /var/run/libpod"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using volume path /var/lib/containers/storage/volumes"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Set libpod namespace to \"\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="[graphdriver] trying provided driver
\"overlay\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that overlay is supported"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that metacopy is being used"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="cached value indicated that native-diff is not being
used"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Not using native diff for overlay, this may cause
> degraded performance for building images: kernel has
> CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="backingFs=extfs, projectQuotaSupported=false,
> useNativeDiff=false, usingMetacopy=true"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Initializing event backend journald"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-qemu: no
> valid executable found for OCI runtime kata-qemu: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-fc: no
> valid executable found for OCI runtime kata-fc: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="using runtime \"/usr/bin/runc\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime crun: no valid
> executable found for OCI runtime crun: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Error initializing configured OCI runtime kata-runtime:
> no valid executable found for OCI runtime kata-runtime: invalid argument"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=info msg="Found CNI network podman (type=bridge) at
> /etc/cni/net.d/87-podman-bridge.conflist"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=warning msg="Default CNI network name podman is unchangeable"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="parsed reference into \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="reference \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
>
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> does not resolve to an image ID"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="parsed reference into \"[
>
overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on
> ]registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: Trying to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5...
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="reference rewritten from '
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5' to '
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5'"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Trying to access \"
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
> <http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> ""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Credentials not found"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Using registries.d directory /etc/containers/registries.d
> for sigstore configuration"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg=" Using \"default-docker\" configuration"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg=" No signature storage configuration found for
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Looking for TLS certificates and private keys in
> /etc/docker/certs.d/registry.redhat.io"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="GET https://registry.redhat.io/v2/"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Ping https://registry.redhat.io/v2/ status 401"
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="GET
>
https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/aut...
> "
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Server response when trying to obtain an access token:
> \n\"unauthorized: Please login to the Red Hat Registry using your Customer
> Portal credentials. Further instructions can be found here:
> https://access.redhat.com/RegistryAuthentication\""
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Accessing \"
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5\
>
<http://registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5%5C...
> failed: unable to retrieve auth token: invalid username/password:
> unauthorized: Please login to the Red Hat Registry using your Customer
> Portal credentials. Further instructions can be found here:
> https://access.redhat.c>
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=debug msg="Error pulling image ref //
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: Error
> initializing source docker://
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to retrieve auth token: invalid username/password: unauthorized: Please
> login to the Red Hat Registr>
> Nov 18 21:19:09 vm-00 podman[10481]: unable to retrieve auth token:
> invalid username/password: unauthorized: Please login to the Red Hat
> Registry using your Customer Portal credentials. Further instructions can
> be found here: https://access.redhat.com/RegistryAuthentication
> Nov 18 21:19:09 vm-00 podman[10481]: time="2020-11-18T21:19:09Z"
> level=error msg="unable to pull
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to pull image: Error initializing source docker://
> registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5: unable
> to retrieve auth token: invalid username/password: unauthorized: Please
> login to the Red >
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Main process exited,
> code=exited, status=125/n/a
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Failed with result
> 'exit-code'.
> Nov 18 21:19:09 vm-00 systemd[1]: test.service: Service RestartSec=100ms
> expired, scheduling restart.
>
>
>
> running 'podman login --get-login registry.redhat.io' always shows im
> logged in though.
>
>
> Are you aware of any reason why it seems like when running a
> container from systemd it cant access the auth file to pull the container
> first?
>
> If you need anymore info or want to see it happen live im more than happy
> to set up a meeting or something just let me know.
>
>
> Thank you!
> -Daniel Pivonka
>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
1462
days inactive
1463
days old
2 comments
3 participants
participants (3)
-
Alexander Wellbrock -
Tom Sweeney -
Valentin Rothberg