[Podman] Re: docker registry token authentication and podman

Hallo Miloslav, I run registry and cesanta/docker_auth in a single pod. Registry listens on port 5004, docker_auth - on port 5005. Ports are bublished with same numbers. Access to registry is preformed through apachei2 proxy which runs on the same system. Access to docker_auth service is performed directly to port 5005 using unencrypted http protocol. Authentication in registry is configured as follows:    auth:      token:        realm:http://regtest-auth.intern.local:5005/auth        service: regtest-auth.intern.local        issuer: "ACME auth server - aa8AhshuoCh5eade"        rootcertbundle: /certs/sign-ca.pem Corresponding part on docker auth is configured as follows:    server:      addr: ":5005"    token:      issuer: "ACME auth server - aa8AhshuoCh5eade"      expiration: 900      certificate: "/config/sign-ca.pem"      key: "/config/ra-private.pem" I run podman login to test authenticationnas follows:    island:podman [v4.7]> strace -f -o /tmp/podman.trace -s 16384 ./bin/podman --log-level debug login regtest.intern.local    INFO[0000] ./bin/podman filtering at log level debug    DEBU[0000] Called login.PersistentPreRunE(./bin/podman --log-level debug login regtest.intern.local)    DEBU[0000] Using conmon: "/usr/bin/conmon"    DEBU[0000] Initializing boltdb state at /home/ivans/.local/share/containers/storage/libpod/bolt_state.db    DEBU[0000] Using graph driver vfs    DEBU[0000] Using graph root /home/ivans/.local/share/containers/storage    DEBU[0000] Using run root /run/user/1007/containers    DEBU[0000] Using static dir /home/ivans/.local/share/containers/storage/libpod    DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp    DEBU[0000] Using volume path /home/ivans/.local/share/containers/storage/volumes    DEBU[0000] Using transient store: false    DEBU[0000] [graphdriver] trying provided driver "vfs"    DEBU[0000] Initializing event backend file    DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument    DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument    DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument    DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument    DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument    DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument    DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument    DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument    DEBU[0000] Using OCI runtime "/usr/bin/crun"    INFO[0000] Setting parallel job count to 25    DEBU[0000] Loading registries configuration "/home/ivans/.config/containers/registries.conf"    DEBU[0000] No credentials matching regtest.intern.local found in /run/user/1007/containers/auth.json    DEBU[0000] No credentials matching regtest.intern.local found in /home/ivans/.config/containers/auth.json    DEBU[0000] No credentials matching regtest.intern.local found in /home/ivans/.docker/config.json    DEBU[0000] No credentials matching regtest.intern.local found in /home/ivans/.dockercfg    DEBU[0000] No credentials for regtest.intern.local found    Username: ivans    Password:    DEBU[0028] Looking for TLS certificates and private keys in /etc/docker/certs.d/regtest.intern.local    DEBU[0028] GEThttps://regtest.intern.local/v2/    DEBU[0028] Pinghttps://regtest.intern.local/v2/ status 401    DEBU[0028] GEThttps://regtest.intern.local/v2/    PARSE HEADER [[Bearer realm="http://regtest-auth.intern.local:5005/auth",service="regtest-auth.intern.local"]]       VALUE: [bearer], PARAMETER: [0xc00052c3f0]    CHALLENGES: [[[1/1]0xc000a861f8]]        bearer           [realm] => [http://regtest-auth.intern.local:5005/auth]           [service] => [regtest-auth.intern.local]    DEBU[0029] error logging into "regtest.intern.local": unable to retrieve auth token: invalid username/password: unauthorized    Error: logging into "regtest.intern.local": invalid username/password    DEBU[0029] Shutting down engines I have added some test messages to podman to verify that it receives WWW-Authenticate header with correct parameters. I also captured the traffic to registry and docker_auth with following command:    tcpdump -ni any port 5005 or port 5004 -s 0 -w /tmp/auth.dump Capture shows only one HTTP exchange:    GET /v2/ HTTP/1.1    Host: regtest.intern.local    User-Agent: containers/5.28.0 (github.com/containers/image)    Authorization: Basic aXZhbnM6TGVuYSMyMDc0    Docker-Distribution-Api-Version: registry/2.0    Accept-Encoding: gzip    X-Forwarded-Proto: https    X-Forwarded-For: 10.255.225.67    X-Forwarded-Host: regtest.intern.local    X-Forwarded-Server: regtest.intern.local    Connection: Keep-Alive    HTTP/1.1 401 Unauthorized    Content-Type: application/json; charset=utf-8    Docker-Distribution-Api-Version: registry/2.0    Www-Authenticate: Bearer realm="http://regtest-auth.intern.local:5005/auth",service="regtest-auth.intern.local"    Date: Wed, 06 Dec 2023 20:13:36 GMT    Content-Length: 87    {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]} As far as I understand podman should try to connect to url specified by realm in WWW-Authenicate header and request a token from it. But as I see form capture no such attempt is detected. I also verified this in strace output, the only tcp connect attempts reported are following:   1 2624033 connect(7, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.194.99.71")}, 16 <unfinished ...>   2 2624033 connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("10.194.99.42")}, 16 <unfinished ...>   3 2624042 connect(7, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.194.99.71")}, 16 <unfinished ...>   4 2624042 connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("10.194.99.42")}, 16 <unfinished ...> (port 53 is evidently name resolver) I tried this with podman from os (4.7.2, debian unstable) and podman build from v4.7 branch (version reported - 4.7.3-dev) with same results. Rgrds, On 06.12.2023 17:31, Miloslav Trmac wrote: