Storage directory perm/mismatch error with LDAP user home on NFS
Hello, podman fails with directory permission errors or directory mismatch errors when I do a pull on my Ubuntu 20.x with an NFS mounted LDAP user home directory. Details are provided below. Would you be able to advise on the best way to resolve the issue? Thanks [user@user-vm2 opr:0]$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL=https://www.ubuntu.com/ SUPPORT_URL=https://help.ubuntu.com/ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy VERSION_CODENAME=focal UBUNTU_CODENAME=focal [user@user-vm2 opr:0]$ [user@user-vm2 opr:127]$ podman --version podman version 4.5.1 [user@user-vm2 opr:125]$ podman pull --log-level debug alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/user/.local/share/containers/storage DEBU[0000] Using run root /run/user/7148269/containers DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Using OCI runtime "/usr/sbin/runc" INFO[0000] Setting parallel job count to 13 DEBU[0000] Pulling image alpine (policy: always) DEBU[0000] Looking up image "alpine" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Trying "localhost/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "alpine" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest" DEBU[0000] Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Trying to pull docker.io/library/alpine:latest... DEBU[0000] Copying source image //alpine:latest to destination image [vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "docker.io/library/alpine:latest" DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/7148269/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.config/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.docker/config.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.dockercfg DEBU[0000] No credentials for docker.io/library/alpine found DEBU[0000] No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/user/.local/share/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0001] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0001] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32ad1... DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339c5e... Getting image source signatures DEBU[0001] Reading /home/user/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1 DEBU[0001] Not looking for sigstore attachments: disabled by configuration DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] DEBU[0001] ... will first try using the original manifest unmodified DEBU[0001] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534f9a... DEBU[0001] Detected compression format gzip DEBU[0001] Using original blob without modification Copying blob 31e352740f53 done DEBU[0001] Start untar layer ERRO[0001] While applying layer: ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360:Copying blob 31e352740f53 done DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 DEBU[0001] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/user/7148269/containers" with "/run/containers/storage" from database ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: mkdir /run/containers/storage: permission denied DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /space/containers/run DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: database storage temporary directory (runroot) "/run/containers/storage" does not match our storage temporary directory (runroot) "/space/containers/run": database configuration mismatch DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$
On 6/25/23 05:48, None via Podman wrote:
Hello,
podman fails with directory permission errors or directory mismatch errors when I do a pull on my Ubuntu 20.x with an NFS mounted LDAP user home directory. Details are provided below. Would you be able to advise on the best way to resolve the issue?
Thanks Please open an issue. Note though that we would want you to reproduce on a newer version of Podman if the Ubuntu 20.04.6 version is ancient.
[user@user-vm2 opr:0]$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL=https://www.ubuntu.com/ SUPPORT_URL=https://help.ubuntu.com/ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy VERSION_CODENAME=focal UBUNTU_CODENAME=focal [user@user-vm2 opr:0]$ [user@user-vm2 opr:127]$ podman --version podman version 4.5.1 [user@user-vm2 opr:125]$ podman pull --log-level debug alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/user/.local/share/containers/storage DEBU[0000] Using run root /run/user/7148269/containers DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Using OCI runtime "/usr/sbin/runc" INFO[0000] Setting parallel job count to 13 DEBU[0000] Pulling image alpine (policy: always) DEBU[0000] Looking up image "alpine" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Trying "localhost/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "alpine" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest" DEBU[0000] Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Trying to pull docker.io/library/alpine:latest... DEBU[0000] Copying source image //alpine:latest to destination image [vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "docker.io/library/alpine:latest" DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/7148269/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.config/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.docker/config.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.dockercfg DEBU[0000] No credentials for docker.io/library/alpine found DEBU[0000] No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/user/.local/share/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0001] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0001] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32ad1... DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339c5e... Getting image source signatures DEBU[0001] Reading /home/user/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1 DEBU[0001] Not looking for sigstore attachments: disabled by configuration DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] DEBU[0001] ... will first try using the original manifest unmodified DEBU[0001] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534f9a... DEBU[0001] Detected compression format gzip DEBU[0001] Using original blob without modification Copying blob 31e352740f53 done DEBU[0001] Start untar layer ERRO[0001] While applying layer: ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360:Copying blob 31e352740f53 done DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 DEBU[0001] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/user/7148269/containers" with "/run/containers/storage" from database ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: mkdir /run/containers/storage: permission denied DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /space/containers/run DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: database storage temporary directory (runroot) "/run/containers/storage" does not match our storage temporary directory (runroot) "/space/containers/run": database configuration mismatch DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$ _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Based on feedback from Valentin Rothburg and the output of the command below (required for the bug report attempt): ------------------------------------------------------------------------ [km@vm2 opr:0]$ podman version ERRO[0000] cannot find UID/GID for user km: no subuid ranges found for user "km" in /etc/subuid - check rootless mode in man pages. WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user Client: Podman Engine Version: 4.5.1 API Version: 4.5.1 Go Version: go1.20.5 Built: Thu Jan 1 00:00:00 1970 OS/Arch: linux/amd64 [km@vm2 opr:0]$ ------------------------------------------------------------------------ I added entries to /etc/subuid and /etc/subgid and the following lines to ~/.config/containers/storage.conf: ------------------------------------------------------------------------------------ [storage] # Default Storage Driver, Must be set for proper operation. driver = "overlay" rootless_storage_path = "/space/containers/storage" ------------------------------------------------------------------------------------ This followed by sudo apt install uidmap podman system migrate (to update uidmap and gidmap) podman system reset (along with manual cleanup of storage directories) helped me get image operations working with podman Thanks a lot for all your help. FYI, Here is what the command debug output looks like: [km@vm2 opr:0]$ podman info host: arch: amd64 buildahVersion: 1.30.0 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon_100:2.1.7-1_amd64 path: /usr/bin/conmon version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d' cpuUtilization: idlePercent: 96.95 systemPercent: 1.89 userPercent: 1.15 cpus: 4 databaseBackend: boltdb distribution: codename: focal distribution: ubuntu version: "20.04" eventLogger: file hostname: vm2 idMappings: gidmap: - container_id: 0 host_id: 23459 size: 1 - container_id: 1 host_id: 1070000 size: 65536 uidmap: - container_id: 0 host_id: 7148269 size: 1 - container_id: 1 host_id: 1070000 size: 65536 kernel: 5.15.0-1037-gcp linkmode: dynamic logDriver: k8s-file memFree: 4288712704 memTotal: 11477905408 networkBackend: cni ociRuntime: name: runc package: runc_1.1.4-0ubuntu1~20.04.3_amd64 path: /usr/sbin/runc version: |- runc version 1.1.4-0ubuntu1~20.04.3 spec: 1.0.2-dev go: go1.18.1 libseccomp: 2.5.1 os: linux remoteSocket: path: /run/user/7148269/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: "" package: "" version: "" swapFree: 0 swapTotal: 0 uptime: 58h 31m 38.00s (Approximately 2.42 days) plugins: authorization: null log: - k8s-file - none - passthrough network: - bridge - macvlan - ipvlan volume: - local registries: search: - docker.io store: configFile: /home/km/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /space/containers/storage graphRootAllocated: 207929917440 graphRootUsed: 67902648320 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 0 runRoot: /run/user/7148269/containers transientStore: false volumePath: /space/containers/storage/volumes version: APIVersion: 4.5.1 Built: 0 BuiltTime: Thu Jan 1 00:00:00 1970 GitCommit: "" GoVersion: go1.20.5 Os: linux OsArch: linux/amd64 Version: 4.5.1 [km@vm2 opr:0]$ [km@vm2 opr:130]$ podman pull --log-level debug alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /run/user/7148269/containers DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is usable DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Using OCI runtime "/usr/sbin/runc" INFO[0000] Setting parallel job count to 13 DEBU[0000] Pulling image alpine (policy: always) DEBU[0000] Looking up image "alpine" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Trying "localhost/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "alpine" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine DEBU[0000] parsed reference into "[overlay@/space/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest" DEBU[0000] Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Trying to pull docker.io/library/alpine:latest... DEBU[0000] Copying source image //alpine:latest to destination image [overlay@/space/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "docker.io/library/alpine:latest" DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/7148269/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/km/.config/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/km/.docker/config.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/km/.dockercfg DEBU[0000] No credentials for docker.io/library/alpine found DEBU[0000] No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/km/.local/share/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0000] Using blob info cache at /home/km/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0000] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32ad1... DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339c5e... Getting image source signatures DEBU[0001] Reading /home/km/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1 DEBU[0001] Not looking for sigstore attachments: disabled by configuration DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] DEBU[0001] ... will first try using the original manifest unmodified DEBU[0001] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534f9a... DEBU[0001] Detected compression format gzip DEBU[0001] Using original blob without modification Copying blob 31e352740f53 done DEBU[0001] Check for idmapped mounts support create mapped mount: operation not permitted Copying blob 31e352740f53 done DEBU[0001] No compression detected DEBU[0001] Compression change for blob sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 ("application/vnd.docker.container.image.v1+json") not supported DEBU[0001] Using original blob without modification Copying config c1aabb73d2 done Writing manifest to image destination Storing signatures DEBU[0001] setting image creation date to 2023-06-14 20:41:59.079795125 +0000 UTC DEBU[0001] created new image ID "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" DEBU[0001] saved image metadata "{\"signatures-sizes\":{\"sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70\":[]}}" DEBU[0001] added name "docker.io/library/alpine:latest" to image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" DEBU[0001] Pulled candidate docker.io/library/alpine:latest successfully DEBU[0001] Looking up image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage DEBU[0001] Trying "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" ... DEBU[0001] parsed reference into "[overlay@/space/containers/storage+/run/user/7148269/containers]@c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" DEBU[0001] Found image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" as "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage DEBU[0001] Found image "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" as "c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" in local containers storage ([overlay@/space/containers/storage+/run/user/7148269/containers]@c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67) DEBU[0001] exporting opaque data as blob "sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 DEBU[0001] Called pull.PersistentPostRunE(podman pull --log-level debug alpine) DEBU[0001] Shutting down engines [km@vm2 opr:0]$ [km@vm2 opr:0]$ [km@vm2 opr:0]$ [km@vm2 opr:0]$ [km@vm2 opr:0]$ [km@vm2 opr:0]$ podman --log-level debug images INFO[0000] podman filtering at log level debug DEBU[0000] Called images.PersistentPreRunE(podman --log-level debug images) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /run/user/7148269/containers DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is usable DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Using OCI runtime "/usr/sbin/runc" INFO[0000] Setting parallel job count to 13 DEBU[0000] parsed reference into "[overlay@/space/containers/storage+/run/user/7148269/containers]@c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" DEBU[0000] exporting opaque data as blob "sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" DEBU[0000] exporting opaque data as blob "sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67" REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/alpine latest c1aabb73d233 3 weeks ago 7.63 MB DEBU[0000] Called images.PersistentPostRunE(podman --log-level debug images) DEBU[0000] Shutting down engines [km@vm2 opr:0]$
participants (2)
-
Daniel Walsh -
kurien.mathew＠mediakind.com