Storage directory perm/mismatch error with LDAP user home on NFS
by kurien.mathew@mediakind.com
Hello,
podman fails with directory permission errors or directory mismatch errors when I do a pull on my Ubuntu 20.x with an NFS mounted LDAP user home directory. Details are provided below. Would you be able to advise on the best way to resolve the issue?
Thanks
[user@user-vm2 opr:0]$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL=https://www.ubuntu.com/
SUPPORT_URL=https://help.ubuntu.com/
BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privac...
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[user@user-vm2 opr:0]$
[user@user-vm2 opr:127]$ podman --version
podman version 4.5.1
[user@user-vm2 opr:125]$ podman pull --log-level debug alpine
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug alpine)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/user/.local/share/containers/storage
DEBU[0000] Using run root /run/user/7148269/containers
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "vfs"
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Using OCI runtime "/usr/sbin/runc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Pulling image alpine (policy: always)
DEBU[0000] Looking up image "alpine" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/alpine:latest" ...
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] Trying "alpine" ...
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine
DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest"
DEBU[0000] Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "docker.io/library/alpine:latest"
DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/7148269/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.docker/config.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.dockercfg
DEBU[0000] No credentials for docker.io/library/alpine found
DEBU[0000] No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/user/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&s...
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0001] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0001] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32...
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest
DEBU[0001] Using default policy section
DEBU[0001] Requirement 0: allowed
DEBU[0001] Overall: allowed
DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339...
Getting image source signatures
DEBU[0001] Reading /home/user/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1
DEBU[0001] Not looking for sigstore attachments: disabled by configuration
DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0001] ... will first try using the original manifest unmodified
DEBU[0001] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval
DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534...
DEBU[0001] Detected compression format gzip
DEBU[0001] Using original blob without modification
Copying blob 31e352740f53 done
DEBU[0001] Start untar layer
ERRO[0001] While applying layer: ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360:Copying blob 31e352740f53 done
DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1
DEBU[0001] Shutting down engines
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage alpine
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage alpine)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/7148269/containers" with "/run/containers/storage" from database
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /space/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /space/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp
DEBU[0000] Using volume path /space/containers/storage/volumes
DEBU[0000] Using transient store: false
Error: mkdir /run/containers/storage: permission denied
DEBU[0000] Shutting down engines
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$
[user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /space/containers/storage
DEBU[0000] Using run root /space/containers/run
DEBU[0000] Using static dir /space/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp
DEBU[0000] Using volume path /space/containers/storage/volumes
DEBU[0000] Using transient store: false
Error: database storage temporary directory (runroot) "/run/containers/storage" does not match our storage temporary directory (runroot) "/space/containers/run": database configuration mismatch
DEBU[0000] Shutting down engines
[user@user-vm2 opr:125]$