Hello, podman fails with directory permission errors or directory mismatch errors when I do a pull on my Ubuntu 20.x with an NFS mounted LDAP user home directory. Details are provided below. Would you be able to advise on the best way to resolve the issue? Thanks [user@user-vm2 opr:0]$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL=https://www.ubuntu.com/ SUPPORT_URL=https://help.ubuntu.com/ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy VERSION_CODENAME=focal UBUNTU_CODENAME=focal [user@user-vm2 opr:0]$ [user@user-vm2 opr:127]$ podman --version podman version 4.5.1 [user@user-vm2 opr:125]$ podman pull --log-level debug alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/user/.local/share/containers/storage DEBU[0000] Using run root /run/user/7148269/containers DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Using OCI runtime "/usr/sbin/runc" INFO[0000] Setting parallel job count to 13 DEBU[0000] Pulling image alpine (policy: always) DEBU[0000] Looking up image "alpine" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Trying "localhost/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "docker.io/library/alpine:latest" ... DEBU[0000] Trying "alpine" ... DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine DEBU[0000] parsed reference into "[vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest" DEBU[0000] Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf) Trying to pull docker.io/library/alpine:latest... DEBU[0000] Copying source image //alpine:latest to destination image [vfs@/home/user/.local/share/containers/storage+/run/user/7148269/containers]docker.io/library/alpine:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "docker.io/library/alpine:latest" DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/7148269/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.config/containers/auth.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.docker/config.json DEBU[0000] No credentials matching docker.io/library/alpine found in /home/user/.dockercfg DEBU[0000] No credentials for docker.io/library/alpine found DEBU[0000] No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/user/.local/share/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/ DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0001] Using blob info cache at /home/user/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0001] Source is a manifest list; copying (only) instance sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 for current system DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:25fad2a32ad1... DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c1aabb73d2339c5e... Getting image source signatures DEBU[0001] Reading /home/user/.local/share/containers/sigstore/library/alpine@sha256=25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70/signature-1 DEBU[0001] Not looking for sigstore attachments: disabled by configuration DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] DEBU[0001] ... will first try using the original manifest unmodified DEBU[0001] Checking if we can reuse blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0001] Failed to retrieve partial blob: blob type not supported for partial retrieval DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:31e352740f534f9a... DEBU[0001] Detected compression format gzip DEBU[0001] Using original blob without modification Copying blob 31e352740f53 done DEBU[0001] Start untar layer ERRO[0001] While applying layer: ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360:Copying blob 31e352740f53 done DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /home/user/.local/share/containers/storage/vfs/dir/78a822fe2a2d2c84f3de4a403188c45f623017d6a4521d23047c9fbb0801794c/.pivot_root3008513360: permission denied exit status 1 DEBU[0001] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/user/7148269/containers" with "/run/containers/storage" from database ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: mkdir /run/containers/storage: permission denied DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ [user@user-vm2 opr:125]$ podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug --root /space/containers/storage --runroot /space/containers/run alpine) DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /space/containers/storage/libpod/bolt_state.db ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files ("/space/containers/storage") to resolve. May prevent use of images created by other tools DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /space/containers/storage DEBU[0000] Using run root /space/containers/run DEBU[0000] Using static dir /space/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/7148269/libpod/tmp DEBU[0000] Using volume path /space/containers/storage/volumes DEBU[0000] Using transient store: false Error: database storage temporary directory (runroot) "/run/containers/storage" does not match our storage temporary directory (runroot) "/space/containers/run": database configuration mismatch DEBU[0000] Shutting down engines [user@user-vm2 opr:125]$