Sharing blob-info-cache-v1.boltdb across multiple machines
We have a cluster of machines where /home is a remote gluster mount. Running podman rootless nicely solves the problem of accessing the remote filesystem with user credentials. Since remote filesystems do not currently support namespaces, podman is run with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first client machine, but an image pulled successfully on one machine will fail to pull on a second. For example, on the second machine: $ podman run --rm -it ubuntu Trying to pull docker.io/library/ubuntu...Getting image source signatures Copying blob c58094023a2e done Copying blob 079b6d2a1e53 done Copying blob 11048ebae908 done Copying blob 22e816666fd6 done Copying config cf0f3ca922 done Writing manifest to image destination Storing signatures ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted Failed Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown Failed Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" Failed Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref //registry.centos.org/ubuntu:latest: Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Failed Error: unable to pull ubuntu: 4 errors occurred: * Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted * Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown * Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" * Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared /home filesystem. Is there a suggested approach to running rootless podman on multiple machines with a shared /home directory? Thanks, Eddie
On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae@us.ibm.com wrote:
We have a cluster of machines where /home is a remote gluster mount. Running podman rootless nicely solves the problem of accessing the remote filesystem with user credentials. Since remote filesystems do not currently support namespaces, podman is run with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first client machine, but an image pulled successfully on one machine will fail to pull on a second. For example, on the second machine:
$ podman run --rm -it ubuntu Trying to pull docker.io/library/ubuntu...Getting image source signatures Copying blob c58094023a2e done Copying blob 079b6d2a1e53 done Copying blob 11048ebae908 done Copying blob 22e816666fd6 done Copying config cf0f3ca922 done Writing manifest to image destination Storing signatures ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted Failed Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown Failed Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" Failed Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref //registry.centos.org/ubuntu:latest: Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Failed Error: unable to pull ubuntu: 4 errors occurred: * Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted * Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown * Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" * Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown
Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared /home filesystem.
Is there a suggested approach to running rootless podman on multiple machines with a shared /home directory?
To run Podman in an HPC like environment with /home on NFS, I am doing the following steps to set up Podman for each user: $ podman info $ sed -e "s,graphroot.*$,graphroot = \"/tmp/container\",g" -i .config/containers/storage.conf' $ rm -f ./.local/share/containers/storage/libpod/bolt_state.db ./.local/share/containers/cache/blob-info-cache-v1.boltdb" If a user now uses Podman it just works. This is for a CentOS 7.7 based system. Maybe that helps for your use case also. Adrian
On 10/23/19 2:31 PM, Adrian Reber wrote:
We have a cluster of machines where /home is a remote gluster mount. Running podman rootless nicely solves the problem of accessing the remote filesystem with user credentials. Since remote filesystems do not currently support namespaces, podman is run with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first client machine, but an image pulled successfully on one machine will fail to pull on a second. For example, on the second machine:
$ podman run --rm -it ubuntu Trying to pull docker.io/library/ubuntu...Getting image source signatures Copying blob c58094023a2e done Copying blob 079b6d2a1e53 done Copying blob 11048ebae908 done Copying blob 22e816666fd6 done Copying config cf0f3ca922 done Writing manifest to image destination Storing signatures ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted Failed Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown Failed Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" Failed Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref //registry.centos.org/ubuntu:latest: Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Failed Error: unable to pull ubuntu: 4 errors occurred: * Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted * Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown * Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" * Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown
Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared /home filesystem.
Is there a suggested approach to running rootless podman on multiple machines with a shared /home directory? To run Podman in an HPC like environment with /home on NFS, I am doing
On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae@us.ibm.com wrote: the following steps to set up Podman for each user:
$ podman info $ sed -e "s,graphroot.*$,graphroot = \"/tmp/container\",g" -i .config/containers/storage.conf' $ rm -f ./.local/share/containers/storage/libpod/bolt_state.db ./.local/share/containers/cache/blob-info-cache-v1.boltdb"
If a user now uses Podman it just works. This is for a CentOS 7.7 based system. Maybe that helps for your use case also.
Adrian _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
I think a nice blog on how to run podman on an NFS Homedir would be something people could use.
Hi Adrian, Thanks for the suggestion. This does allow my user to run rootless podman on multiple machines with a shared /home directory, but other user IDs with the same configuration are blocked: $ podman info Error: could not get runtime: mkdir /tmp/container/mounts: permission denied Is this expected? Thanks, Eddie
Ooops, silly me. Setting graphroot = /tmp/user/container solves that problem. Thanks again
participants (3)
-
Adrian Reber -
Daniel Walsh -
eae＠us.ibm.com