11:16 a.m.
We have a cluster of machines where /home is a remote gluster mount. Running podman
rootless nicely solves the problem of accessing the remote filesystem with user
credentials. Since remote filesystems do not currently support namespaces, podman is run
with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first
client machine, but an image pulled successfully on one machine will fail to pull on a
second. For example, on the second machine:
$ podman run --rm -it ubuntu
Trying to pull docker.io/library/ubuntu...Getting image source signatures
Copying blob c58094023a2e done
Copying blob 079b6d2a1e53 done
Copying blob 11048ebae908 done
Copying blob 22e816666fd6 done
Copying config cf0f3ca922 done
Writing manifest to image destination
Storing signatures
ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown
/etc/gshadow: operation not permitted
ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image:
error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
Failed
Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref
//registry.fedoraproject.org/ubuntu:latest: Error initializing source
docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in
registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown
Failed
Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref
//quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error
reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid
character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC
\"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not
Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
found on the server. If you entered the URL manually please check your spelling and try
again.</p>\n"
Failed
Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref
//registry.centos.org/ubuntu:latest: Error initializing source
docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in
registry.centos.org/ubuntu: manifest unknown: manifest unknown
Failed
Error: unable to pull ubuntu: 4 errors occurred:
* Error committing the finished image: error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
* Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error
reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest
unknown
* Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest
in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<'
looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2
Final//EN\">\n<title>404 Not Found</title>\n<h1>Not
Found</h1>\n<p>The requested URL was not found on the server. If you entered
the URL manually please check your spelling and try again.</p>\n"
* Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading
manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown
Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared
/home filesystem.
Is there a suggested approach to running rootless podman on multiple machines with a
shared /home directory?
Thanks,
Eddie
1:31 p.m.
New subject: Sharing blob-info-cache-v1.boltdb across multiple machines
On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae(a)us.ibm.com wrote:
We have a cluster of machines where /home is a remote gluster mount.
Running podman rootless nicely solves the problem of accessing the remote filesystem with
user credentials. Since remote filesystems do not currently support namespaces, podman is
run with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first
client machine, but an image pulled successfully on one machine will fail to pull on a
second. For example, on the second machine:
$ podman run --rm -it ubuntu
Trying to pull docker.io/library/ubuntu...Getting image source signatures
Copying blob c58094023a2e done
Copying blob 079b6d2a1e53 done
Copying blob 11048ebae908 done
Copying blob 22e816666fd6 done
Copying config cf0f3ca922 done
Writing manifest to image destination
Storing signatures
ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown
/etc/gshadow: operation not permitted
ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image:
error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
Failed
Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref
//registry.fedoraproject.org/ubuntu:latest: Error initializing source
docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in
registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown
Failed
Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref
//quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error
reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid
character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC
\"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not
Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
found on the server. If you entered the URL manually please check your spelling and try
again.</p>\n"
Failed
Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref
//registry.centos.org/ubuntu:latest: Error initializing source
docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in
registry.centos.org/ubuntu: manifest unknown: manifest unknown
Failed
Error: unable to pull ubuntu: 4 errors occurred:
* Error committing the finished image: error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
* Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error
reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest
unknown
* Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest
latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character
'<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC
\"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not
Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
found on the server. If you entered the URL manually please check your spelling and try
again.</p>\n"
* Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading
manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown
Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared
/home filesystem.
Is there a suggested approach to running rootless podman on multiple machines with a
shared /home directory?
To run Podman in an HPC like environment with /home on NFS, I am doing
the following steps to set up Podman for each user:
$ podman info
$ sed -e "s,graphroot.*$,graphroot = \"/tmp/container\",g" -i
.config/containers/storage.conf'
$ rm -f ./.local/share/containers/storage/libpod/bolt_state.db
./.local/share/containers/cache/blob-info-cache-v1.boltdb"
If a user now uses Podman it just works. This is for a CentOS 7.7 based
system. Maybe that helps for your use case also.
Adrian
3:24 p.m.
New subject: Sharing blob-info-cache-v1.boltdb across multiple machines
On 10/23/19 2:31 PM, Adrian Reber wrote:
On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae(a)us.ibm.com wrote:
> We have a cluster of machines where /home is a remote gluster mount. Running podman
rootless nicely solves the problem of accessing the remote filesystem with user
credentials. Since remote filesystems do not currently support namespaces, podman is run
with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first
client machine, but an image pulled successfully on one machine will fail to pull on a
second. For example, on the second machine:
>
> $ podman run --rm -it ubuntu
> Trying to pull docker.io/library/ubuntu...Getting image source signatures
> Copying blob c58094023a2e done
> Copying blob 079b6d2a1e53 done
> Copying blob 11048ebae908 done
> Copying blob 22e816666fd6 done
> Copying config cf0f3ca922 done
> Writing manifest to image destination
> Storing signatures
> ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr:
lchown /etc/gshadow: operation not permitted
> ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished
image: error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
> Failed
> Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref
//registry.fedoraproject.org/ubuntu:latest: Error initializing source
docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in
registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown
> Failed
> Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref
//quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error
reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid
character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC
\"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not
Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
found on the server. If you entered the URL manually please check your spelling and try
again.</p>\n"
> Failed
> Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref
//registry.centos.org/ubuntu:latest: Error initializing source
docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in
registry.centos.org/ubuntu: manifest unknown: manifest unknown
> Failed
> Error: unable to pull ubuntu: 4 errors occurred:
> * Error committing the finished image: error adding layer with blob
"sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91":
ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted
> * Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error
reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest
unknown
> * Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest
latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character
'<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC
\"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not
Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not
found on the server. If you entered the URL manually please check your spelling and try
again.</p>\n"
> * Error initializing source docker://registry.centos.org/ubuntu:latest: Error
reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown
>
> Our guess is that this is happening because blob-info-cache-v1.boltdb is in the
shared /home filesystem.
>
> Is there a suggested approach to running rootless podman on multiple machines with a
shared /home directory?
To run Podman in an HPC like environment with /home on NFS, I am doing
the following steps to set up Podman for each user:
$ podman info
$ sed -e "s,graphroot.*$,graphroot = \"/tmp/container\",g" -i
.config/containers/storage.conf'
$ rm -f ./.local/share/containers/storage/libpod/bolt_state.db
./.local/share/containers/cache/blob-info-cache-v1.boltdb"
If a user now uses Podman it just works. This is for a CentOS 7.7 based
system. Maybe that helps for your use case also.
Adrian
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
I think a nice blog on how to run podman on an NFS Homedir would be
something people could use.
10:49 a.m.
New subject: Sharing blob-info-cache-v1.boltdb across multiple machines
Hi Adrian,
Thanks for the suggestion. This does allow my user to run rootless podman on multiple
machines with a shared /home directory, but other user IDs with the same configuration are
blocked:
$ podman info
Error: could not get runtime: mkdir /tmp/container/mounts: permission denied
Is this expected?
Thanks,
Eddie
2:25 p.m.
New subject: Sharing blob-info-cache-v1.boltdb across multiple machines
Ooops, silly me. Setting graphroot = /tmp/user/container solves that problem.
Thanks again
1638
days inactive
1639
days old
4 comments
3 participants
participants (3)
-
Adrian Reber -
Daniel Walsh -
eae＠us.ibm.com