From eae at us.ibm.com Wed Oct 23 16:16:24 2019 Content-Type: multipart/mixed; boundary="===============0554273172293916299==" MIME-Version: 1.0 From: eae at us.ibm.com To: podman at lists.podman.io Subject: [Podman] Sharing blob-info-cache-v1.boltdb across multiple machines Date: Wed, 23 Oct 2019 16:16:17 +0000 Message-ID: <20191023161617.27175.75732@lists.podman.io> --===============0554273172293916299== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable We have a cluster of machines where /home is a remote gluster mount. Runnin= g podman rootless nicely solves the problem of accessing the remote filesys= tem with user credentials. Since remote filesystems do not currently suppor= t namespaces, podman is run with --root, --runroot, and --tmpdir set to be = /tmp/$USER. All works well on the first client machine, but an image pulled= successfully on one machine will fail to pull on a second. For example, on= the second machine: $ podman run --rm -it ubuntu Trying to pull docker.io/library/ubuntu...Getting image source signatures Copying blob c58094023a2e done Copying blob 079b6d2a1e53 done Copying blob 11048ebae908 done Copying blob 22e816666fd6 done Copying config cf0f3ca922 done Writing manifest to image destination Storing signatures ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: st= derr: lchown /etc/gshadow: operation not permitted = ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the fi= nished image: error adding layer with blob "sha256:22e816666fd6516bccd19765= 947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout:= stderr: lchown /etc/gshadow: operation not permitted = Failed Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling= image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing s= ource docker://registry.fedoraproject.org/ubuntu:latest: Error reading mani= fest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifes= t unknown = Failed Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay.i= o/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: = Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 res= ponse body: invalid character '<' looking for beginning of value: "\n404 Not Found</ti= tle>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server.= If you entered the URL manually please check your spelling and try again.<= /p>\n" = Failed Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image = ref //registry.centos.org/ubuntu:latest: Error initializing source docker:/= /registry.centos.org/ubuntu:latest: Error reading manifest latest in regist= ry.centos.org/ubuntu: manifest unknown: manifest unknown = Failed Error: unable to pull ubuntu: 4 errors occurred: * Error committing the finished image: error adding layer with blob "sha25= 6:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyL= ayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not perm= itted * Error initializing source docker://registry.fedoraproject.org/ubuntu:lat= est: Error reading manifest latest in registry.fedoraproject.org/ubuntu: ma= nifest unknown: manifest unknown * Error initializing source docker://quay.io/ubuntu:latest: Error reading = manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: in= valid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC = \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found\n

Not = Found

\n

The requested URL was not found on the server. If you entere= d the URL manually please check your spelling and try again.

\n" * Error initializing source docker://registry.centos.org/ubuntu:latest: Er= ror reading manifest latest in registry.centos.org/ubuntu: manifest unknown= : manifest unknown Our guess is that this is happening because blob-info-cache-v1.boltdb is in= the shared /home filesystem. Is there a suggested approach to running rootless podman on multiple machin= es with a shared /home directory? Thanks, Eddie --===============0554273172293916299==-- From adrian at lisas.de Wed Oct 23 18:37:51 2019 Content-Type: multipart/mixed; boundary="===============3869027567488819077==" MIME-Version: 1.0 From: Adrian Reber To: podman at lists.podman.io Subject: [Podman] Re: Sharing blob-info-cache-v1.boltdb across multiple machines Date: Wed, 23 Oct 2019 20:31:01 +0200 Message-ID: <20191023183101.GI28864@lisas.de> In-Reply-To: 20191023161617.27175.75732@lists.podman.io --===============3869027567488819077== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae(a)us.ibm.com wrote: > We have a cluster of machines where /home is a remote gluster mount. Runn= ing podman rootless nicely solves the problem of accessing the remote files= ystem with user credentials. Since remote filesystems do not currently supp= ort namespaces, podman is run with --root, --runroot, and --tmpdir set to b= e /tmp/$USER. All works well on the first client machine, but an image pull= ed successfully on one machine will fail to pull on a second. For example, = on the second machine: > = > $ podman run --rm -it ubuntu > Trying to pull docker.io/library/ubuntu...Getting image source signatures > Copying blob c58094023a2e done > Copying blob 079b6d2a1e53 done > Copying blob 11048ebae908 done > Copying blob 22e816666fd6 done > Copying config cf0f3ca922 done > Writing manifest to image destination > Storing signatures > ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: = stderr: lchown /etc/gshadow: operation not permitted = > ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the = finished image: error adding layer with blob "sha256:22e816666fd6516bccd197= 65947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdou= t: stderr: lchown /etc/gshadow: operation not permitted = > Failed > Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulli= ng image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing= source docker://registry.fedoraproject.org/ubuntu:latest: Error reading ma= nifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manif= est unknown = > Failed > Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay= .io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest= : Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 r= esponse body: invalid character '<' looking for beginning of value: "\n404 Not Found</= title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the serve= r. If you entered the URL manually please check your spelling and try again= .</p>\n" = > Failed > Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling imag= e ref //registry.centos.org/ubuntu:latest: Error initializing source docker= ://registry.centos.org/ubuntu:latest: Error reading manifest latest in regi= stry.centos.org/ubuntu: manifest unknown: manifest unknown = > Failed > Error: unable to pull ubuntu: 4 errors occurred: > * Error committing the finished image: error adding layer with blob "sha= 256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": Appl= yLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not pe= rmitted > * Error initializing source docker://registry.fedoraproject.org/ubuntu:l= atest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: = manifest unknown: manifest unknown > * Error initializing source docker://quay.io/ubuntu:latest: Error readin= g manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: = invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLI= C \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found\n

No= t Found

\n

The requested URL was not found on the server. If you ente= red the URL manually please check your spelling and try again.

\n" > * Error initializing source docker://registry.centos.org/ubuntu:latest: = Error reading manifest latest in registry.centos.org/ubuntu: manifest unkno= wn: manifest unknown > = > Our guess is that this is happening because blob-info-cache-v1.boltdb is = in the shared /home filesystem. > = > Is there a suggested approach to running rootless podman on multiple mach= ines with a shared /home directory? To run Podman in an HPC like environment with /home on NFS, I am doing the following steps to set up Podman for each user: $ podman info $ sed -e "s,graphroot.*$,graphroot =3D \"/tmp/container\",g" -i .config/con= tainers/storage.conf' $ rm -f ./.local/share/containers/storage/libpod/bolt_state.db ./.local/sha= re/containers/cache/blob-info-cache-v1.boltdb" If a user now uses Podman it just works. This is for a CentOS 7.7 based system. Maybe that helps for your use case also. Adrian --===============3869027567488819077==-- From dwalsh at redhat.com Wed Oct 23 20:31:16 2019 Content-Type: multipart/mixed; boundary="===============2864586759351882657==" MIME-Version: 1.0 From: Daniel Walsh To: podman at lists.podman.io Subject: [Podman] Re: Sharing blob-info-cache-v1.boltdb across multiple machines Date: Wed, 23 Oct 2019 16:24:40 -0400 Message-ID: In-Reply-To: 20191023183101.GI28864@lisas.de --===============2864586759351882657== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 10/23/19 2:31 PM, Adrian Reber wrote: > On Wed, Oct 23, 2019 at 04:16:17PM -0000, eae(a)us.ibm.com wrote: >> We have a cluster of machines where /home is a remote gluster mount. Run= ning podman rootless nicely solves the problem of accessing the remote file= system with user credentials. Since remote filesystems do not currently sup= port namespaces, podman is run with --root, --runroot, and --tmpdir set to = be /tmp/$USER. All works well on the first client machine, but an image pul= led successfully on one machine will fail to pull on a second. For example,= on the second machine: >> >> $ podman run --rm -it ubuntu >> Trying to pull docker.io/library/ubuntu...Getting image source signatures >> Copying blob c58094023a2e done >> Copying blob 079b6d2a1e53 done >> Copying blob 11048ebae908 done >> Copying blob 22e816666fd6 done >> Copying config cf0f3ca922 done >> Writing manifest to image destination >> Storing signatures >> ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: = stderr: lchown /etc/gshadow: operation not permitted = >> ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the= finished image: error adding layer with blob "sha256:22e816666fd6516bccd19= 765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdo= ut: stderr: lchown /etc/gshadow: operation not permitted = >> Failed >> Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pull= ing image ref //registry.fedoraproject.org/ubuntu:latest: Error initializin= g source docker://registry.fedoraproject.org/ubuntu:latest: Error reading m= anifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: mani= fest unknown = >> Failed >> Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //qua= y.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:lates= t: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 = response body: invalid character '<' looking for beginning of value: "\n404 Not Found<= /title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the serv= er. If you entered the URL manually please check your spelling and try agai= n.</p>\n" = >> Failed >> Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling ima= ge ref //registry.centos.org/ubuntu:latest: Error initializing source docke= r://registry.centos.org/ubuntu:latest: Error reading manifest latest in reg= istry.centos.org/ubuntu: manifest unknown: manifest unknown = >> Failed >> Error: unable to pull ubuntu: 4 errors occurred: >> * Error committing the finished image: error adding layer with blob "sh= a256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": App= lyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not p= ermitted >> * Error initializing source docker://registry.fedoraproject.org/ubuntu:= latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu:= manifest unknown: manifest unknown >> * Error initializing source docker://quay.io/ubuntu:latest: Error readi= ng manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body:= invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBL= IC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found\n

N= ot Found

\n

The requested URL was not found on the server. If you ent= ered the URL manually please check your spelling and try again.

\n" >> * Error initializing source docker://registry.centos.org/ubuntu:latest:= Error reading manifest latest in registry.centos.org/ubuntu: manifest unkn= own: manifest unknown >> >> Our guess is that this is happening because blob-info-cache-v1.boltdb is= in the shared /home filesystem. >> >> Is there a suggested approach to running rootless podman on multiple mac= hines with a shared /home directory? > To run Podman in an HPC like environment with /home on NFS, I am doing > the following steps to set up Podman for each user: > > $ podman info > $ sed -e "s,graphroot.*$,graphroot =3D \"/tmp/container\",g" -i .config/c= ontainers/storage.conf' > $ rm -f ./.local/share/containers/storage/libpod/bolt_state.db ./.local/s= hare/containers/cache/blob-info-cache-v1.boltdb" > > If a user now uses Podman it just works. This is for a CentOS 7.7 based > system. Maybe that helps for your use case also. > > Adrian > _______________________________________________ > Podman mailing list -- podman(a)lists.podman.io > To unsubscribe send an email to podman-leave(a)lists.podman.io I think a nice blog on how to run podman on an NFS Homedir would be something people could use. --===============2864586759351882657==-- From eae at us.ibm.com Thu Oct 24 15:49:47 2019 Content-Type: multipart/mixed; boundary="===============2017933593876272758==" MIME-Version: 1.0 From: eae at us.ibm.com To: podman at lists.podman.io Subject: [Podman] Re: Sharing blob-info-cache-v1.boltdb across multiple machines Date: Thu, 24 Oct 2019 15:49:40 +0000 Message-ID: <20191024154940.27175.74957@lists.podman.io> In-Reply-To: 20191023183101.GI28864@lisas.de --===============2017933593876272758== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Adrian, Thanks for the suggestion. This does allow my user to run rootless podman o= n multiple machines with a shared /home directory, but other user IDs with = the same configuration are blocked: $ podman info Error: could not get runtime: mkdir /tmp/container/mounts: permission denied Is this expected? Thanks, Eddie --===============2017933593876272758==-- From eae at us.ibm.com Thu Oct 24 19:25:25 2019 Content-Type: multipart/mixed; boundary="===============2495014681061745014==" MIME-Version: 1.0 From: eae at us.ibm.com To: podman at lists.podman.io Subject: [Podman] Re: Sharing blob-info-cache-v1.boltdb across multiple machines Date: Thu, 24 Oct 2019 19:25:20 +0000 Message-ID: <20191024192520.27175.88771@lists.podman.io> In-Reply-To: 20191024154940.27175.74957@lists.podman.io --===============2495014681061745014== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Ooops, silly me. Setting graphroot =3D /tmp/user/container solves that prob= lem. Thanks again --===============2495014681061745014==--