- Podman - Podman List Archives

Firewalling services provided by containers
by Alexander E. Patrakov 04 Nov '19

04 Nov '19

Hello. I have tried Podman in Fedora 31. Not a rootless setup. Software versions: podman-1.6.2-2.fc31.x86_64 containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31.x86_64 IP and netmask of the Fedora machine in my network: 192.168.5.130/24. Podman creates, on the start of the first container, its default cni-podman0 bridge with IP and netmask 10.88.0.1/16. I wanted to play through a situation when we are migrating from a service (let's say, 9999/tcp) formerly provided by some software installed directly on the host to the same service provided by the same software, but in a podman container. And this software needs to be firewalled: there is a whitelist of IP addresses (let's say 192.158.5.30 and 192.168.5.44) that have the privilege to talk to 192.168.5.130:9999. With the old, non-containerized setup, implementing this kind of whitelist is trivial. Add a new firewalld zone, add thenecessary ports and whitelisted client IPs to it, set the target to REJECT or DROP, done. However, once I switch to a containerized service, the firewall becomes ineffective, because the packets hit the FORWARD chain, not INPUT. I could not find a good solution that works in terms of the exposed port (i.e. 9999, even if inside the container a different port is used). I could either add iptables rules (yuck... firewalld exists for a reason) to "raw" or "mangle" tables (but then I cannot reject), or do something in the "filter" table with "-p tcp -m tcp -m conntrack --ctorigdstport 9999" (that's better). I think that firewald could see some improvement here. In order to apply a whitelist of hosts that can connect, I should not need to care whether the service is provided by something running on the host, or by a container. OK, another crazy idea: is it possible to use slirp4netns instead of the default bridge for root-owned containers, just to avoid these INPUT-vs-FORWARD firewall troubles? -- Alexander E. Patrakov

3 5

Trying to run podman within a locked down podman.
by Daniel Walsh 02 Nov '19

02 Nov '19

# cat ~/Dockerfile.podman FROM podman/stable RUN useradd podman # podman run -ti --security-opt seccomp=/tmp/seccomp.json --user podman --rm podman podman unshare cat /etc/subuid ERRO[0000] unable to write system event: "write unixgram @000ea->/run/systemd/journal/socket: sendmsg: no such file or directory" podman:100000:65536 # podman run -ti --security-opt seccomp=unconfined --user podman --rm podman podman unshare cat /proc/self/uid_map ERRO[0000] unable to write system event: "write unixgram @000df->/run/systemd/journal/socket: sendmsg: no such file or directory" 0 1000 1 # podman run -ti --security-opt seccomp=/tmp/seccomp.json --user podman --rm podman podman unshare cat /proc/self/uid_map ERRO[0000] unable to write system event: "write unixgram @000e6->/run/systemd/journal/socket: sendmsg: no such file or directory" 0 1000 1 Running with Debug shows DEBU[0000] error from newuidmap: newuidmap: write to uid_map failed: Operation not permitted WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids User Namespace does not seem to be working unless I add "clone" syscall, and SETUID, SETGID # podman run -ti --cap-add SETUID,SETGID --security-opt seccomp=/tmp/seccomp.json --user podman --rm podman podman unshare cat /proc/self/uid_map ERRO[0000] unable to write system event: "write unixgram @00103->/run/systemd/journal/socket: sendmsg: no such file or directory" 0 1000 1 1 100000 65536 ```Need these SELinux Rules: allow container_t nsfs_t:file read; allow container_t proc_t:filesystem mount; allow container_t tmpfs_t:filesystem { mount unmount }; ``` I am getting close with this: diff /usr/share/containers/seccomp.json /tmp/seccomp.json 367c367,370 < "unshare" --- > "unshare", > "clone", > "keyctl", > "pivot_root" # podman run -ti --privileged --cap-add SETUID,SETGID --security-opt seccomp=/tmp/seccomp.json --user podman --rm podman podman run --net=host --cgroup-manager cgroupfs alpine echo hello ERRO[0000] unable to write system event: "write unixgram @0016a->/run/systemd/journal/socket: sendmsg: no such file or directory" Trying to pull docker.io/library/alpine... Getting image source signatures Copying blob 89d9c30c1d48 done Copying config 965ea09ff2 done Writing manifest to image destination Storing signatures ERRO[0004] unable to write pod event: "write unixgram @0016a->/run/systemd/journal/socket: sendmsg: no such file or directory" Error: cannot configure rootless cgroup using the cgroupfs manager executable file not found in $PATH: No such file or directory: OCI runtime command not found error

1 0

port bindings are not yet supported by rootless containers
by Álvaro Castillo 30 Oct '19

30 Oct '19

Hello, I am interesting to run a container with port redirects. I was trying run nginx container with ports redirect likes 80:1024 80:1200 80:81... But It's does give me the same error always. port bindings are not yet supported by rootless containers My OS is CentOS 8, but I've tried with Fedora 31 beta and It's same happens. Can you help me? Thanks.

5 5

podman references
by Álvaro Castillo 29 Oct '19

29 Oct '19

Hello all, I am newbie here, I am interesting in get more information about how works podman with Kubernetes without uses Docker. Books, pdf, articles, howtos... Greetings!

3 2

rootless podman group credentials limited to users primary group?
by eae＠us.ibm.com 29 Oct '19

29 Oct '19

Scenario: rootless user with primary and secondary group membership starts a container with mounted filesystem. Expected behavior: the group credentials of podman container would respect the results of newgrp before starting container. Actual behavior: the group credentials for access are always the primary group.

2 1

[varlink] how to do exec using podman-remote?
by Francesco Romani 28 Oct '19

28 Oct '19

Hi all, I'm using podman-remote and I'm trying to exec commands in containers. It seems it fails, but I don't really understand why. I'm on fedora 30 (with last updates), I tried podman 1.6.1 (from RPMs) but also 1.6.2 compiled from scratch: $ podman version Version: 1.6.2 RemoteAPI Version: 1 Go Version: go1.12.10 OS/Arch: linux/amd64 Ultimately, I'd need to exec commands from a golang program, but for now experimenting with command line is fine, but can't make even this work :) here's what I tried: $ varlink call -m unix:/run/podman/io.podman/io.podman.CreateContainer '{"create":{"args":["fedora:30", "/bin/sleep", "10h"]}}' { "container": "c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853" } $ varlink call -m unix:/run/podman/io.podman/io.podman.StartContainer '{"name":"c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853"}' { "container": "c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853" } The container IS running now: root 7836 0.0 0.0 77876 1756 ? Ssl 09:11 0:00 /usr/bin/conmon --api-version 1 -s -c c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 -u c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/c24e28054f89c0a root 7848 0.0 0.0 2320 684 ? Ss 09:11 0:00 \_ /bin/sleep 10h So I do: $ varlink call -m unix:/run/podman/io.podman/io.podman.ExecContainer '{ "opts": { "name": "c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853", "tty": true, "privileged": true, "cmd": ["/bin/date"] } }' Call failed with error: io.podman.ErrorOccurred { "reason": "client must use upgraded connection to exec" } So I downloaded go-varlink-cmd (https://github.com/varlink/go-varlink-cmd) and patched to supporte the upgraded connection (on client side)[1], but doesn't look much better: $ ~/bin/go-varlink-cmd call -upgrade unix:/run/podman/io.podman/io.podman.ExecContainer '{ "opts": { "name": "c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853", "tty": true, "privileged": true, "cmd": ["/bin/date"] } }' recv -> 0 # return value retval -> map[string]interface {}(nil) # what got as answer {} # answer translated to JSON No luck with minimal command line either: $ ~/bin/go-varlink-cmd call -upgrade unix:/run/podman/io.podman/io.podman.ExecContainer '{ "opts": { "name": "c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853", "cmd": ["/bin/date"] } }' recv -> 0 retval -> map[string]interface {}(nil) {} Maybe, just wondering: do I need to set something when I create the container? If so the docs aren't crystal clear :\ I tried to look at logs but can't make much sense. Here's the logs for podman-remote on my system, with log level increased to debug: Oct 28 09:21:09 myhost.lan systemd[1]: Started Podman Remote API Service. Oct 28 09:21:09 myhost.lan audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=io.podman comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using varlink socket: unix:/run/podman/io.podman" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="using conmon: \"/usr/bin/conmon\"" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using graph driver overlay" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using graph root /var/lib/containers/storage" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using run root /var/run/containers/storage" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using tmp dir /var/run/libpod" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Set libpod namespace to \"\"" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="[graphdriver] trying provided driver \"overlay\"" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="cached value indicated that overlay is supported" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="cached value indicated that metacopy is being used" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="cached value indicated that native-diff is not being used" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Initializing event backend journald" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="using runtime \"/usr/bin/runc\"" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=warning msg="Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=info msg="Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Creating new exec session in container c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 with session id 6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=error msg="ExecContainer failed to HANG-UP on c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853: write unix /run/podman/io.podman->@: write: broken pipe" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=error msg="Exec Container err: write unix /run/podman/io.podman->@: write: broken pipe" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="/usr/bin/conmon messages will be logged to syslog" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -s -c c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 -u 6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c -r /usr/bin/runc -b /> Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="disabling SD notify" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853.scope" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=warning msg="Failed to add conmon to systemd sandbox cgroup: Unit libpod-conmon-c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853.scope already exists." Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Attaching to container c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 exec session 6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="connecting to socket /var/run/libpod/socket/6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c/attach" Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.B2039Z} Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: attach sock path: /var/run/libpod/socket/6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c/attach Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: addr{sun_family=AF_UNIX, sun_path=/var/run/libpod/socket/6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c/attach} Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: ctl fifo path: /var/lib/containers/storage/overlay-containers/c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853/userdata/6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c/ctl Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: terminal_ctrl_fd: 18 Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ndebug>: sending attach message to parent Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ndebug>: sent attach message to parent Oct 28 09:21:09 myhost.lan conmon[8108]: conmon c24e28054f89c0a0ac9c <ndebug>: exec with attach is waiting for start message from parent Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Received: 0" Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: Accepted connection 20 Oct 28 09:21:09 myhost.lan conmon[8108]: conmon c24e28054f89c0a0ac9c <ndebug>: exec with attach got start message from parent Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: about to accept from console_socket_fd: 14 Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: about to recvfd from connfd: 21 Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ninfo>: console = {.name = '/dev/ptmx8 09:21:09 conmon: conmon c24e28054f89c0a0ac9c <ninfo>: about to recvfd from connfd: 21 '; .fd = 14} Oct 28 09:21:09 myhost.lan systemd[2088]: run-runc-c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853-runc.xJTNZd.mount: Succeeded. Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ndebug>: couldn't find cb for pid 8121 Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ndebug>: container status and pid were found prior to callback being registered. calling manually Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <ndebug>: container PID: 8121 Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <nwarn>: Failed to open cgroups file: /proc/8121/cgroup Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <nwarn>: Failed to get memory cgroup path. Container may have exited Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Received: 8121" Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <nwarn>: stdio_input read failed Input/output error Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <nwarn>: Failed to write to socket Oct 28 09:21:09 myhost.lan conmon[8106]: conmon c24e28054f89c0a0ac9c <error>: Unable to send container stderr message to parent Bad file descriptor Oct 28 09:21:09 myhost.lan podman[8087]: 2019-10-28 09:21:09.632987596 +0100 CET m=+0.247438867 container exec c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853 (image=docker.io/library/fedora:30, name=sad_thompson) Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=debug msg="Successfully started exec session 6a1ccdea916ee245420320fa3ce02ff68ea066b8b6b7ba1534c014c15054f67c in container c24e28054f89c0a0ac9c5e3690ab0c5ef5a8b859ed51d032ae38cc618a164853" Oct 28 09:21:09 myhost.lan podman[8087]: time="2019-10-28T09:21:09+01:00" level=error msg="write unix /run/podman/io.podman->@: write: broken pipe" I'm out of ideas and can't find anything in the docs - apologies if I missed anything, please feel free to send me there Any help or comment would be appreciated. Thanks and bests! -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh

2 5

Sharing blob-info-cache-v1.boltdb across multiple machines
by eae＠us.ibm.com 24 Oct '19

24 Oct '19

We have a cluster of machines where /home is a remote gluster mount. Running podman rootless nicely solves the problem of accessing the remote filesystem with user credentials. Since remote filesystems do not currently support namespaces, podman is run with --root, --runroot, and --tmpdir set to be /tmp/$USER. All works well on the first client machine, but an image pulled successfully on one machine will fail to pull on a second. For example, on the second machine: $ podman run --rm -it ubuntu Trying to pull docker.io/library/ubuntu...Getting image source signatures Copying blob c58094023a2e done Copying blob 079b6d2a1e53 done Copying blob 11048ebae908 done Copying blob 22e816666fd6 done Copying config cf0f3ca922 done Writing manifest to image destination Storing signatures ERRO[0168] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted ERRO[0200] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted Failed Trying to pull registry.fedoraproject.org/ubuntu...ERRO[0200] Error pulling image ref //registry.fedoraproject.org/ubuntu:latest: Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown Failed Trying to pull quay.io/ubuntu...ERRO[0201] Error pulling image ref //quay.io/ubuntu:latest: Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" Failed Trying to pull registry.centos.org/ubuntu...ERRO[0201] Error pulling image ref //registry.centos.org/ubuntu:latest: Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Failed Error: unable to pull ubuntu: 4 errors occurred: * Error committing the finished image: error adding layer with blob "sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91": ApplyLayer exit status 1 stdout: stderr: lchown /etc/gshadow: operation not permitted * Error initializing source docker://registry.fedoraproject.org/ubuntu:latest: Error reading manifest latest in registry.fedoraproject.org/ubuntu: manifest unknown: manifest unknown * Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n" * Error initializing source docker://registry.centos.org/ubuntu:latest: Error reading manifest latest in registry.centos.org/ubuntu: manifest unknown: manifest unknown Our guess is that this is happening because blob-info-cache-v1.boltdb is in the shared /home filesystem. Is there a suggested approach to running rootless podman on multiple machines with a shared /home directory? Thanks, Eddie

3 4

podman pull container-registry.oracle.com auth
by Gilbert Standen 21 Oct '19

21 Oct '19

Hello Podsters, Trying to pull from container-registry.oracle.com using podman instead of docker but auth does not seem to work right for container-registry.oracle.com from podman, while it does work as advertised from docker pull command. Have a look at this and LMK if this is a bug, will be fixed, etc. Note that I previously followed the step of logging in to container-registry.oracle.com at the URL as described here https://docs.oracle.com/cd/E37670_01/E75728/html/oracle-registry-server.html . Then I try to pull images using podman (does not authenticate properly - apparently - see below): --- ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ podman pull container-registry.oracle.com/os/oraclelinux:7.6 Trying to pull container-registry.oracle.com/os/oraclelinux:7.6... unable to retrieve auth token: invalid username/password Error: error pulling image "container-registry.oracle.com/os/oraclelinux:7.6": unable to pull container-registry.oracle.com/os/oraclelinux:7.6: unable to pull image: Error initializing source docker://container-registry.oracle.com/os/oraclelinux:7.6: unable to retrieve auth token: invalid username/password --- now I've previously logged in to URL as instructed and tied an 8-hour auth to my oracle single sign-on --- ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ podman login container-registry.oracle.com Username: gilbert(a)orabuntu-lxc.com Password: Error: error logging into "container-registry.oracle.com": invalid username/password ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ podman login container-registry.oracle.com Username: gilbert(a)orabuntu-lxc.com Password: Error: error logging into "container-registry.oracle.com": invalid username/password --- I try a couple times using the password and also the password enclosed in " ". Neither authenticates successfully using podman --- --- Now I try same login using docker and authentication succeeds (note my pw has upper/lower case, numerals, and special character) --- ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ sudo docker login container-registry.oracle.com Username: gilbert(a)orabuntu-lxc.com Password: Login Succeeded --- Trying a podman pull again after the successful docker login - no go does not auth --- ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ podman pull container-registry.oracle.com/os/oraclelinux:7.6 Trying to pull container-registry.oracle.com/os/oraclelinux:7.6... open /home/ubuntu/.docker/config.json: permission denied Error: error pulling image "container-registry.oracle.com/os/oraclelinux:7.6": unable to pull container-registry.oracle.com/os/oraclelinux:7.6: unable to pull image: Error initializing source docker://container-registry.oracle.com/os/oraclelinux:7.6: error getting username and password: error reading JSON file "/home/ubuntu/.docker/config.json": open /home/ubuntu/.docker/config.json: permission denied --- Try a pull using docker auth that has been set and it works fine --- ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ sudo docker pull container-registry.oracle.com/os/oraclelinux:7.6 7.6: Pulling from os/oraclelinux 04e172e76262: Pull complete Digest: sha256:a6ee3b4c6a52a529a6de77bb25c3d0a99e99e1ad81cd936cd14b100288ffe246 Status: Downloaded newer image for container-registry.oracle.com/os/oraclelinux:7.6 ubuntu@ubuntu-ThinkPad-P72:~/Desktop$ So is this an issue with podman that it is not able to handle the authentication to Oracle container registry, whereas docker can and does ? If yes can we file this as an enhancement request or bug fix ? Thanks Gilbert

2 2

rootless centos + gpu problem? plz advise.
by Lou DeGenaro 18 Oct '19

18 Oct '19

Script started on Mon 14 Oct 2019 10:51:34 AM CDT ]0;eae@ducc-saicluster-gpu-801 ~]$ podman info host: BuildahVersion: 1.9.0 Conmon: package: podman-1.4.4-4.el7.centos.x86_64 path: /usr/libexec/podman/conmon version: 'conmon version 0.3.0, commit: unknown' Distribution: distribution: '"centos"' version: "7" MemFree: 52633743360 MemTotal: 63154479104 OCIRuntime: package: runc-1.0.0-65.rc8.el7.centos.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 2146758656 SwapTotal: 2146758656 arch: amd64 cpus: 8 hostname: ducc-saicluster-gpu-801.sl.cloud9.ibm.com kernel: 3.10.0-1062.1.2.el7.x86_64 os: linux rootless: true uptime: 145h 40m 2.43s (Approximately 6.04 days) registries: blocked: null insecure: null search: - registry.access.redhat.com - docker.io - registry.fedoraproject.org - quay.io - registry.centos.org store: ConfigFile: /home/eae/.config/containers/storage.conf ContainerStore: number: 0 GraphDriverName: vfs GraphOptions: null GraphRoot: /tmp/eae/containers/storage GraphStatus: {} ImageStore: number: 2 RunRoot: /run/user/13642 VolumePath: /tmp/eae/containers/storage/volumes ]0;eae@ducc-saicluster-gpu-801:~ [eae@ducc-saicluster-gpu-801 ~]$ cat .config/containers/libpod.conf volume_path = "/tmp/eae/containers/storage/volumes" image_default_transport = "docker://" runtime = "runc" conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon"] conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"] cgroup_manager = "cgroupfs" init_path = "/usr/libexec/podman/catatonit" static_dir = "/tmp/eae/containers/storage/libpod" tmp_dir = "/run/user/13642/libpod/tmp" max_log_size = -1 no_pivot_root = false cni_config_dir = "/etc/cni/net.d/" cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"] infra_image = "k8s.gcr.io/pause:3.1" infra_command = "/pause" enable_port_reservation = true label = true network_cmd_path = "" num_locks = 2048 events_logger = "journald" EventsLogFilePath = "" detach_keys = "ctrl-p,ctrl-q" hooks_dir = ["/usr/share/containers/oci/hooks.d"] [runtimes] runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"] ]0;eae@ducc-saicluster-gpu-801:~ [eae@ducc-saicluster-gpu-801 ~]$ cat /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json { "version": "1.0.0", "hook": { "path": "/usr/bin/nvidia-container-toolkit", "args": ["nvidia-container-toolkit", "prestart"], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ] }, "when": { "always": true, "commands": [".*"] }, "stages": ["prestart"] } [eae@ducc-saicluster-gpu-801 ~]$ podman --log-level=debug run --rm nvidia/cuda nvidia-smi [36mINFO [0m[0000] running as rootless [37mDEBU [0m[0000] Initializing boltdb state at /tmp/eae/containers/storage/libpod/bolt_state.db [37mDEBU [0m[0000] Using graph driver vfs [37mDEBU [0m[0000] Using graph root /tmp/eae/containers/storage [37mDEBU [0m[0000] Using run root /run/user/13642 [37mDEBU [0m[0000] Using static dir /tmp/eae/containers/storage/libpod [37mDEBU [0m[0000] Using tmp dir /run/user/13642/libpod/tmp [37mDEBU [0m[0000] Using volume path /tmp/eae/containers/storage/volumes [37mDEBU [0m[0000] Set libpod namespace to "" [37mDEBU [0m[0000] [graphdriver] trying provided driver "vfs" [37mDEBU [0m[0000] Initializing event backend journald [37mDEBU [0m[0000] parsed reference into "[vfs@ /tmp/eae/containers/storage+/run/user/13642]docker.io/nvidia/cuda:latest" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] Got mounts: [] [37mDEBU [0m[0000] Got volumes: [] [37mDEBU [0m[0000] Using slirp4netns netmode [37mDEBU [0m[0000] created OCI spec and options for new container [37mDEBU [0m[0000] Allocated lock 0 for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0009] created container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has work directory "/tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has run directory "/run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata" [37mDEBU [0m[0009] New container created "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has CgroupParent "/libpod_parent/libpod-75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] Not attaching to stdin [37mDEBU [0m[0009] mounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" at "/tmp/eae/containers/storage/vfs/dir/dfce96c6e34e6c12aad6da967a42c56db04a30664bf8c6a081ee5efb1dcb7b19" [37mDEBU [0m[0009] Created root filesystem for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 at /tmp/eae/containers/storage/vfs/dir/dfce96c6e34e6c12aad6da967a42c56db04a30664bf8c6a081ee5efb1dcb7b19 [37mDEBU [0m[0009] /etc/system-fips does not exist on host, not mounting FIPS mode secret [37mDEBU [0m[0009] reading hooks from /usr/share/containers/oci/hooks.d [37mDEBU [0m[0009] added hook /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json [37mDEBU [0m[0009] hook oci-nvidia-hook.json matched; adding to stages [prestart] [37mDEBU [0m[0009] Created OCI spec for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 at /tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/config.json [37mDEBU [0m[0009] /usr/libexec/podman/conmon messages will be logged to syslog [37mDEBU [0m[0009] running conmon: /usr/libexec/podman/conmon [37margs [0m="[-c 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 -u 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 -n flamboyant_pare -r /usr/bin/runc -b /tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata -p /run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/pidfile --exit-dir /run/user/13642/libpod/tmp/exits --conmon-pidfile /run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /tmp/eae/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/13642 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/13642/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 --socket-dir-path /run/user/13642/libpod/tmp/socket -l k8s-file:/tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/ctr.log --log-level debug --syslog]" [33mWARN [0m[0009] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/libpod_parent: permission denied [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied [37mDEBU [0m[0010] Received container pid: -1 [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] unmounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [31mERRO [0m[0010] container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/user.slice/devices.allow: permission denied\\\\n\\\"\"" : OCI runtime error Script done on Mon 14 Oct 2019 10:53:23 AM CDT

6 10

Re: rootless centos + gpu problem? plz advise.
by Edward Epstein 18 Oct '19

18 Oct '19

We discovered that the ubuntu 18.04 machine had a configuration change to get rootless working with nvidia: "no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml Unfortunately this config change did not work on Centos 7, but it did change the rootless error to: nvidia-container-cli: initialization error: cuda error: unknown error\\\\n \\\"\"" This config change breaks podman running from root, with the error: Failed to initialize NVML: Unknown Error Interestingly, root on ubuntu gets the same error even though rootless works.

2 1