rootless centos + gpu problem? plz advise.
Script started on Mon 14 Oct 2019 10:51:34 AM CDT ]0;eae@ducc-saicluster-gpu-801 ~]$ podman info host: BuildahVersion: 1.9.0 Conmon: package: podman-1.4.4-4.el7.centos.x86_64 path: /usr/libexec/podman/conmon version: 'conmon version 0.3.0, commit: unknown' Distribution: distribution: '"centos"' version: "7" MemFree: 52633743360 MemTotal: 63154479104 OCIRuntime: package: runc-1.0.0-65.rc8.el7.centos.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 2146758656 SwapTotal: 2146758656 arch: amd64 cpus: 8 hostname: ducc-saicluster-gpu-801.sl.cloud9.ibm.com kernel: 3.10.0-1062.1.2.el7.x86_64 os: linux rootless: true uptime: 145h 40m 2.43s (Approximately 6.04 days) registries: blocked: null insecure: null search: - registry.access.redhat.com - docker.io - registry.fedoraproject.org - quay.io - registry.centos.org store: ConfigFile: /home/eae/.config/containers/storage.conf ContainerStore: number: 0 GraphDriverName: vfs GraphOptions: null GraphRoot: /tmp/eae/containers/storage GraphStatus: {} ImageStore: number: 2 RunRoot: /run/user/13642 VolumePath: /tmp/eae/containers/storage/volumes ]0;eae@ducc-saicluster-gpu-801:~ [eae@ducc-saicluster-gpu-801 ~]$ cat .config/containers/libpod.conf volume_path = "/tmp/eae/containers/storage/volumes" image_default_transport = "docker://" runtime = "runc" conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon"] conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"] cgroup_manager = "cgroupfs" init_path = "/usr/libexec/podman/catatonit" static_dir = "/tmp/eae/containers/storage/libpod" tmp_dir = "/run/user/13642/libpod/tmp" max_log_size = -1 no_pivot_root = false cni_config_dir = "/etc/cni/net.d/" cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"] infra_image = "k8s.gcr.io/pause:3.1" infra_command = "/pause" enable_port_reservation = true label = true network_cmd_path = "" num_locks = 2048 events_logger = "journald" EventsLogFilePath = "" detach_keys = "ctrl-p,ctrl-q" hooks_dir = ["/usr/share/containers/oci/hooks.d"] [runtimes] runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"] ]0;eae@ducc-saicluster-gpu-801:~ [eae@ducc-saicluster-gpu-801 ~]$ cat /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json { "version": "1.0.0", "hook": { "path": "/usr/bin/nvidia-container-toolkit", "args": ["nvidia-container-toolkit", "prestart"], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ] }, "when": { "always": true, "commands": [".*"] }, "stages": ["prestart"] } [eae@ducc-saicluster-gpu-801 ~]$ podman --log-level=debug run --rm nvidia/cuda nvidia-smi [36mINFO [0m[0000] running as rootless [37mDEBU [0m[0000] Initializing boltdb state at /tmp/eae/containers/storage/libpod/bolt_state.db [37mDEBU [0m[0000] Using graph driver vfs [37mDEBU [0m[0000] Using graph root /tmp/eae/containers/storage [37mDEBU [0m[0000] Using run root /run/user/13642 [37mDEBU [0m[0000] Using static dir /tmp/eae/containers/storage/libpod [37mDEBU [0m[0000] Using tmp dir /run/user/13642/libpod/tmp [37mDEBU [0m[0000] Using volume path /tmp/eae/containers/storage/volumes [37mDEBU [0m[0000] Set libpod namespace to "" [37mDEBU [0m[0000] [graphdriver] trying provided driver "vfs" [37mDEBU [0m[0000] Initializing event backend journald [37mDEBU [0m[0000] parsed reference into "[vfs@ /tmp/eae/containers/storage+/run/user/13642]docker.io/nvidia/cuda:latest" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] Got mounts: [] [37mDEBU [0m[0000] Got volumes: [] [37mDEBU [0m[0000] Using slirp4netns netmode [37mDEBU [0m[0000] created OCI spec and options for new container [37mDEBU [0m[0000] Allocated lock 0 for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/13642]@946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:946e78c7b2984354477ae4b75bf519940f4df648c092564d1d9c83ea8c92c8f3" [37mDEBU [0m[0009] created container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has work directory "/tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has run directory "/run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata" [37mDEBU [0m[0009] New container created "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" has CgroupParent "/libpod_parent/libpod-75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0009] Not attaching to stdin [37mDEBU [0m[0009] mounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" at "/tmp/eae/containers/storage/vfs/dir/dfce96c6e34e6c12aad6da967a42c56db04a30664bf8c6a081ee5efb1dcb7b19" [37mDEBU [0m[0009] Created root filesystem for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 at /tmp/eae/containers/storage/vfs/dir/dfce96c6e34e6c12aad6da967a42c56db04a30664bf8c6a081ee5efb1dcb7b19 [37mDEBU [0m[0009] /etc/system-fips does not exist on host, not mounting FIPS mode secret [37mDEBU [0m[0009] reading hooks from /usr/share/containers/oci/hooks.d [37mDEBU [0m[0009] added hook /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json [37mDEBU [0m[0009] hook oci-nvidia-hook.json matched; adding to stages [prestart] [37mDEBU [0m[0009] Created OCI spec for container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 at /tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/config.json [37mDEBU [0m[0009] /usr/libexec/podman/conmon messages will be logged to syslog [37mDEBU [0m[0009] running conmon: /usr/libexec/podman/conmon [37margs [0m="[-c 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 -u 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 -n flamboyant_pare -r /usr/bin/runc -b /tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata -p /run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/pidfile --exit-dir /run/user/13642/libpod/tmp/exits --conmon-pidfile /run/user/13642/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /tmp/eae/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/13642 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/13642/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 --socket-dir-path /run/user/13642/libpod/tmp/socket -l k8s-file:/tmp/eae/containers/storage/vfs-containers/75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8/userdata/ctr.log --log-level debug --syslog]" [33mWARN [0m[0009] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/libpod_parent: permission denied [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied [37mDEBU [0m[0010] Received container pid: -1 [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] unmounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [31mERRO [0m[0010] container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/user.slice/devices.allow: permission denied\\\\n\\\"\"" : OCI runtime error Script done on Mon 14 Oct 2019 10:53:23 AM CDT
Lou DeGenaro <lou.degenaro@gmail.com> writes:
[37mDEBU [0m[0010] Received container pid: -1 [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] unmounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [31mERRO [0m[0010] container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/user.slice/devices.allow: permission denied\\\\n\\\"\""
looks like the NVIDIA OCI hook tries to handle the devices cgroup for the container. As rootless user you have not enough privileges for doing that. Giuseppe
Works rootless on Ubuntu 18. Does not work rootless on CentOS 7...how come? Lou. On Tue, Oct 15, 2019 at 9:15 AM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
Lou DeGenaro <lou.degenaro@gmail.com> writes:
[37mDEBU [0m[0010] Received container pid: -1 [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] unmounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [31mERRO [0m[0010] container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/user.slice/devices.allow: permission denied\\\\n\\\"\""
looks like the NVIDIA OCI hook tries to handle the devices cgroup for the container. As rootless user you have not enough privileges for doing that.
Giuseppe
Giuseppe, Is that something that will potentially be fixed with cgroups v2? My gut says it would be: 1. Get the world to cgroup v2 2. Nvidia might have to redesign some things? Until then, it's not possible right? On Tue, Oct 15, 2019 at 9:16 AM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
Lou DeGenaro <lou.degenaro@gmail.com> writes:
[37mDEBU [0m[0010] Received container pid: -1 [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] unmounted container "75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8" [37mDEBU [0m[0010] Cleaning up container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 [37mDEBU [0m[0010] Network is already cleaned up, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [37mDEBU [0m[0010] Container 75bb8e197bea3d0c56f5060ab5e1388a1bdcab354e9820bd5554d3bf273a54d8 storage is already unmounted, skipping... [31mERRO [0m[0010] container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: open failed: /sys/fs/cgroup/devices/user.slice/devices.allow: permission denied\\\\n\\\"\""
looks like the NVIDIA OCI hook tries to handle the devices cgroup for the container. As rootless user you have not enough privileges for doing that.
Giuseppe _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
-- -- Scott McCarty, RHCA Product Management - Containers, Red Hat Enterprise Linux & OpenShift Email: smccarty@redhat.com Phone: 312-660-3535 Cell: 330-807-1043 Web: http://crunchtools.com Easily create customer SELinux policies with Podman, thanks to Udica: https://red.ht/35cdy2d
Hi Scott, Scott McCarty <smccarty@redhat.com> writes:
Giuseppe, Is that something that will potentially be fixed with cgroups v2? My gut says it would be:
1. Get the world to cgroup v2 2. Nvidia might have to redesign some things?
Until then, it's not possible right?
no, cgroups v2 won't solve access to the devices cgroups for rootless. Configuring the devices cgroups on cgroup v2 requires using eBPF that is a privileged operation. Giuseppe
Hi All I'll have to follow this thread as gpu's are very much on my radar too. Bryan On Tue, Oct 15, 2019 at 2:34 PM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
Hi Scott,
Scott McCarty <smccarty@redhat.com> writes:
Giuseppe, Is that something that will potentially be fixed with cgroups v2? My gut says it would be:
1. Get the world to cgroup v2 2. Nvidia might have to redesign some things?
Until then, it's not possible right?
no, cgroups v2 won't solve access to the devices cgroups for rootless. Configuring the devices cgroups on cgroup v2 requires using eBPF that is a privileged operation.
Giuseppe _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Here is console for successful run rootless on Ubuntu 18. So why not working on CentOS 7.x? ===== Script started on 2019-10-14 11:55:21-0400 ]0;eae@f12n13: /home/eae eae@f12n13:~$ podman info host: BuildahVersion: 1.9.0 Conmon: package: 'conmon: /usr/libexec/podman/conmon' path: /usr/libexec/podman/conmon version: 'conmon version 1.0.0-rc2, commit: unknown' Distribution: distribution: ubuntu version: "18.04" MemFree: 284860088320 MemTotal: 548435001344 OCIRuntime: package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc' path: /usr/lib/cri-o-runc/sbin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 23127326720 SwapTotal: 23152492544 arch: ppc64le cpus: 192 hostname: f12n13 kernel: 4.15.0-55-generic os: linux rootless: true uptime: 979h 10m 16.18s (Approximately 40.79 days) registries: blocked: null insecure: null search: - docker.io store: ConfigFile: /home/eae/.config/containers/storage.conf ContainerStore: number: 0 GraphDriverName: vfs GraphOptions: null GraphRoot: /tmp/eae/containers/storage GraphStatus: {} ImageStore: number: 3 RunRoot: /run/user/2018 VolumePath: /tmp/eae/containers/storage/volumes ]0;eae@f12n13: /home/eae eae@f12n13:~$ cat ./.config/containers/libpod.conf volume_path = "/tmp/eae/containers/storage/volumes" image_default_transport = "docker://" runtime = "runc" conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon"] conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"] cgroup_manager = "cgroupfs" init_path = "/usr/libexec/podman/catatonit" static_dir = "/tmp/eae/containers/storage/libpod" tmp_dir = "/run/user/2018/libpod/tmp" max_log_size = -1 no_pivot_root = false cni_config_dir = "/etc/cni/net.d/" cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"] infra_image = "k8s.gcr.io/pause:3.1" infra_command = "/pause" enable_port_reservation = true label = true network_cmd_path = "" num_locks = 2048 events_logger = "journald" EventsLogFilePath = "" detach_keys = "ctrl-p,ctrl-q" hooks_dir = ["/etc/containers/oci/hooks.d"] [runtimes] runc = ["/usr/lib/cri-o-runc/sbin/runc"] ]0;eae@f12n13: /home/eae eae@f12n13:~$ cat /etc/containers/oci/hooks.d/oci-nvidia-hook.json { "version": "1.0.0", "hook": { "path": "/usr/bin/nvidia-container-runtime-hook", "args": ["nvidia-container-runtime-hook", "prestart"] }, "when": { "always": true }, "stages": ["prestart"] } ]0;eae@f12n13: /home/eae eae@f12n13:~$ podman --log-level=debug run --rm ibmcom/tensorflow-ppc64le:1.14.0-gpu nvidia-smi [36mINFO [0m[0000] running as rootless [37mDEBU [0m[0000] Initializing boltdb state at /tmp/eae/containers/storage/libpod/bolt_state.db [37mDEBU [0m[0000] Using graph driver vfs [37mDEBU [0m[0000] Using graph root /tmp/eae/containers/storage [37mDEBU [0m[0000] Using run root /run/user/2018 [37mDEBU [0m[0000] Using static dir /tmp/eae/containers/storage/libpod [37mDEBU [0m[0000] Using tmp dir /run/user/2018/libpod/tmp [37mDEBU [0m[0000] Using volume path /tmp/eae/containers/storage/volumes [37mDEBU [0m[0000] Set libpod namespace to "" [37mDEBU [0m[0000] [graphdriver] trying provided driver "vfs" [37mDEBU [0m[0000] Initializing event backend journald [37mDEBU [0m[0000] parsed reference into "[vfs@ /tmp/eae/containers/storage+/run/user/2018] docker.io/ibmcom/tensorflow-ppc64le:1.14.0-gpu" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] Got mounts: [] [37mDEBU [0m[0000] Got volumes: [] [37mDEBU [0m[0000] Using slirp4netns netmode [37mDEBU [0m[0000] created OCI spec and options for new container [37mDEBU [0m[0000] Allocated lock 0 for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0002] created container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" [37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has work directory "/tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata" [37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has run directory "/run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata" [37mDEBU [0m[0002] New container created "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" [37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has CgroupParent "/libpod_parent/libpod-427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" [37mDEBU [0m[0002] Not attaching to stdin [37mDEBU [0m[0002] mounted container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" at "/tmp/eae/containers/storage/vfs/dir/6493d899b4a3aa9facb67ec96a1ba381c135c5003b384ac6da1af433d816714f" [37mDEBU [0m[0002] Created root filesystem for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 at /tmp/eae/containers/storage/vfs/dir/6493d899b4a3aa9facb67ec96a1ba381c135c5003b384ac6da1af433d816714f [37mDEBU [0m[0002] /etc/system-fips does not exist on host, not mounting FIPS mode secret [37mDEBU [0m[0002] reading hooks from /etc/containers/oci/hooks.d [37mDEBU [0m[0002] added hook /etc/containers/oci/hooks.d/oci-nvidia-hook.json [37mDEBU [0m[0002] hook oci-nvidia-hook.json matched; adding to stages [prestart] [37mDEBU [0m[0002] Created OCI spec for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 at /tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/config.json [37mDEBU [0m[0002] /usr/libexec/podman/conmon messages will be logged to syslog [37mDEBU [0m[0002] running conmon: /usr/libexec/podman/conmon [37margs [0m="[-c 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 -u 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 -n dazzling_cartwright -r /usr/lib/cri-o-runc/sbin/runc -b /tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata -p /run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/pidfile --exit-dir /run/user/2018/libpod/tmp/exits --conmon-pidfile /run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /tmp/eae/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/2018 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/2018/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 --socket-dir-path /run/user/2018/libpod/tmp/socket -l k8s-file:/tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/ctr.log --log-level debug --syslog]" [33mWARN [0m[0002] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpuset: mkdir /sys/fs/cgroup/cpuset/libpod_parent: permission denied [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied [37mDEBU [0m[0003] Received container pid: 181118 [37mDEBU [0m[0003] Created container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 in OCI runtime [37mDEBU [0m[0003] Attaching to container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 [37mDEBU [0m[0003] connecting to socket /run/user/2018/libpod/tmp/socket/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/attach [37mDEBU [0m[0003] Starting container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 with command [nvidia-smi] [37mDEBU [0m[0003] Started container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 Mon Oct 14 17:18:44 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 On | 00000002:03:00.0 Off | 0 | | N/A 32C P8 26W / 149W | 0MiB / 11441MiB | 2% Default | +-------------------------------+----------------------+----------------------+ [37mDEBU [0m[0003] Enabling signal proxying | 1 Tesla K80 On | 00000002:04:00.0 Off | 0 | | N/A 29C P8 31W / 149W | 0MiB / 11441MiB | 2% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ [37mDEBU [0m[0004] Checking container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 status... [37mDEBU [0m[0004] Attempting to read container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 exit code from file /run/user/2018/libpod/tmp/exits/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6-old [37mDEBU [0m[0004] [graphdriver] trying provided driver "vfs" ]0;eae@f12n13: /home/eae eae@f12n13:~$ exit Script done on 2019-10-14 13:18:47-0400 On Tue, Oct 15, 2019 at 9:54 AM Bryan Hepworth <bryan.hepworth@gmail.com> wrote:
Hi All
I'll have to follow this thread as gpu's are very much on my radar too.
Bryan
On Tue, Oct 15, 2019 at 2:34 PM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
Hi Scott,
Scott McCarty <smccarty@redhat.com> writes:
Giuseppe, Is that something that will potentially be fixed with cgroups v2? My gut says it would be:
1. Get the world to cgroup v2 2. Nvidia might have to redesign some things?
Until then, it's not possible right?
no, cgroups v2 won't solve access to the devices cgroups for rootless. Configuring the devices cgroups on cgroup v2 requires using eBPF that is a privileged operation.
Giuseppe _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
What exactly does the OCI Hook do? If all it is doing is bind mounting the device from the host into the container, it might work, and this could be something that is not supported in Centos7? We currently do not support RHEL7/Centos7 for rootless containers running on fuse-overlayfs. Should happen in rhel7.8 release. Not sure if this is happening because of this. On 10/15/19 10:02 AM, Lou DeGenaro wrote:
Here is console for successful run rootless on Ubuntu 18. So why not working on CentOS 7.x?
=====
Script started on 2019-10-14 11:55:21-0400
]0;eae@f12n13: /home/eae eae@f12n13:~$ podman info host: BuildahVersion: 1.9.0 Conmon: package: 'conmon: /usr/libexec/podman/conmon' path: /usr/libexec/podman/conmon version: 'conmon version 1.0.0-rc2, commit: unknown' Distribution: distribution: ubuntu version: "18.04" MemFree: 284860088320 MemTotal: 548435001344 OCIRuntime: package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc' path: /usr/lib/cri-o-runc/sbin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 23127326720 SwapTotal: 23152492544 arch: ppc64le cpus: 192 hostname: f12n13 kernel: 4.15.0-55-generic os: linux rootless: true uptime: 979h 10m 16.18s (Approximately 40.79 days) registries: blocked: null insecure: null search: - docker.io <http://docker.io> store: ConfigFile: /home/eae/.config/containers/storage.conf ContainerStore: number: 0 GraphDriverName: vfs GraphOptions: null GraphRoot: /tmp/eae/containers/storage GraphStatus: {} ImageStore: number: 3 RunRoot: /run/user/2018 VolumePath: /tmp/eae/containers/storage/volumes
]0;eae@f12n13: /home/eae eae@f12n13:~$ cat ./.config/containers/libpod.conf volume_path = "/tmp/eae/containers/storage/volumes" image_default_transport = "docker://" runtime = "runc" conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon"] conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"] cgroup_manager = "cgroupfs" init_path = "/usr/libexec/podman/catatonit" static_dir = "/tmp/eae/containers/storage/libpod" tmp_dir = "/run/user/2018/libpod/tmp" max_log_size = -1 no_pivot_root = false cni_config_dir = "/etc/cni/net.d/" cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"] infra_image = "k8s.gcr.io/pause:3.1 <http://k8s.gcr.io/pause:3.1>" infra_command = "/pause" enable_port_reservation = true label = true network_cmd_path = "" num_locks = 2048 events_logger = "journald" EventsLogFilePath = "" detach_keys = "ctrl-p,ctrl-q" hooks_dir = ["/etc/containers/oci/hooks.d"] [runtimes] runc = ["/usr/lib/cri-o-runc/sbin/runc"]
]0;eae@f12n13: /home/eae eae@f12n13:~$ cat /etc/containers/oci/hooks.d/oci-nvidia-hook.json { "version": "1.0.0", "hook": { "path": "/usr/bin/nvidia-container-runtime-hook", "args": ["nvidia-container-runtime-hook", "prestart"] }, "when": { "always": true }, "stages": ["prestart"] }
]0;eae@f12n13: /home/eae eae@f12n13:~$ podman --log-level=debug run --rm ibmcom/tensorflow-ppc64le:1.14.0-gpu nvidia-smi [36mINFO [0m[0000] running as rootless [37mDEBU [0m[0000] Initializing boltdb state at /tmp/eae/containers/storage/libpod/bolt_state.db [37mDEBU [0m[0000] Using graph driver vfs [37mDEBU [0m[0000] Using graph root /tmp/eae/containers/storage [37mDEBU [0m[0000] Using run root /run/user/2018 [37mDEBU [0m[0000] Using static dir /tmp/eae/containers/storage/libpod [37mDEBU [0m[0000] Using tmp dir /run/user/2018/libpod/tmp [37mDEBU [0m[0000] Using volume path /tmp/eae/containers/storage/volumes [37mDEBU [0m[0000] Set libpod namespace to "" [37mDEBU [0m[0000] [graphdriver] trying provided driver "vfs" [37mDEBU [0m[0000] Initializing event backend journald [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]docker.io/ibmcom/tensorflow-ppc64le:1.14.0-gpu <http://docker.io/ibmcom/tensorflow-ppc64le:1.14.0-gpu>" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727"
[37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727"
[37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727"
[37mDEBU [0m[0000] Got mounts: [] [37mDEBU [0m[0000] Got volumes: [] [37mDEBU [0m[0000] Using slirp4netns netmode [37mDEBU [0m[0000] created OCI spec and options for new container [37mDEBU [0m[0000] Allocated lock 0 for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 [37mDEBU [0m[0000] parsed reference into "[vfs@/tmp/eae/containers/storage+/run/user/2018]@09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727"
[37mDEBU [0m[0000] exporting opaque data as blob "sha256:09f6383a1fbb35ca5fbccf6f0bd21a5c3d5264b6c0b56597fd47a97c99421727" [37mDEBU [0m[0002] created container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" [37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has work directory "/tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata"
[37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has run directory "/run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata"
[37mDEBU [0m[0002] New container created "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" [37mDEBU [0m[0002] container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" has CgroupParent "/libpod_parent/libpod-427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6"
[37mDEBU [0m[0002] Not attaching to stdin [37mDEBU [0m[0002] mounted container "427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6" at "/tmp/eae/containers/storage/vfs/dir/6493d899b4a3aa9facb67ec96a1ba381c135c5003b384ac6da1af433d816714f"
[37mDEBU [0m[0002] Created root filesystem for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 at /tmp/eae/containers/storage/vfs/dir/6493d899b4a3aa9facb67ec96a1ba381c135c5003b384ac6da1af433d816714f
[37mDEBU [0m[0002] /etc/system-fips does not exist on host, not mounting FIPS mode secret [37mDEBU [0m[0002] reading hooks from /etc/containers/oci/hooks.d [37mDEBU [0m[0002] added hook /etc/containers/oci/hooks.d/oci-nvidia-hook.json [37mDEBU [0m[0002] hook oci-nvidia-hook.json matched; adding to stages [prestart] [37mDEBU [0m[0002] Created OCI spec for container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 at /tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/config.json
[37mDEBU [0m[0002] /usr/libexec/podman/conmon messages will be logged to syslog [37mDEBU [0m[0002] running conmon: /usr/libexec/podman/conmon [37margs [0m="[-c 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 -u 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 -n dazzling_cartwright -r /usr/lib/cri-o-runc/sbin/runc -b /tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata -p /run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/pidfile --exit-dir /run/user/2018/libpod/tmp/exits --conmon-pidfile /run/user/2018/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /tmp/eae/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/2018 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/2018/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 --socket-dir-path /run/user/2018/libpod/tmp/socket -l k8s-file:/tmp/eae/containers/storage/vfs-containers/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/userdata/ctr.log --log-level debug --syslog]" [33mWARN [0m[0002] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpuset: mkdir /sys/fs/cgroup/cpuset/libpod_parent: permission denied [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
[37mDEBU [0m[0003] Received container pid: 181118 [37mDEBU [0m[0003] Created container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 in OCI runtime [37mDEBU [0m[0003] Attaching to container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 [37mDEBU [0m[0003] connecting to socket /run/user/2018/libpod/tmp/socket/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6/attach
[37mDEBU [0m[0003] Starting container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 with command [nvidia-smi] [37mDEBU [0m[0003] Started container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 Mon Oct 14 17:18:44 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 On | 00000002:03:00.0 Off | 0 | | N/A 32C P8 26W / 149W | 0MiB / 11441MiB | 2% Default | +-------------------------------+----------------------+----------------------+ [37mDEBU [0m[0003] Enabling signal proxying | 1 Tesla K80 On | 00000002:04:00.0 Off | 0 | | N/A 29C P8 31W / 149W | 0MiB / 11441MiB | 2% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ [37mDEBU [0m[0004] Checking container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 status... [37mDEBU [0m[0004] Attempting to read container 427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6 exit code from file /run/user/2018/libpod/tmp/exits/427b4716bca98d8f83fbe6b18095840f75faf8217d5b3e685505ec19c92c21d6-old
[37mDEBU [0m[0004] [graphdriver] trying provided driver "vfs" ]0;eae@f12n13: /home/eae eae@f12n13:~$ exit
Script done on 2019-10-14 13:18:47-0400
On Tue, Oct 15, 2019 at 9:54 AM Bryan Hepworth <bryan.hepworth@gmail.com <mailto:bryan.hepworth@gmail.com>> wrote:
Hi All
I'll have to follow this thread as gpu's are very much on my radar too.
Bryan
On Tue, Oct 15, 2019 at 2:34 PM Giuseppe Scrivano <gscrivan@redhat.com <mailto:gscrivan@redhat.com>> wrote:
Hi Scott,
Scott McCarty <smccarty@redhat.com <mailto:smccarty@redhat.com>> writes:
> Giuseppe, > Is that something that will potentially be fixed with cgroups v2? My gut says it would be: > > 1. Get the world to cgroup v2 > 2. Nvidia might have to redesign some things? > > Until then, it's not possible right?
no, cgroups v2 won't solve access to the devices cgroups for rootless. Configuring the devices cgroups on cgroup v2 requires using eBPF that is a privileged operation.
Giuseppe _______________________________________________ Podman mailing list -- podman@lists.podman.io <mailto:podman@lists.podman.io> To unsubscribe send an email to podman-leave@lists.podman.io <mailto:podman-leave@lists.podman.io>
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless working with nvidia: "no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml Unfortunately this config change did not work on Centos 7, but it did change the rootless error to: nvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\"" This config change breaks podman running from root, with the error: Failed to initialize NVML: Unknown Error Interestingly, root on ubuntu gets the same error even though rootless works.
On 10/17/19 4:33 PM, eae@us.ibm.com wrote:
We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless working with nvidia: "no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml
Unfortunately this config change did not work on Centos 7, but it did change the rootless error to: nvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\""
This config change breaks podman running from root, with the error: Failed to initialize NVML: Unknown Error
Interestingly, root on ubuntu gets the same error even though rootless works. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you know of anyone from Nvidia that is working on this plugin to see why it does not work with podman?
On 10/17/19 4:33 PM, eae@us.ibm.com wrote:
We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless working with nvidia: "no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml
Unfortunately this config change did not work on Centos 7, but it did change the rootless error to: nvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\""
This config change breaks podman running from root, with the error: Failed to initialize NVML: Unknown Error
Interestingly, root on ubuntu gets the same error even though rootless works. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Opened an issue with nvidia. https://github.com/NVIDIA/nvidia-container-runtime/issues/85
participants (6)
-
Bryan Hepworth -
Daniel Walsh -
eae＠us.ibm.com -
Giuseppe Scrivano -
Lou DeGenaro -
Scott McCarty