We discovered that the ubuntu 18.04 machine had a configuration change to get rootless working with nvidia:
"no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml

Unfortunately this config change did not work on Centos 7, but it did change the rootless error to:
nvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\""

This config change breaks podman running from root, with the error:
Failed to initialize NVML: Unknown Error

Interestingly, root on ubuntu gets the same error even though rootless works.

_______________________________________________
Podman mailing list -- podman@lists.podman.io
To unsubscribe send an email to podman-leave@lists.podman.io

I think we would need someone more experienced in nvidia plugin to comment on this.