We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless
working with nvidia:
"no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml
Unfortunately this config change did not work on Centos 7, but it did change the rootless
error to:
nvidia-container-cli: initialization error: cuda error: unknown
error\\\\n\\\"\""
This config change breaks podman running from root, with the error:
Failed to initialize NVML: Unknown Error
Interestingly, root on ubuntu gets the same error even though rootless works.