Hi all. I have created a system user on a host and I am trying to run the prometheus container under this user. The prometheus container has been built to use `nobody` for user/group: https://github.com/prometheus/prometheus/blob/master/Dockerfile Which on my system is: ``` $ id nobody uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) ``` But i am not able to pull this image: ``` prometheus@my-host:/home/my-user$ /usr/bin/podman run --rm --publish 9090:9090 --volume=/etc/prometheus:/etc/prometheus --storage-opt ignore_chown_errors prom/prometheus:v2.15.2 Trying to pull docker.io/prom/prometheus:v2.15.2... Getting image source signatures Copying blob 777554bd8f5f done Copying blob b1fd22a632bd done Copying blob 0b9be5b44b0b done Copying blob 0f8c40e1270f done Copying blob 626a2a3fee8c done Copying blob 2fa3c248a32e done Copying blob 537aa55b0cd8 done Copying blob 51148e52a043 done Copying blob 764607aff129 done Copying blob 340dd5e6b5d3 done Copying blob 02800ddf4bd5 done Copying blob 6fd47d0ba3e2 done Copying config b715301fa5 done Writing manifest to image destination Storing signatures ERRO[0005] Error while applying layer: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument Error: unable to pull prom/prometheus:v2.15.2: 1 error occurred: * Error committing the finished image: error adding layer with blob "sha256:0f8c40e1270f10d085dda8ce12b7c5b17cd808f055df5a7222f54837ca0feae0": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument ``` Note: The same error happens w/ and w/o the `--storage-opt ignore_chown_error` which i picked up from https://www.redhat.com/sysadmin/rootless-podman while trying to understand/fix the error. The user: ``` prometheus@my-host:/home/my-user$ whoami prometheus prometheus@my-host:/home/my-user$ id uid=995(prometheus) gid=1008(prometheus) groups=1008(prometheus) prometheus@my-host:/home/my-user$ ``` The u/gid maps: ``` prometheus@my-host:/home/my-user$ cat /etc/subuid lxd:100000:65536 root:100000:65536 my-user:165536:65536 prometheus:231072:65536 prometheus@my-host:/home/my-user$ cat /etc/subgid lxd:100000:65536 root:100000:65536 my-user:165536:65536 prometheus:231072:65536 ``` What am I missing?

Thanks for getting back to me! As the user that the container is intended to run under (`prometheus`): ``` prometheus@my-host:/home/karl$ podman unshare cat /proc/self/uid_map 0 995 1 prometheus@my-host:/home/karl$ podman system migrate ``` But still no luck: ``` Jan 22 19:33:13 my-host podman[12701]: time="2020-01-22T19:33:13-08:00" level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument" Jan 22 19:33:13 my-host podman[12701]: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument Jan 22 19:33:13 my-host podman[12701]: Error: unable to pull prom/prometheus:v2.15.2: 1 error occurred: Jan 22 19:33:13 my-host podman[12701]: * Error committing the finished image: error adding layer with blob "sha256:0f8c40e1270f10d085dda8ce12b7c5b17cd808f055df5a7222f54837ca0feae0": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument Jan 22 19:33:13 my-host systemd[1]: prometheus.service: Main process exited, code=exited, status=125/n/a Jan 22 19:33:13 my-host systemd[1]: prometheus.service: Failed with result 'exit-code'. ``` The `prometheus` user is a SYSTEM user (UID < 1000) and the `systemd` invocation is: ``` $ me@my-host $ systemctl show prometheus.service Type=simple Restart=on-failure NotifyAccess=none RestartUSec=30s TimeoutStartUSec=1min 30s TimeoutStopUSec=30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=exit-code UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Wed 2020-01-22 19:33:07 PST ExecMainStartTimestampMonotonic=629059626061 ExecMainExitTimestamp=Wed 2020-01-22 19:33:13 PST ExecMainExitTimestampMonotonic=629065968504 ExecMainPID=12701 ExecMainCode=1 ExecMainStatus=125 ExecStart={ path=/usr/bin/podman ; argv[]=/usr/bin/podman run --rm --publish 9090:9090 --volume=/etc/prometheus:/etc/prometheus prom/prometheus:v2.15.2 ; ignore_errors=no ; start_time=[Wed 2020-01-22 19:33:07 PST] ; stop_time=[Wed 2020-01-22 19:33:13 PST] ; pid=12701 ; code=exited ; status=125 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=4915 IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=0 LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=4096 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=30338 LimitNPROCSoft=30338 LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=30338 LimitSIGPENDINGSoft=30338 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= User=prometheus Group=prometheus DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=yes PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=full SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=yes SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=prometheus.service Names=prometheus.service Requires=system.slice network-online.target -.mount sysinit.target WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target multi-user.target After=-.mount systemd-journald.socket sysinit.target network-online.target basic.target system.slice systemd-tmpfiles-setup.service RequiresMountsFor=/tmp /var/tmp Documentation=https://github.com/prometheus/prometheus Description="Prometheus Monitoring/Time Series DB" LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/etc/systemd/system/prometheus.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Wed 2020-01-22 19:33:23 PST StateChangeTimestampMonotonic=629075601686 InactiveExitTimestamp=Wed 2020-01-22 19:33:13 PST InactiveExitTimestampMonotonic=629065986648 ActiveEnterTimestamp=Wed 2020-01-22 19:33:07 PST ActiveEnterTimestampMonotonic=629059626117 ActiveExitTimestamp=Wed 2020-01-22 19:33:13 PST ActiveExitTimestampMonotonic=629065968701 InactiveEnterTimestamp=Wed 2020-01-22 19:33:23 PST InactiveEnterTimestampMonotonic=629075601686 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Wed 2020-01-22 19:33:07 PST ConditionTimestampMonotonic=629059624480 AssertTimestamp=Wed 2020-01-22 19:33:07 PST AssertTimestampMonotonic=629059624480 Transient=no Perpetual=no StartLimitIntervalUSec=1min StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=93b5d43f782a44aeb0d04a70b34a9136 CollectMode=inactive