karl@touchpoint.io writes:
Thanks for getting back to me!
As the user that the container is intended to run under (`prometheus`):
``` prometheus@my-host:/home/karl$ podman unshare cat /proc/self/uid_map 0 995 1 prometheus@my-host:/home/karl$ podman system migrate ```
But still no luck:
```
Jan 22 19:33:13 my-host podman[12701]: time="2020-01-22T19:33:13-08:00" level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument" Jan 22 19:33:13 my-host podman[12701]: ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument Jan 22 19:33:13 my-host podman[12701]: Error: unable to pull prom/prometheus:v2.15.2: 1 error occurred: Jan 22 19:33:13 my-host podman[12701]: * Error committing the finished image: error adding layer with blob "sha256:0f8c40e1270f10d085dda8ce12b7c5b17cd808f055df5a7222f54837ca0feae0": ApplyLayer exit status 1 stdout: stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument Jan 22 19:33:13 my-host systemd[1]: prometheus.service: Main process exited, code=exited, status=125/n/a Jan 22 19:33:13 my-host systemd[1]: prometheus.service: Failed with result 'exit-code'. ```
The `prometheus` user is a SYSTEM user (UID < 1000) and the `systemd` invocation is:
``` $ me@my-host $ systemctl show prometheus.service Type=simple Restart=on-failure NotifyAccess=none RestartUSec=30s TimeoutStartUSec=1min 30s TimeoutStopUSec=30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=exit-code UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Wed 2020-01-22 19:33:07 PST ExecMainStartTimestampMonotonic=629059626061 ExecMainExitTimestamp=Wed 2020-01-22 19:33:13 PST ExecMainExitTimestampMonotonic=629065968504 ExecMainPID=12701 ExecMainCode=1 ExecMainStatus=125 ExecStart={ path=/usr/bin/podman ; argv[]=/usr/bin/podman run --rm --publish 9090:9090 --volume=/etc/prometheus:/etc/prometheus prom/prometheus:v2.15.2 ; ignore_errors=no ; start_time=[Wed 2020-01-22 19:33:07 PST] ; stop_time=[Wed 2020-01-22 19:33:13 PST] ; pid=12701 ; code=exited ; status=125 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=4915 IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=0 LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=4096 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=30338 LimitNPROCSoft=30338 LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=30338 LimitSIGPENDINGSoft=30338 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= User=prometheus Group=prometheus DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=yes PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=full SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=yes SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=prometheus.service Names=prometheus.service Requires=system.slice network-online.target -.mount sysinit.target WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target multi-user.target After=-.mount systemd-journald.socket sysinit.target network-online.target basic.target system.slice systemd-tmpfiles-setup.service RequiresMountsFor=/tmp /var/tmp Documentation=https://github.com/prometheus/prometheus Description="Prometheus Monitoring/Time Series DB" LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/etc/systemd/system/prometheus.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Wed 2020-01-22 19:33:23 PST StateChangeTimestampMonotonic=629075601686 InactiveExitTimestamp=Wed 2020-01-22 19:33:13 PST InactiveExitTimestampMonotonic=629065986648 ActiveEnterTimestamp=Wed 2020-01-22 19:33:07 PST ActiveEnterTimestampMonotonic=629059626117 ActiveExitTimestamp=Wed 2020-01-22 19:33:13 PST ActiveExitTimestampMonotonic=629065968701 InactiveEnterTimestamp=Wed 2020-01-22 19:33:23 PST InactiveEnterTimestampMonotonic=629075601686 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Wed 2020-01-22 19:33:07 PST ConditionTimestampMonotonic=629059624480 AssertTimestamp=Wed 2020-01-22 19:33:07 PST AssertTimestampMonotonic=629059624480 Transient=no Perpetual=no StartLimitIntervalUSec=1min StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=93b5d43f782a44aeb0d04a70b34a9136 CollectMode=inactive
there are a bunch of security constraints in the unit file that could block newuidmap/newgidmap. Giuseppe