[Podman] Re: Trying to understand what work must be done to run rootless containers as a system user.

Of course. I create system users with the `--create-home` flag for this reason. Primary benefit to system level users is that they come with easily distinguishable UIDs and no password expiration policy. It does appear that `useradd` will **not** automatically add a subg/uid entry for system level users, though. The `man` page does not indicate a way to change this. Hence me doing it manually in my OP. Creating a regular user does automatically add the subg/uids though. Sure. That's a perfectly logical place to store the container images... But for intuitiveness, this is a terrible place to store files that the container may persist to the host's disk. If a container is expected to produce a log file that will be scraped by another process on the host, then mounting the location the container process expects to write out it's log file to the host path `/var/log/$service/{app,error}.log` makes infinitely more sense than sequestering it away inside `/home/$service/logs/{app,error}.log`. Additionally, now I have to give the `$log_ingester` process read/write (for rotation) access to `/home/$service/logs/*`. regardless, **any** location that is to be mapped into the container for persistent storage requires xx4/6/7 level permissions **or** pre-existing knowledge of which sub id will be used to do the actual file writes. This is acceptable for some things like logs which are known not to contain any sensitive info, but absolutely unacceptable for other workloads, like a database. I'll have more on this below... This may be where understanding is broken, then. I am **not** running the container as root. I thought i made that clear in the OP. To be explicit: I am running the `prometheus` container as the `prometheus` user on the host. Within the container, the `prometheus` container image is built so that the process within the container runs as `nobody`. Should i be using the hosts root user to invoke `podman run` and passing in the `--user=prometheus` flag, then? Should i be building my own prometheus container image so that the service runs as root? so to recap, then: - create regular users **or** manually create a home dir + subg/uid range for a system user - there is no reliable or automatic way to determine which UID should "own" any files/dirs that are to be mapped into a container - any file/dir that is to be mapped into the container will need to be world read/write initially. - once the container has successfully started up and written to (host) disk, the sub-ids that should "own" files can be known and the world read/write permissions can be revoked and the user/group ID can be properly set on host. My "process" for determining which user/group should "own" the files mounted into the container: ## # I have made a place for the configuration file to live: /tmp/prom. # The host user `prom` owns this directory and the file in it ## ``` root@com:/tmp/prom# ls -lah total 12K drwxr-x--- 2 prom prom 4.0K Jan 28 19:06 . drwxrwxrwt 17 root root 4.0K Jan 28 19:05 .. -rw-r----- 1 prom prom 551 Jan 27 19:54 prometheus.yml ``` ## # The host user prom has a proper subgid and subuid map # The map range is big enough as it allows for `65536` which is # just bigger than the `65534` i should need for this container ## ``` root@com:/tmp/prom# cat /etc/subuid | grep prom prom:624288:65536 root@com:/tmp/prom# cat /etc/subgid | grep prom prom:624288:65536 ``` ## # So the host user `prom` should be able to start up the prometheus container # and read the files that it owns. ## ``` prom@com:/tmp/prom$ whoami prom prom@com:/tmp/prom$ cat /tmp/prom/prometheus.yml | wc -l 27 ``` ## # But then the container can't actually read the file ## ``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus prom/prometheus level=info ts=2020-01-28T19:14:31.801Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:14:31.801Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 dfc11d48288e (none))" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:14:31.802Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:14:31.802Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:14:31.810Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:14:31.810Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:14:31.811Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:14:31.811Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:14:31.811Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:14:31.811Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:14:31.813Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:14:31.814Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:14:31.814Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" ``` ## # So i then make the yaml file full 777 and try again. # Additionally, i'd like the data that this container will create/store to persist to (host) disk. # I'll make a folder on the host for this data: /tmp/prom/data ## ``` root@com:/tmp/prom# chmod 777 * prom@com:/tmp/prom$ mkdir data prom@com:/tmp/prom$ chmod 0750 data/ prom@com:/tmp/prom$ ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxr-x--- 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml ``` ## # The container is configured to store all of it's data in the `/prometheus` path # so there are two mount arguments fed into the podman run command ## ``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus Error: container_linux.go:346: starting container process caused "chdir to cwd (\"/prometheus\") set in config.json failed: permission denied": OCI runtime permission denied error ``` ### # No such luck! # Make /tmp/prom/data 777 and try again... ## ``` root@com:/tmp/prom# chmod 777 data/ root@com:/tmp/prom# ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:26:44.143Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:26:44.143Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 eacd0e5fbbe2 (none))" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:26:44.144Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:26:44.144Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:26:44.151Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:26:44.151Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:26:44.153Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:26:44.153Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:26:44.153Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:26:44.155Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:26:44.155Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:26:44.155Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" prom@com:/tmp/prom$ ``` ## # Still, no such luck. # But what if the folder *containing* _all_ the prometheus data was WORLD READ/WRITE... ## ``` root@com:/tmp# chmod 777 prom/ root@com:/tmp# ls -lah prom/ total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:35:14.704Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:35:14.704Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 aeeb1ee9e821 (none))" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:35:14.706Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:35:14.706Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:35:14.709Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:35:14.710Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 level=info ts=2020-01-28T19:35:14.711Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 level=info ts=2020-01-28T19:35:14.713Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:35:14.713Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:35:14.713Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:617 msg="Server is ready to receive web requests." ``` ## # Oh hey! That worked! # Taking a look at the UID that created the data files... ## ``` root@com:/tmp/prom/data# ls lock queries.active wal root@com:/tmp/prom/data# ls -lah total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 . drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .. -rw-r--r-- 1 689821 689821 0 Jan 28 19:26 lock -rw-r--r-- 1 689821 689821 20K Jan 28 19:35 queries.active drwxr-xr-x 2 689821 689821 4.0K Jan 28 19:35 wal ```