Trying to understand what work must be done to run rootless containers as a system user.
This is a spinoff / continuation of my prior thread that should hopefully be a bit more generic and therefore applicable to more people. Simply put: what work do i need to do to a host prior to invoking `podman run...` on a rootless container? As best i can tell: - Create a system level user (usually a U/GID under 1000 and no home-dir, password, shell) - Create a new sub UID/GID range in /etc/subuid and /etc/subgid file that the user/groups *in* the container will map to *on* the host - Create space on the host for the volumes and other files that'll need to get mounted into the container And then this is where I get lost. I'd *like* to make the permissions applied to the on-host directories as narrow as possible, but I've not found a reliable way to determine which U/GID should be applied to the file/folder. If I create a host system user with UID 995, this UID won't be what gets mapped into the container which will result in "not permitted" errors when the process inside the container tries to touch files that are mapped from the host into the container. So i've started to use a rather crude approach: - chmod -R 777 /path/to/dir/that/mounts/into/container - podman run ... - ls -lah /path/to/dir/that/mounts/into/container - chown $(uid from above step) /path/to/dir/that/mounts/into/container - chmod -R 0750 /path/to/dir/that/mounts/into/container My question is there a better way? In the specific case of the prometheus container, the container wants to run as the `nobody` user which has the ID `65534`. See: ``` / $ id nobody uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) ``` And if i look at my `/etc/subuid` file, i see that `prometheus` has `65536` IDs allocated to it, starting from `427680`. See: ``` $ cat /etc/subuid <...snip...> prometheus:427680:65536 ``` And using the (crude) method from above, i can see that the files are being written to disk as the user `493213`. See: ``` prometheus@my-host:/tmp/prom/data$ ls -lah total 16K drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:43 . drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:17 .. -rw-r--r-- 1 493213 493213 0 Jan 24 12:43 lock -rw-r--r-- 1 493213 493213 20K Jan 24 12:43 queries.active drwxr-xr-x 2 493213 493213 4.0K Jan 24 12:43 wal ``` So doing a bit of math we can see that 493213 - 427680 = 65533. Or, said differently, starting with the user ID 427680, and adding another 65534 users (counting from ID 0) we get the user id 493213. I can now change the permissions on the `/tmp/prom/data` path from `drwxrwxrwx & prometheus prometheus` to `drwx------ & 427680 427680` on the host. So this brings me to my basic question: Is there a simpler way to get the value `427680` from podman **prior** to running the container? Thanks for for your time/help! -K
This is a spinoff / continuation of my prior thread that should hopefully be a bit more generic and therefore applicable to more people.
Simply put: what work do i need to do to a host prior to invoking `podman run...` on a rootless container?
As best i can tell:
- Create a system level user (usually a U/GID under 1000 and no home-dir, password, shell) No. Rootless users are usually normal users, and should have home
On 1/24/20 10:33 PM, karl@touchpoint.io wrote: directories at least, this is where the containers and images are stored.
- Create a new sub UID/GID range in /etc/subuid and /etc/subgid file that the user/groups *in* the container will map to *on* the host
- Create space on the host for the volumes and other files that'll need to get mounted into the container Usually this is just the homedir. And then this is where I get lost.
I'd *like* to make the permissions applied to the on-host directories as narrow as possible, but I've not found a reliable way to determine which U/GID should be applied to the file/folder. If you are running the rootless container as root then the UID should be
Yes. useradd will do this automatically. the uid of the user.
If I create a host system user with UID 995, this UID won't be what gets mapped into the container which will result in "not permitted" errors when the process inside the container tries to touch files that are mapped from the host into the container.
By default rootless users UID is mapped into the container as UID=0.
So i've started to use a rather crude approach:
- chmod -R 777 /path/to/dir/that/mounts/into/container - podman run ... - ls -lah /path/to/dir/that/mounts/into/container - chown $(uid from above step) /path/to/dir/that/mounts/into/container - chmod -R 0750 /path/to/dir/that/mounts/into/container
My question is there a better way? In the specific case of the prometheus container, the container wants to run as the `nobody` user which has the ID `65534`. See:
Then you need to create a range of users that is as large as 65534
``` / $ id nobody uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) ```
And if i look at my `/etc/subuid` file, i see that `prometheus` has `65536` IDs allocated to it, starting from `427680`. See:
``` $ cat /etc/subuid <...snip...> prometheus:427680:65536 ```
And using the (crude) method from above, i can see that the files are being written to disk as the user `493213`. See:
``` prometheus@my-host:/tmp/prom/data$ ls -lah total 16K drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:43 . drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:17 .. -rw-r--r-- 1 493213 493213 0 Jan 24 12:43 lock -rw-r--r-- 1 493213 493213 20K Jan 24 12:43 queries.active drwxr-xr-x 2 493213 493213 4.0K Jan 24 12:43 wal ```
So doing a bit of math we can see that 493213 - 427680 = 65533. Or, said differently, starting with the user ID 427680, and adding another 65534 users (counting from ID 0) we get the user id 493213.
I can now change the permissions on the `/tmp/prom/data` path from `drwxrwxrwx & prometheus prometheus` to `drwx------ & 427680 427680` on the host.
So this brings me to my basic question: Is there a simpler way to get the value `427680` from podman **prior** to running the container?
Thanks for for your time/help!
-K
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
No. Rootless users are usually normal users, and should have home directories at least, this is where the containers and images are stored.
Of course. I create system users with the `--create-home` flag for this reason. Primary benefit to system level users is that they come with easily distinguishable UIDs and no password expiration policy. It does appear that `useradd` will **not** automatically add a subg/uid entry for system level users, though. The `man` page does not indicate a way to change this. Hence me doing it manually in my OP. Creating a regular user does automatically add the subg/uids though.
Usually this is just the homedir.
Sure. That's a perfectly logical place to store the container images... But for intuitiveness, this is a terrible place to store files that the container may persist to the host's disk. If a container is expected to produce a log file that will be scraped by another process on the host, then mounting the location the container process expects to write out it's log file to the host path `/var/log/$service/{app,error}.log` makes infinitely more sense than sequestering it away inside `/home/$service/logs/{app,error}.log`. Additionally, now I have to give the `$log_ingester` process read/write (for rotation) access to `/home/$service/logs/*`. regardless, **any** location that is to be mapped into the container for persistent storage requires xx4/6/7 level permissions **or** pre-existing knowledge of which sub id will be used to do the actual file writes. This is acceptable for some things like logs which are known not to contain any sensitive info, but absolutely unacceptable for other workloads, like a database. I'll have more on this below...
If you are running the rootless container as root then the UID should be the uid of the user.
This may be where understanding is broken, then. I am **not** running the container as root. I thought i made that clear in the OP. To be explicit: I am running the `prometheus` container as the `prometheus` user on the host. Within the container, the `prometheus` container image is built so that the process within the container runs as `nobody`. Should i be using the hosts root user to invoke `podman run` and passing in the `--user=prometheus` flag, then? Should i be building my own prometheus container image so that the service runs as root? so to recap, then: - create regular users **or** manually create a home dir + subg/uid range for a system user - there is no reliable or automatic way to determine which UID should "own" any files/dirs that are to be mapped into a container - any file/dir that is to be mapped into the container will need to be world read/write initially. - once the container has successfully started up and written to (host) disk, the sub-ids that should "own" files can be known and the world read/write permissions can be revoked and the user/group ID can be properly set on host. My "process" for determining which user/group should "own" the files mounted into the container: ## # I have made a place for the configuration file to live: /tmp/prom. # The host user `prom` owns this directory and the file in it ## ``` root@com:/tmp/prom# ls -lah total 12K drwxr-x--- 2 prom prom 4.0K Jan 28 19:06 . drwxrwxrwt 17 root root 4.0K Jan 28 19:05 .. -rw-r----- 1 prom prom 551 Jan 27 19:54 prometheus.yml ``` ## # The host user prom has a proper subgid and subuid map # The map range is big enough as it allows for `65536` which is # just bigger than the `65534` i should need for this container ## ``` root@com:/tmp/prom# cat /etc/subuid | grep prom prom:624288:65536 root@com:/tmp/prom# cat /etc/subgid | grep prom prom:624288:65536 ``` ## # So the host user `prom` should be able to start up the prometheus container # and read the files that it owns. ## ``` prom@com:/tmp/prom$ whoami prom prom@com:/tmp/prom$ cat /tmp/prom/prometheus.yml | wc -l 27 ``` ## # But then the container can't actually read the file ## ``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus prom/prometheus level=info ts=2020-01-28T19:14:31.801Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:14:31.801Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 dfc11d48288e (none))" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:14:31.802Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:14:31.802Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:14:31.810Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:14:31.810Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:14:31.811Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:14:31.811Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:14:31.811Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:14:31.811Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:14:31.813Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:14:31.814Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:14:31.814Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" ``` ## # So i then make the yaml file full 777 and try again. # Additionally, i'd like the data that this container will create/store to persist to (host) disk. # I'll make a folder on the host for this data: /tmp/prom/data ## ``` root@com:/tmp/prom# chmod 777 * prom@com:/tmp/prom$ mkdir data prom@com:/tmp/prom$ chmod 0750 data/ prom@com:/tmp/prom$ ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxr-x--- 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml ``` ## # The container is configured to store all of it's data in the `/prometheus` path # so there are two mount arguments fed into the podman run command ## ``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus Error: container_linux.go:346: starting container process caused "chdir to cwd (\"/prometheus\") set in config.json failed: permission denied": OCI runtime permission denied error ``` ### # No such luck! # Make /tmp/prom/data 777 and try again... ## ``` root@com:/tmp/prom# chmod 777 data/ root@com:/tmp/prom# ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:26:44.143Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:26:44.143Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 eacd0e5fbbe2 (none))" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:26:44.144Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:26:44.144Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:26:44.151Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:26:44.151Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:26:44.153Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:26:44.153Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:26:44.153Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:26:44.155Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:26:44.155Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:26:44.155Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" prom@com:/tmp/prom$ ``` ## # Still, no such luck. # But what if the folder *containing* _all_ the prometheus data was WORLD READ/WRITE... ## ``` root@com:/tmp# chmod 777 prom/ root@com:/tmp# ls -lah prom/ total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:35:14.704Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:35:14.704Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 aeeb1ee9e821 (none))" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:35:14.706Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:35:14.706Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:35:14.709Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:35:14.710Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 level=info ts=2020-01-28T19:35:14.711Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 level=info ts=2020-01-28T19:35:14.713Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:35:14.713Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:35:14.713Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:617 msg="Server is ready to receive web requests." ``` ## # Oh hey! That worked! # Taking a look at the UID that created the data files... ## ``` root@com:/tmp/prom/data# ls lock queries.active wal root@com:/tmp/prom/data# ls -lah total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 . drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .. -rw-r--r-- 1 689821 689821 0 Jan 28 19:26 lock -rw-r--r-- 1 689821 689821 20K Jan 28 19:35 queries.active drwxr-xr-x 2 689821 689821 4.0K Jan 28 19:35 wal ```
What process is going to start this user container? Is this any more secure then just running as root with the --promethious user? I would guess it is slightly more, since root inside of the container is not root on the host, but if you set the --no-new-privs on the container then, the container process can never become root. On 1/28/20 2:46 PM, karl@touchpoint.io wrote:
No. Rootless users are usually normal users, and should have home directories at least, this is where the containers and images are stored. Of course. I create system users with the `--create-home` flag for this reason. Primary benefit to system level users is that they come with easily distinguishable UIDs and no password expiration policy. It does appear that `useradd` will **not** automatically add a subg/uid entry for system level users, though. The `man` page does not indicate a way to change this. Hence me doing it manually in my OP. Creating a regular user does automatically add the subg/uids though.
Usually this is just the homedir. Sure. That's a perfectly logical place to store the container images... But for intuitiveness, this is a terrible place to store files that the container may persist to the host's disk.
If a container is expected to produce a log file that will be scraped by another process on the host, then mounting the location the container process expects to write out it's log file to the host path `/var/log/$service/{app,error}.log` makes infinitely more sense than sequestering it away inside `/home/$service/logs/{app,error}.log`. Additionally, now I have to give the `$log_ingester` process read/write (for rotation) access to `/home/$service/logs/*`.
regardless, **any** location that is to be mapped into the container for persistent storage requires xx4/6/7 level permissions **or** pre-existing knowledge of which sub id will be used to do the actual file writes.
This is acceptable for some things like logs which are known not to contain any sensitive info, but absolutely unacceptable for other workloads, like a database.
I'll have more on this below...
If you are running the rootless container as root then the UID should be the uid of the user. This may be where understanding is broken, then. I am **not** running the container as root. I thought i made that clear in the OP.
To be explicit: I am running the `prometheus` container as the `prometheus` user on the host. Within the container, the `prometheus` container image is built so that the process within the container runs as `nobody`.
Should i be using the hosts root user to invoke `podman run` and passing in the `--user=prometheus` flag, then? Should i be building my own prometheus container image so that the service runs as root?
so to recap, then:
- create regular users **or** manually create a home dir + subg/uid range for a system user
- there is no reliable or automatic way to determine which UID should "own" any files/dirs that are to be mapped into a container
- any file/dir that is to be mapped into the container will need to be world read/write initially.
- once the container has successfully started up and written to (host) disk, the sub-ids that should "own" files can be known and the world read/write permissions can be revoked and the user/group ID can be properly set on host.
My "process" for determining which user/group should "own" the files mounted into the container:
## # I have made a place for the configuration file to live: /tmp/prom. # The host user `prom` owns this directory and the file in it ##
``` root@com:/tmp/prom# ls -lah total 12K drwxr-x--- 2 prom prom 4.0K Jan 28 19:06 . drwxrwxrwt 17 root root 4.0K Jan 28 19:05 .. -rw-r----- 1 prom prom 551 Jan 27 19:54 prometheus.yml ```
## # The host user prom has a proper subgid and subuid map # The map range is big enough as it allows for `65536` which is # just bigger than the `65534` i should need for this container ##
``` root@com:/tmp/prom# cat /etc/subuid | grep prom prom:624288:65536 root@com:/tmp/prom# cat /etc/subgid | grep prom prom:624288:65536 ```
## # So the host user `prom` should be able to start up the prometheus container # and read the files that it owns. ##
``` prom@com:/tmp/prom$ whoami prom prom@com:/tmp/prom$ cat /tmp/prom/prometheus.yml | wc -l 27 ```
## # But then the container can't actually read the file ##
``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus prom/prometheus level=info ts=2020-01-28T19:14:31.801Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:14:31.801Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 dfc11d48288e (none))" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:14:31.801Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:14:31.802Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:14:31.802Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:14:31.810Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:14:31.810Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:14:31.811Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:14:31.811Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:14:31.811Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:14:31.811Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:14:31.812Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:14:31.812Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:14:31.813Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:14:31.814Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:14:31.814Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" ```
## # So i then make the yaml file full 777 and try again. # Additionally, i'd like the data that this container will create/store to persist to (host) disk. # I'll make a folder on the host for this data: /tmp/prom/data ##
``` root@com:/tmp/prom# chmod 777 * prom@com:/tmp/prom$ mkdir data prom@com:/tmp/prom$ chmod 0750 data/ prom@com:/tmp/prom$ ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxr-x--- 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
```
## # The container is configured to store all of it's data in the `/prometheus` path # so there are two mount arguments fed into the podman run command ##
``` prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus Error: container_linux.go:346: starting container process caused "chdir to cwd (\"/prometheus\") set in config.json failed: permission denied": OCI runtime permission denied error ```
### # No such luck! # Make /tmp/prom/data 777 and try again... ##
``` root@com:/tmp/prom# chmod 777 data/ root@com:/tmp/prom# ls -lah total 16K drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 2 prom prom 4.0K Jan 28 19:20 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:26:44.143Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:26:44.143Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 eacd0e5fbbe2 (none))" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:26:44.143Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:26:44.144Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:26:44.144Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:26:44.151Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:26:44.151Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 level=info ts=2020-01-28T19:26:44.153Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:26:44.153Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:26:44.153Z caller=main.go:517 msg="Stopping scrape discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:531 msg="Stopping notify discovery manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:553 msg="Stopping scrape manager..." level=info ts=2020-01-28T19:26:44.153Z caller=main.go:527 msg="Notify discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:513 msg="Scrape discovery manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..." level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped" level=info ts=2020-01-28T19:26:44.153Z caller=main.go:547 msg="Scrape manager stopped" level=info ts=2020-01-28T19:26:44.155Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..." level=info ts=2020-01-28T19:26:44.155Z caller=main.go:718 msg="Notifier manager stopped" level=error ts=2020-01-28T19:26:44.155Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied" prom@com:/tmp/prom$
```
## # Still, no such luck. # But what if the folder *containing* _all_ the prometheus data was WORLD READ/WRITE... ##
``` root@com:/tmp# chmod 777 prom/ root@com:/tmp# ls -lah prom/ total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 . drwxrwxrwt 17 root root 4.0K Jan 28 19:21 .. drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 data -rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus level=info ts=2020-01-28T19:35:14.704Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d level=info ts=2020-01-28T19:35:14.704Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:332 host_details="(Linux 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 aeeb1ee9e821 (none))" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)" level=info ts=2020-01-28T19:35:14.704Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)" level=info ts=2020-01-28T19:35:14.706Z caller=main.go:648 msg="Starting TSDB ..." level=info ts=2020-01-28T19:35:14.706Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090 level=info ts=2020-01-28T19:35:14.709Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile" level=info ts=2020-01-28T19:35:14.710Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 level=info ts=2020-01-28T19:35:14.711Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 level=info ts=2020-01-28T19:35:14.713Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC level=info ts=2020-01-28T19:35:14.713Z caller=main.go:664 msg="TSDB started" level=info ts=2020-01-28T19:35:14.713Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-01-28T19:35:14.715Z caller=main.go:617 msg="Server is ready to receive web requests."
```
## # Oh hey! That worked! # Taking a look at the UID that created the data files... ##
``` root@com:/tmp/prom/data# ls lock queries.active wal root@com:/tmp/prom/data# ls -lah total 16K drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 . drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .. -rw-r--r-- 1 689821 689821 0 Jan 28 19:26 lock -rw-r--r-- 1 689821 689821 20K Jan 28 19:35 queries.active drwxr-xr-x 2 689821 689821 4.0K Jan 28 19:35 wal ``` _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Hmm, looks like my email didn't make it into the list. Probably for the better, that was sent from a mobile device and probably had typos. *re* posting this from web. ---
What process is going to start this user container?
systemd in my specific case, but i presume that the same question would exist w/ any process supervisor system.
Is this any more secure then just running as root with the --promethious user?
I still don't understand your question? I am **trying** to run as the prometheus user. Up until now, i had been configuring systemd to launch invoke the `podman run ...` command **as** the host/prometheus user with a `User=prom` directive in the `.service` file. However, i had **not** been using the `--user=root` option on the container. By forcing the container image to run as `root` rather than whatever the `USER` directive in the Dockerfile was, completely sidesteps the need for subuids and managing their access to files on the host. As seen here: ## # Ignore the 777 perms on the dir, the *important* part is the uid/gid map to the host/prom user ## ``` root@com:/tmp/prom/data# ls -lah total 16K drwxrwxrwx 3 prom prom 4.0K Jan 29 00:23 . drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .. -rw-r--r-- 1 prom prom 0 Jan 29 00:23 lock -rw-r--r-- 1 prom prom 20K Jan 29 00:23 queries.active drwxr-xr-x 2 prom prom 4.0K Jan 29 00:23 wal ``` ## # Note that the host/prom user is invoking the `podman run` command... my original goal! ## prom@com:/home/karl$ podman run --user=root -p 9090:9090 -v /tmp/prom:/etc/prometheus -v /tmp/prom/data:/prometheus prom/prometheus <snip> level=info ts=2020-01-29T00:23:11.027Z caller=main.go:617 msg="Server is ready to receive web requests." ``` to recap: - System level users are fine for rootless containers, provided that they do have a home dir for container image storage. - There is no simple way to determine precisely which UID will "own" the host files that are mapped into a container. It is possible to do so after the container has been run once. For a variety of reasons, this method should be avoided! - The simple solution is to make the sub g/uid range irrelevant by supplying the `--user-=root` flag to the `podman run` command, effectively making the "owner" of all the files mapped into the container equal to the UID of the host/user that invoked `podman run` plus an offset of 0. - Supplying the `--user=root` flag is not necessary if the container image is built without a `USER` directive... but a professionally done pre-made container image without the `USER` directive is unlikely as Docker's published best practices suggest running the process in the container under an explicit non-root UID.
participants (2)
-
Daniel Walsh -
karl＠touchpoint.io