3:33 p.m.
This is a spinoff / continuation of my prior thread that should hopefully be a bit more
generic and therefore applicable to more people.
Simply put: what work do i need to do to a host prior to invoking `podman run...` on a
rootless container?
As best i can tell:
- Create a system level user (usually a U/GID under 1000 and no home-dir, password,
shell)
- Create a new sub UID/GID range in /etc/subuid and /etc/subgid file that the user/groups
*in* the container will map to *on* the host
- Create space on the host for the volumes and other files that'll need to get mounted
into the container
And then this is where I get lost.
I'd *like* to make the permissions applied to the on-host directories as narrow as
possible, but I've not found a reliable way to determine which U/GID should be applied
to the file/folder.
If I create a host system user with UID 995, this UID won't be what gets mapped into
the container which will result in "not permitted" errors when the process
inside the container tries to touch files that are mapped from the host into the
container.
So i've started to use a rather crude approach:
- chmod -R 777 /path/to/dir/that/mounts/into/container
- podman run ...
- ls -lah /path/to/dir/that/mounts/into/container
- chown $(uid from above step) /path/to/dir/that/mounts/into/container
- chmod -R 0750 /path/to/dir/that/mounts/into/container
My question is there a better way?
In the specific case of the prometheus container, the container wants to run as the
`nobody` user which has the ID `65534`. See:
```
/ $ id nobody
uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)
```
And if i look at my `/etc/subuid` file, i see that `prometheus` has `65536` IDs allocated
to it, starting from `427680`. See:
```
$ cat /etc/subuid
<...snip...>
prometheus:427680:65536
```
And using the (crude) method from above, i can see that the files are being written to
disk as the user `493213`. See:
```
prometheus@my-host:/tmp/prom/data$ ls -lah
total 16K
drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:43 .
drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:17 ..
-rw-r--r-- 1 493213 493213 0 Jan 24 12:43 lock
-rw-r--r-- 1 493213 493213 20K Jan 24 12:43 queries.active
drwxr-xr-x 2 493213 493213 4.0K Jan 24 12:43 wal
```
So doing a bit of math we can see that 493213 - 427680 = 65533. Or, said differently,
starting with the user ID 427680, and adding another 65534 users (counting from ID 0) we
get the user id 493213.
I can now change the permissions on the `/tmp/prom/data` path from `drwxrwxrwx &
prometheus prometheus` to `drwx------ & 427680 427680` on the host.
So this brings me to my basic question: Is there a simpler way to get the value `427680`
from podman **prior** to running the container?
Thanks for for your time/help!
-K
6:43 a.m.
New subject: Trying to understand what work must be done to run rootless containers as a system user.
On 1/24/20 10:33 PM, karl(a)touchpoint.io wrote:
This is a spinoff / continuation of my prior thread that should
hopefully be a bit more generic and therefore applicable to more people.
Simply put: what work do i need to do to a host prior to invoking `podman run...` on a
rootless container?
As best i can tell:
- Create a system level user (usually a U/GID under 1000 and no home-dir, password,
shell)
No. Rootless users are usually normal users, and should have home
directories at least, this is where the containers and images are stored.
- Create a new sub UID/GID range in /etc/subuid and /etc/subgid file
that the user/groups *in* the container will map to *on* the host
Yes.
useradd will do this automatically.
- Create space on the host for the volumes and other files
that'll need to get mounted into the container
Usually this is just the
homedir.
And then this is where I get lost.
I'd *like* to make the permissions applied to the on-host directories as narrow as
possible, but I've not found a reliable way to determine which U/GID should be applied
to the file/folder.
If you are running the rootless container as root then the UID
should be
the uid of the user.
If I create a host system user with UID 995, this UID won't be what gets mapped into
the container which will result in "not permitted" errors when the process
inside the container tries to touch files that are mapped from the host into the
container.
By default rootless users UID is mapped into the container as UID=0.
So i've started to use a rather crude approach:
- chmod -R 777 /path/to/dir/that/mounts/into/container
- podman run ...
- ls -lah /path/to/dir/that/mounts/into/container
- chown $(uid from above step) /path/to/dir/that/mounts/into/container
- chmod -R 0750 /path/to/dir/that/mounts/into/container
My question is there a better way?
In the specific case of the prometheus container, the container wants to run as the
`nobody` user which has the ID `65534`. See:
Then you need to create a range of
users that is as large as
65534
```
/ $ id nobody
uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)
```
And if i look at my `/etc/subuid` file, i see that `prometheus` has `65536` IDs allocated
to it, starting from `427680`. See:
```
$ cat /etc/subuid
<...snip...>
prometheus:427680:65536
```
And using the (crude) method from above, i can see that the files are being written to
disk as the user `493213`. See:
```
prometheus@my-host:/tmp/prom/data$ ls -lah
total 16K
drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:43 .
drwxrwxrwx 3 prometheus prometheus 4.0K Jan 24 12:17 ..
-rw-r--r-- 1 493213 493213 0 Jan 24 12:43 lock
-rw-r--r-- 1 493213 493213 20K Jan 24 12:43 queries.active
drwxr-xr-x 2 493213 493213 4.0K Jan 24 12:43 wal
```
So doing a bit of math we can see that 493213 - 427680 = 65533. Or, said differently,
starting with the user ID 427680, and adding another 65534 users (counting from ID 0) we
get the user id 493213.
I can now change the permissions on the `/tmp/prom/data` path from `drwxrwxrwx &
prometheus prometheus` to `drwx------ & 427680 427680` on the host.
So this brings me to my basic question: Is there a simpler way to get the value `427680`
from podman **prior** to running the container?
Thanks for for your time/help!
-K
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
1:46 p.m.
New subject: Trying to understand what work must be done to run rootless containers as a system user.
No. Rootless users are usually normal users, and should have home
directories at least, this is where the containers and images are stored.
Of course. I create system users with the `--create-home` flag for this reason. Primary
benefit to system level users is that they come with easily distinguishable UIDs and no
password expiration policy. It does appear that `useradd` will **not** automatically add
a subg/uid entry for system level users, though. The `man` page does not indicate a way to
change this. Hence me doing it manually in my OP. Creating a regular user does
automatically add the subg/uids though.
Usually this is just the homedir.
Sure. That's a perfectly logical place to store the container images... But for
intuitiveness, this is a terrible place to store files that the container may persist to
the host's disk.
If a container is expected to produce a log file that will be scraped by another process
on the host, then mounting the location the container process expects to write out
it's log file to the host path `/var/log/$service/{app,error}.log` makes infinitely
more sense than sequestering it away inside `/home/$service/logs/{app,error}.log`.
Additionally, now I have to give the `$log_ingester` process read/write (for rotation)
access to `/home/$service/logs/*`.
regardless, **any** location that is to be mapped into the container for persistent
storage requires xx4/6/7 level permissions **or** pre-existing knowledge of which sub id
will be used to do the actual file writes.
This is acceptable for some things like logs which are known not to contain any sensitive
info, but absolutely unacceptable for other workloads, like a database.
I'll have more on this below...
If you are running the rootless container as root then the UID should
be the uid of the user.
This may be where understanding is broken, then. I am **not** running the container as
root. I thought i made that clear in the OP.
To be explicit: I am running the `prometheus` container as the `prometheus` user on the
host. Within the container, the `prometheus` container image is built so that the process
within the container runs as `nobody`.
Should i be using the hosts root user to invoke `podman run` and passing in the
`--user=prometheus` flag, then?
Should i be building my own prometheus container image so that the service runs as root?
so to recap, then:
- create regular users **or** manually create a home dir + subg/uid range for a system
user
- there is no reliable or automatic way to determine which UID should "own" any
files/dirs that are to be mapped into a container
- any file/dir that is to be mapped into the container will need to be world read/write
initially.
- once the container has successfully started up and written to (host) disk, the sub-ids
that should "own" files can be known and the world read/write permissions can be
revoked and the user/group ID can be properly set on host.
My "process" for determining which user/group should "own" the files
mounted into the container:
##
# I have made a place for the configuration file to live: /tmp/prom.
# The host user `prom` owns this directory and the file in it
##
```
root@com:/tmp/prom# ls -lah
total 12K
drwxr-x--- 2 prom prom 4.0K Jan 28 19:06 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:05 ..
-rw-r----- 1 prom prom 551 Jan 27 19:54 prometheus.yml
```
##
# The host user prom has a proper subgid and subuid map
# The map range is big enough as it allows for `65536` which is
# just bigger than the `65534` i should need for this container
##
```
root@com:/tmp/prom# cat /etc/subuid | grep prom
prom:624288:65536
root@com:/tmp/prom# cat /etc/subgid | grep prom
prom:624288:65536
```
##
# So the host user `prom` should be able to start up the prometheus container
# and read the files that it owns.
##
```
prom@com:/tmp/prom$ whoami
prom
prom@com:/tmp/prom$ cat /tmp/prom/prometheus.yml | wc -l
27
```
##
# But then the container can't actually read the file
##
```
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus prom/prometheus
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 dfc11d48288e
(none))"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:334 vm_limits="(soft=unlimited,
hard=unlimited)"
level=info ts=2020-01-28T19:14:31.802Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:14:31.802Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:14:31.810Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:14:31.810Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=0
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:517 msg="Stopping scrape
discovery manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:531 msg="Stopping notify
discovery manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:553 msg="Stopping scrape
manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:513 msg="Scrape discovery
manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:814 component="rule
manager" msg="Stopping rule manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:820 component="rule
manager" msg="Rule manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:527 msg="Notify discovery
manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:547 msg="Scrape manager
stopped"
level=info ts=2020-01-28T19:14:31.813Z caller=notifier.go:598 component=notifier
msg="Stopping notification manager..."
level=info ts=2020-01-28T19:14:31.814Z caller=main.go:718 msg="Notifier manager
stopped"
level=error ts=2020-01-28T19:14:31.814Z caller=main.go:727 err="error loading config
from \"/etc/prometheus/prometheus.yml\": couldn't load configuration
(--config.file=\"/etc/prometheus/prometheus.yml\"): open
/etc/prometheus/prometheus.yml: permission denied"
```
##
# So i then make the yaml file full 777 and try again.
# Additionally, i'd like the data that this container will create/store to persist to
(host) disk.
# I'll make a folder on the host for this data: /tmp/prom/data
##
```
root@com:/tmp/prom# chmod 777 *
prom@com:/tmp/prom$ mkdir data
prom@com:/tmp/prom$ chmod 0750 data/
prom@com:/tmp/prom$ ls -lah
total 16K
drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxr-x--- 2 prom prom 4.0K Jan 28 19:20 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
```
##
# The container is configured to store all of it's data in the `/prometheus` path
# so there are two mount arguments fed into the podman run command
##
```
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
Error: container_linux.go:346: starting container process caused "chdir to cwd
(\"/prometheus\") set in config.json failed: permission denied": OCI
runtime permission denied error
```
###
# No such luck!
# Make /tmp/prom/data 777 and try again...
##
```
root@com:/tmp/prom# chmod 777 data/
root@com:/tmp/prom# ls -lah
total 16K
drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxrwxrwx 2 prom prom 4.0K Jan 28 19:20 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 eacd0e5fbbe2
(none))"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:334 vm_limits="(soft=unlimited,
hard=unlimited)"
level=info ts=2020-01-28T19:26:44.144Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:26:44.144Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:26:44.151Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:26:44.151Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=0
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:517 msg="Stopping scrape
discovery manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:531 msg="Stopping notify
discovery manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:553 msg="Stopping scrape
manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:527 msg="Notify discovery
manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:513 msg="Scrape discovery
manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:814 component="rule
manager" msg="Stopping rule manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:820 component="rule
manager" msg="Rule manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:547 msg="Scrape manager
stopped"
level=info ts=2020-01-28T19:26:44.155Z caller=notifier.go:598 component=notifier
msg="Stopping notification manager..."
level=info ts=2020-01-28T19:26:44.155Z caller=main.go:718 msg="Notifier manager
stopped"
level=error ts=2020-01-28T19:26:44.155Z caller=main.go:727 err="error loading config
from \"/etc/prometheus/prometheus.yml\": couldn't load configuration
(--config.file=\"/etc/prometheus/prometheus.yml\"): open
/etc/prometheus/prometheus.yml: permission denied"
prom@com:/tmp/prom$
```
##
# Still, no such luck.
# But what if the folder *containing* _all_ the prometheus data was WORLD READ/WRITE...
##
```
root@com:/tmp# chmod 777 prom/
root@com:/tmp# ls -lah prom/
total 16K
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 aeeb1ee9e821
(none))"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:334 vm_limits="(soft=unlimited,
hard=unlimited)"
level=info ts=2020-01-28T19:35:14.706Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:35:14.706Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:35:14.709Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:35:14.710Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=1
level=info ts=2020-01-28T19:35:14.711Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=1 maxSegment=1
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:35:14.715Z caller=main.go:762 msg="Completed loading of
configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:35:14.715Z caller=main.go:617 msg="Server is ready to
receive web requests."
```
##
# Oh hey! That worked!
# Taking a look at the UID that created the data files...
##
```
root@com:/tmp/prom/data# ls
lock queries.active wal
root@com:/tmp/prom/data# ls -lah
total 16K
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 .
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 ..
-rw-r--r-- 1 689821 689821 0 Jan 28 19:26 lock
-rw-r--r-- 1 689821 689821 20K Jan 28 19:35 queries.active
drwxr-xr-x 2 689821 689821 4.0K Jan 28 19:35 wal
```
3:30 p.m.
New subject: Trying to understand what work must be done to run rootless containers as a system user.
What process is going to start this user container? Is this any more
secure then just running as root with the --promethious user?
I would guess it is slightly more, since root inside of the container is
not root on the host, but if you set the --no-new-privs on the container
then, the container process can never become root.
On 1/28/20 2:46 PM, karl(a)touchpoint.io wrote:
> No. Rootless users are usually normal users, and should have
home directories at least, this is where the containers and images are stored.
Of course. I create system users with the `--create-home` flag for this reason. Primary
benefit to system level users is that they come with easily distinguishable UIDs and no
password expiration policy. It does appear that `useradd` will **not** automatically add
a subg/uid entry for system level users, though. The `man` page does not indicate a way to
change this. Hence me doing it manually in my OP. Creating a regular user does
automatically add the subg/uids though.
> Usually this is just the homedir.
Sure. That's a perfectly logical place to store the container images... But for
intuitiveness, this is a terrible place to store files that the container may persist to
the host's disk.
If a container is expected to produce a log file that will be scraped by another process
on the host, then mounting the location the container process expects to write out
it's log file to the host path `/var/log/$service/{app,error}.log` makes infinitely
more sense than sequestering it away inside `/home/$service/logs/{app,error}.log`.
Additionally, now I have to give the `$log_ingester` process read/write (for rotation)
access to `/home/$service/logs/*`.
regardless, **any** location that is to be mapped into the container for persistent
storage requires xx4/6/7 level permissions **or** pre-existing knowledge of which sub id
will be used to do the actual file writes.
This is acceptable for some things like logs which are known not to contain any sensitive
info, but absolutely unacceptable for other workloads, like a database.
I'll have more on this below...
> If you are running the rootless container as root then the UID should be the uid of
the user.
This may be where understanding is broken, then. I am **not** running the container as
root. I thought i made that clear in the OP.
To be explicit: I am running the `prometheus` container as the `prometheus` user on the
host. Within the container, the `prometheus` container image is built so that the process
within the container runs as `nobody`.
Should i be using the hosts root user to invoke `podman run` and passing in the
`--user=prometheus` flag, then?
Should i be building my own prometheus container image so that the service runs as root?
so to recap, then:
- create regular users **or** manually create a home dir + subg/uid range for a system
user
- there is no reliable or automatic way to determine which UID should "own" any
files/dirs that are to be mapped into a container
- any file/dir that is to be mapped into the container will need to be world read/write
initially.
- once the container has successfully started up and written to (host) disk, the sub-ids
that should "own" files can be known and the world read/write permissions can be
revoked and the user/group ID can be properly set on host.
My "process" for determining which user/group should "own" the files
mounted into the container:
##
# I have made a place for the configuration file to live: /tmp/prom.
# The host user `prom` owns this directory and the file in it
##
```
root@com:/tmp/prom# ls -lah
total 12K
drwxr-x--- 2 prom prom 4.0K Jan 28 19:06 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:05 ..
-rw-r----- 1 prom prom 551 Jan 27 19:54 prometheus.yml
```
##
# The host user prom has a proper subgid and subuid map
# The map range is big enough as it allows for `65536` which is
# just bigger than the `65534` i should need for this container
##
```
root@com:/tmp/prom# cat /etc/subuid | grep prom
prom:624288:65536
root@com:/tmp/prom# cat /etc/subgid | grep prom
prom:624288:65536
```
##
# So the host user `prom` should be able to start up the prometheus container
# and read the files that it owns.
##
```
prom@com:/tmp/prom$ whoami
prom
prom@com:/tmp/prom$ cat /tmp/prom/prometheus.yml | wc -l
27
```
##
# But then the container can't actually read the file
##
```
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus prom/prometheus
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 dfc11d48288e
(none))"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:14:31.801Z caller=main.go:334
vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-01-28T19:14:31.802Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:14:31.802Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:14:31.810Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:14:31.810Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=0
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:14:31.811Z caller=main.go:517 msg="Stopping scrape
discovery manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:531 msg="Stopping notify
discovery manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:553 msg="Stopping scrape
manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:513 msg="Scrape discovery
manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:814 component="rule
manager" msg="Stopping rule manager..."
level=info ts=2020-01-28T19:14:31.812Z caller=manager.go:820 component="rule
manager" msg="Rule manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:527 msg="Notify discovery
manager stopped"
level=info ts=2020-01-28T19:14:31.812Z caller=main.go:547 msg="Scrape manager
stopped"
level=info ts=2020-01-28T19:14:31.813Z caller=notifier.go:598 component=notifier
msg="Stopping notification manager..."
level=info ts=2020-01-28T19:14:31.814Z caller=main.go:718 msg="Notifier manager
stopped"
level=error ts=2020-01-28T19:14:31.814Z caller=main.go:727 err="error loading config
from \"/etc/prometheus/prometheus.yml\": couldn't load configuration
(--config.file=\"/etc/prometheus/prometheus.yml\"): open
/etc/prometheus/prometheus.yml: permission denied"
```
##
# So i then make the yaml file full 777 and try again.
# Additionally, i'd like the data that this container will create/store to persist to
(host) disk.
# I'll make a folder on the host for this data: /tmp/prom/data
##
```
root@com:/tmp/prom# chmod 777 *
prom@com:/tmp/prom$ mkdir data
prom@com:/tmp/prom$ chmod 0750 data/
prom@com:/tmp/prom$ ls -lah
total 16K
drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxr-x--- 2 prom prom 4.0K Jan 28 19:20 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
```
##
# The container is configured to store all of it's data in the `/prometheus` path
# so there are two mount arguments fed into the podman run command
##
```
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
Error: container_linux.go:346: starting container process caused "chdir to cwd
(\"/prometheus\") set in config.json failed: permission denied": OCI
runtime permission denied error
```
###
# No such luck!
# Make /tmp/prom/data 777 and try again...
##
```
root@com:/tmp/prom# chmod 777 data/
root@com:/tmp/prom# ls -lah
total 16K
drwxr-x--- 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxrwxrwx 2 prom prom 4.0K Jan 28 19:20 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 eacd0e5fbbe2
(none))"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:26:44.143Z caller=main.go:334
vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-01-28T19:26:44.144Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:26:44.144Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:26:44.151Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:26:44.151Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=0
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:517 msg="Stopping scrape
discovery manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:531 msg="Stopping notify
discovery manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:553 msg="Stopping scrape
manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:527 msg="Notify discovery
manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:513 msg="Scrape discovery
manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:814 component="rule
manager" msg="Stopping rule manager..."
level=info ts=2020-01-28T19:26:44.153Z caller=manager.go:820 component="rule
manager" msg="Rule manager stopped"
level=info ts=2020-01-28T19:26:44.153Z caller=main.go:547 msg="Scrape manager
stopped"
level=info ts=2020-01-28T19:26:44.155Z caller=notifier.go:598 component=notifier
msg="Stopping notification manager..."
level=info ts=2020-01-28T19:26:44.155Z caller=main.go:718 msg="Notifier manager
stopped"
level=error ts=2020-01-28T19:26:44.155Z caller=main.go:727 err="error loading config
from \"/etc/prometheus/prometheus.yml\": couldn't load configuration
(--config.file=\"/etc/prometheus/prometheus.yml\"): open
/etc/prometheus/prometheus.yml: permission denied"
prom@com:/tmp/prom$
```
##
# Still, no such luck.
# But what if the folder *containing* _all_ the prometheus data was WORLD READ/WRITE...
##
```
root@com:/tmp# chmod 777 prom/
root@com:/tmp# ls -lah prom/
total 16K
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 .
drwxrwxrwt 17 root root 4.0K Jan 28 19:21 ..
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 data
-rwxrwxrwx 1 prom prom 551 Jan 27 19:54 prometheus.yml
prom@com:/tmp/prom$ podman run -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:294 msg="no time or size
retention was set so using the default time retention" duration=15d
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:330 msg="Starting
Prometheus" version="(version=2.15.2, branch=HEAD,
revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:331
build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:332 host_details="(Linux
4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 aeeb1ee9e821
(none))"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:333 fd_limits="(soft=1024,
hard=1024)"
level=info ts=2020-01-28T19:35:14.704Z caller=main.go:334
vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-01-28T19:35:14.706Z caller=main.go:648 msg="Starting TSDB
..."
level=info ts=2020-01-28T19:35:14.706Z caller=web.go:506 component=web msg="Start
listening for connections" address=0.0.0.0:9090
level=info ts=2020-01-28T19:35:14.709Z caller=head.go:584 component=tsdb
msg="replaying WAL, this may take awhile"
level=info ts=2020-01-28T19:35:14.710Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=0 maxSegment=1
level=info ts=2020-01-28T19:35:14.711Z caller=head.go:632 component=tsdb msg="WAL
segment loaded" segment=1 maxSegment=1
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:664 msg="TSDB started"
level=info ts=2020-01-28T19:35:14.713Z caller=main.go:734 msg="Loading configuration
file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:35:14.715Z caller=main.go:762 msg="Completed loading of
configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-01-28T19:35:14.715Z caller=main.go:617 msg="Server is ready to
receive web requests."
```
##
# Oh hey! That worked!
# Taking a look at the UID that created the data files...
##
```
root@com:/tmp/prom/data# ls
lock queries.active wal
root@com:/tmp/prom/data# ls -lah
total 16K
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:26 .
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 ..
-rw-r--r-- 1 689821 689821 0 Jan 28 19:26 lock
-rw-r--r-- 1 689821 689821 20K Jan 28 19:35 queries.active
drwxr-xr-x 2 689821 689821 4.0K Jan 28 19:35 wal
```
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
6:43 p.m.
New subject: Trying to understand what work must be done to run rootless containers as a system user.
Hmm, looks like my email didn't make it into the list. Probably for the better, that
was sent from a mobile device and probably had typos.
*re* posting this from web.
---
What process is going to start this user container?
systemd in my specific case, but i presume that the same question would exist w/ any
process supervisor system.
Is this any more secure then just running as root with the
--promethious user?
I still don't understand your question? I am **trying** to run as the prometheus
user.
Up until now, i had been configuring systemd to launch invoke the `podman run ...` command
**as** the host/prometheus user with a `User=prom` directive in the `.service` file.
However, i had **not** been using the `--user=root` option on the container. By forcing
the container image to run as `root` rather than whatever the `USER` directive in the
Dockerfile was, completely sidesteps the need for subuids and managing their access to
files on the host. As seen here:
##
# Ignore the 777 perms on the dir, the *important* part is the uid/gid map to the
host/prom user
##
```
root@com:/tmp/prom/data# ls -lah
total 16K
drwxrwxrwx 3 prom prom 4.0K Jan 29 00:23 .
drwxrwxrwx 3 prom prom 4.0K Jan 28 19:20 ..
-rw-r--r-- 1 prom prom 0 Jan 29 00:23 lock
-rw-r--r-- 1 prom prom 20K Jan 29 00:23 queries.active
drwxr-xr-x 2 prom prom 4.0K Jan 29 00:23 wal
```
##
# Note that the host/prom user is invoking the `podman run` command... my original goal!
##
prom@com:/home/karl$ podman run --user=root -p 9090:9090 -v /tmp/prom:/etc/prometheus -v
/tmp/prom/data:/prometheus prom/prometheus
<snip>
level=info ts=2020-01-29T00:23:11.027Z caller=main.go:617 msg="Server is ready to
receive web requests."
```
to recap:
- System level users are fine for rootless containers, provided that they do have a home
dir for container image storage.
- There is no simple way to determine precisely which UID will "own" the host
files that are mapped into a container. It is possible to do so after the container has
been run once. For a variety of reasons, this method should be avoided!
- The simple solution is to make the sub g/uid range irrelevant by supplying the
`--user-=root` flag to the `podman run` command, effectively making the "owner"
of all the files mapped into the container equal to the UID of the host/user that invoked
`podman run` plus an offset of 0.
- Supplying the `--user=root` flag is not necessary if the container image is built
without a `USER` directive... but a professionally done pre-made container image without
the `USER` directive is unlikely as Docker's published best practices suggest running
the process in the container under an explicit non-root UID.
1793
days inactive
1798
days old
4 comments
2 participants
participants (2)
-
Daniel Walsh -
karl＠touchpoint.io