Should I run podman-based systemd services as root?
Hello! I'm aware of the age-old advice of not running services as root; I've been administering UNIX-like systems for decades now. If you follow the advice given in, for example, this page: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomi... ... What you'll get is a redis container running as root (unless the redis image drops privileges itself - I don't know, I've never run it). I've set up a few production systems running services that are inside podman containers. I'm lucky enough to be using 98% software that can run inside completely unprivileged containers. For all of these containers, I've run each container under its own user ID. The systemd unit for each, for example, does something along these lines: [Service] Type=exec User=_cardant Group=_cardant ExecStart=/usr/bin/podman run ... However, doing things this way is a little messy. For example, if for some reason I want to do something like `podman exec` in a container, I have to `sudo -u _cardant podman exec ...`. `podman ps` will obviously only show me the containers running for the current user. Additionally, any images downloaded from the registry for each service will effectively end up in the home directory of each service user, complicating storage accounting somewhat. The UIDs/GIDs are yet another thing I have to manage, even though they don't have any useful meaning (they don't identify people, they're solely there because the containers have to run as _something_). Containers also leak internal UID/GID values (from the /etc/subuid ranges) into the filesystem, which can complicate things. Additionally, there are some containers that stubbornly make it awkward to run as a non-root user despite not actually needing privileges. The PostgreSQL image is a good example; you can run it as a non-root user and it'll switch to another UID inside the container and then that UID/GID will end on the database files that are inevitably mounted inside the container. You'll also have to match these unpredictable weird UID/GIDs if you want to supply the container with TLS keys/certs, because postgres will refuse to open them unless the UID/GID matches. You can't get around this by telling postgres to run as UID 0; it'll refuse, even though UID 0 inside the container isn't UID 0 outside of it when running unprivileged. I'm running all of these services on systems that have SELinux in enforcing mode. My understanding is that containers will all have the container_t domain and therefore even if they all ran as root, a compromised container would not be able to do any meaningful harm to the system. I'm therefore not certain if the usual "don't run as root" advice applies as containers don't have the same security properties (especially when combined with SELinux). I feel like it'd simplify things if I could safely run all of the containers as root. At the very least, I'd be able to predict UID/GID values inside the containers from outside! I can't get any clear advice on how people are expected to run podman containers in production. All of the various bits of documentation in Linux distributions that talk about running under systemd never bother to talk about UIDs or GIDs. Any documentation on running podman rootless seems to only talk about it in the context of developers running unprivileged containers on their local machines for experimentation/development. If you set up containers via Fedora Server's cockpit UI, you'll get containers running as root everywhere. What is the best practice here? -- Mark Raynsford | https://www.io7m.com
On 7/27/23 14:12, Mark Raynsford via Podman wrote:
Hello!
I'm aware of the age-old advice of not running services as root; I've been administering UNIX-like systems for decades now.
If you follow the advice given in, for example, this page:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomi...
... What you'll get is a redis container running as root (unless the redis image drops privileges itself - I don't know, I've never run it).
I've set up a few production systems running services that are inside podman containers. I'm lucky enough to be using 98% software that can run inside completely unprivileged containers. For all of these containers, I've run each container under its own user ID. The systemd unit for each, for example, does something along these lines:
[Service] Type=exec User=_cardant Group=_cardant ExecStart=/usr/bin/podman run ...
However, doing things this way is a little messy. For example, if for some reason I want to do something like `podman exec` in a container, I have to `sudo -u _cardant podman exec ...`. `podman ps` will obviously only show me the containers running for the current user. Additionally, any images downloaded from the registry for each service will effectively end up in the home directory of each service user, complicating storage accounting somewhat. The UIDs/GIDs are yet another thing I have to manage, even though they don't have any useful meaning (they don't identify people, they're solely there because the containers have to run as _something_). Containers also leak internal UID/GID values (from the /etc/subuid ranges) into the filesystem, which can complicate things.
Additionally, there are some containers that stubbornly make it awkward to run as a non-root user despite not actually needing privileges. The PostgreSQL image is a good example; you can run it as a non-root user and it'll switch to another UID inside the container and then that UID/GID will end on the database files that are inevitably mounted inside the container. You'll also have to match these unpredictable weird UID/GIDs if you want to supply the container with TLS keys/certs, because postgres will refuse to open them unless the UID/GID matches.
You can't get around this by telling postgres to run as UID 0; it'll refuse, even though UID 0 inside the container isn't UID 0 outside of it when running unprivileged.
I'm running all of these services on systems that have SELinux in enforcing mode. My understanding is that containers will all have the container_t domain and therefore even if they all ran as root, a compromised container would not be able to do any meaningful harm to the system.
I'm therefore not certain if the usual "don't run as root" advice applies as containers don't have the same security properties (especially when combined with SELinux).
I feel like it'd simplify things if I could safely run all of the containers as root. At the very least, I'd be able to predict UID/GID values inside the containers from outside!
I can't get any clear advice on how people are expected to run podman containers in production. All of the various bits of documentation in Linux distributions that talk about running under systemd never bother to talk about UIDs or GIDs. Any documentation on running podman rootless seems to only talk about it in the context of developers running unprivileged containers on their local machines for experimentation/development. If you set up containers via Fedora Server's cockpit UI, you'll get containers running as root everywhere.
What is the best practice here?
Running containers with the least privs possible is always the goal, but it really is up to the application. I wrote a blog a couple of years ago, which I have never published about containers as root but using --userns=auto, which would automatically pick a separate user namespace for each container. I kind of like this idea, but it has one key weekness that the Podman command would be running as root, and there is a potential attack from just pulling the image and writing it to disk. A bug in the podman command or the storage layer, could potentially allow a malformed image to overwrite content on the system outside of containers/storage. The issue with running all of your containers as a non root users, is if every container runs as a non-root user, then the containers would be allowed to attack the user account and every other container, if they were to escape confinement (SELinux).
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 2023-07-28T09:13:14 -0400 Daniel Walsh <dwalsh@redhat.com> wrote:
The issue with running all of your containers as a non root users, is if every container runs as a non-root user, then the containers would be allowed to attack the user account and every other container, if they were to escape confinement (SELinux).
Hello! I read back what I wrote and realized it was a bit ambiguous. To be clear: I'm running each container as a separate non-root user; one user ID per container (not one user ID shared between all containers).
Running containers with the least privs possible is always the goal, but it really is up to the application.
This is where I'm still not entirely clear: Is running a container as root with SELinux and with flags such as --unprivileged=false really more powerful than as a regular user with the same kinds of flags? I haven't heard of anyone escaping SELinux confinement, although I'm guessing it has probably been done. I'd assume a kernel-level exploit would probably be required and, in that case, running under a different UID wouldn't help matters. I've tried setting up machines with all of the containers running as root, and it's certainly a lot less of an administrative burden. I run my own registry so I'm not _too_ concerned about hostile images making it into production containers. I feel like there's a huge hole in the documentation around this subject, and it's really weird that noone appears to be talking about it. Fedora Server runs all containers as root if configured via Cockpit, so presumably someone at least considered the issue? -- Mark Raynsford | https://www.io7m.com
On 2023-07-28T13:27:30 +0000 Mark Raynsford via Podman <podman@lists.podman.io> wrote:
This is where I'm still not entirely clear: Is running a container as root with SELinux and with flags such as --unprivileged=false really
s/unprivileged/privileged -- Mark Raynsford | https://www.io7m.com
On 7/28/23 09:27, Mark Raynsford via Podman wrote:
On 2023-07-28T09:13:14 -0400 Daniel Walsh <dwalsh@redhat.com> wrote:
The issue with running all of your containers as a non root users, is if every container runs as a non-root user, then the containers would be allowed to attack the user account and every other container, if they were to escape confinement (SELinux). Hello!
I read back what I wrote and realized it was a bit ambiguous. To be clear: I'm running each container as a separate non-root user; one user ID per container (not one user ID shared between all containers).
Running containers with the least privs possible is always the goal, but it really is up to the application. This is where I'm still not entirely clear: Is running a container as root with SELinux and with flags such as --unprivileged=false really more powerful than as a regular user with the same kinds of flags? No a regular user is find, and since you are doing each podman run with a different user, you are doing it the most securely in my opinion, with a great deal of extra work, however. I haven't heard of anyone escaping SELinux confinement, although I'm guessing it has probably been done. I'd assume a kernel-level exploit would probably be required and, in that case, running under a different UID wouldn't help matters.
Well, none that I am aware of in the last few years, but I believe in Defense in Depth, I believe each security measure has a chance of failure, but the more you combine the less likely that the entire system is vulnerable.
I've tried setting up machines with all of the containers running as root, and it's certainly a lot less of an administrative burden. I run my own registry so I'm not _too_ concerned about hostile images making it into production containers.
I feel like there's a huge hole in the documentation around this subject, and it's really weird that noone appears to be talking about it. Fedora Server runs all containers as root if configured via Cockpit, so presumably someone at least considered the issue?
Well I do cover a lot of security concerns in my book Podman in Action, Chapters 10 and 11.
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 7/28/23 09:27, Mark Raynsford via Podman wrote:
I read back what I wrote and realized it was a bit ambiguous. To be clear: I'm running each container as a separate non-root user; one user ID per container (not one user ID shared between all containers).
I think your architecture question is fascinating, and certainly a topic I've been interested in. I agree wholeheartedly with Dan's defense-in-depth philosophy, those are wise words to stand upon. This is elegantly reflected by the architecture you've described (multiple host users running rootless containers). Lots and lots of safety layers there. I also hear you loud and clear about this being a PITA to manage. My experience is, it can get even trickier when you bring in something like Ansible to "help" manage it :( Esp., as you also found, the storage ownership mess very quickly gets out of hand. --- Side-note: This is what prompted my earlier question to this list: "Ansible `template` tasks and rootless podman volume content management" --- In all cases, try to step back to first-principals: There's always a push-and-pull between "security" vs "usability" (or "ease of use", "management", "convenience", etc.). Most often, you give up one to get some of the other. Exceptions to this seem to be incredibly rare (SELinux may be one from a user's POV). So for an architecture like you described, perhaps it's simply past the "reasonably practical" point. Maybe it's "okay" to use a single (host) user to run all the containers? You can still leverage pods to segregate networking, cgroups for resources, and volumes to carve up storage (with careful attention to your `:z` and `:Z`s). I would also agree this is slightly "less safe". However, the alternative may (unfortunately) be 100% secure: The system never gets deployed at all, because it's simply too unwieldy for a human (and/or Ansible) to manage. I'm still curious to hear other opinions on this topic though. --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it.
On 7/31/23 15:26, Chris Evich wrote:
On 7/28/23 09:27, Mark Raynsford via Podman wrote:
I read back what I wrote and realized it was a bit ambiguous. To be clear: I'm running each container as a separate non-root user; one user ID per container (not one user ID shared between all containers).
I think your architecture question is fascinating, and certainly a topic I've been interested in. I agree wholeheartedly with Dan's defense-in-depth philosophy, those are wise words to stand upon. This is elegantly reflected by the architecture you've described (multiple host users running rootless containers). Lots and lots of safety layers there.
I also hear you loud and clear about this being a PITA to manage. My experience is, it can get even trickier when you bring in something like Ansible to "help" manage it :( Esp., as you also found, the storage ownership mess very quickly gets out of hand.
--- Side-note: This is what prompted my earlier question to this list: "Ansible `template` tasks and rootless podman volume content management" ---
In all cases, try to step back to first-principals: There's always a push-and-pull between "security" vs "usability" (or "ease of use", "management", "convenience", etc.). Most often, you give up one to get some of the other. Exceptions to this seem to be incredibly rare (SELinux may be one from a user's POV).
So for an architecture like you described, perhaps it's simply past the "reasonably practical" point. Maybe it's "okay" to use a single (host) user to run all the containers? You can still leverage pods to segregate networking, cgroups for resources, and volumes to carve up storage (with careful attention to your `:z` and `:Z`s).
I would also agree this is slightly "less safe". However, the alternative may (unfortunately) be 100% secure: The system never gets deployed at all, because it's simply too unwieldy for a human (and/or Ansible) to manage.
I'm still curious to hear other opinions on this topic though.
--- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
An easier way to do this would be to create a non root user with a huge amount of UIDs And then run each container within that user with Userns=auto. That would get you doing podman pulls without being root, and have all of your containers isolated from each other with user namespaces.
On 8/1/23 05:51, Daniel Walsh wrote:
An easier way to do this would be to create a non root user with a huge amount of UIDs And then run each container within that user with Userns=auto. That would get you doing podman pulls without being root, and have all of your containers isolated from each other with user namespaces.
I'm assuming you mean a huge amount of sub-UIDs (and sub-GIDs). Neat! So with --userns=auto, the container UID/GID assignment is: "nil (Host User UID is not mapped into container.)". Surely that would make life easier and also simplify the ownership of storage items. I'm guessing it wouldn't matter, but would it make any difference to have many entries of smaller ranges (for the host-user), rather than one massive one? Reason I ask is, one entry/range per pod or app would be simpler to manage with tools like Ansible - where it's trivial to manipulate lines in `/etc/sub{u,g}id` - rather than trying to "assign" out of a single huge entry (and ensuring it's always large enough) to the user's pods/apps. In the `newuidmap` I couldn't find mention of any maximum number of ranges. Would it be limited to the max command-line length then? --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it.
On 8/1/23 10:48, Chris Evich wrote:
On 8/1/23 05:51, Daniel Walsh wrote:
An easier way to do this would be to create a non root user with a huge amount of UIDs And then run each container within that user with Userns=auto. That would get you doing podman pulls without being root, and have all of your containers isolated from each other with user namespaces.
I'm assuming you mean a huge amount of sub-UIDs (and sub-GIDs).
Neat! So with --userns=auto, the container UID/GID assignment is: "nil (Host User UID is not mapped into container.)". Surely that would make life easier and also simplify the ownership of storage items.
I'm guessing it wouldn't matter, but would it make any difference to have many entries of smaller ranges (for the host-user), rather than one massive one?
Reason I ask is, one entry/range per pod or app would be simpler to manage with tools like Ansible - where it's trivial to manipulate lines in `/etc/sub{u,g}id` - rather than trying to "assign" out of a single huge entry (and ensuring it's always large enough) to the user's pods/apps.
In the `newuidmap` I couldn't find mention of any maximum number of ranges. Would it be limited to the max command-line length then?
There used to be a max ranges of 5 and Podman might not allow you to specify that many. Anyways why worry about individual ranges. Just define cevich with 100000000 UIDs and go to town.
--- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 8/1/23 17:18, Daniel Walsh wrote:
There used to be a max ranges of 5 and Podman might not allow you to specify that many.
Anyways why worry about individual ranges. Just define cevich with 100000000 UIDs and go to town. The reason to define multiple ranges (one per pod/app) is, it may be easier to codify the individual ranges in orchestration tools like Ansible. Either way you do it, it seems there's a risk of accidentally use overlapping IDs across pods/apps. IIUC, having multiple entries defined would offer some protection against that.
It's okay there's some entry limit, I s'pose it's even helpful. If you're managing "too many" separate ranges, it's probably a sign you should at least be using kube YAML if not minikube or full-blown kube rather than trying to stuff everything into generic tooling like Ansible. Or (shock) trying to manage a complex setup by hand. --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it.
On 8/2/23 10:50, Chris Evich wrote:
On 8/1/23 17:18, Daniel Walsh wrote:
There used to be a max ranges of 5 and Podman might not allow you to specify that many.
Anyways why worry about individual ranges. Just define cevich with 100000000 UIDs and go to town. The reason to define multiple ranges (one per pod/app) is, it may be easier to codify the individual ranges in orchestration tools like Ansible. Either way you do it, it seems there's a risk of accidentally use overlapping IDs across pods/apps. IIUC, having multiple entries defined would offer some protection against that.
It's okay there's some entry limit, I s'pose it's even helpful. If you're managing "too many" separate ranges, it's probably a sign you should at least be using kube YAML if not minikube or full-blown kube rather than trying to stuff everything into generic tooling like Ansible. Or (shock) trying to manage a complex setup by hand.
Sure, but Podman has smarts about currently used ranges and can use minimal ranges based on the image in use. If you use something like Ansible then it may or may not know about existing containers, and ranges in use, as well as would need to use 65k per container.
--- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 8/3/23 06:20, Daniel Walsh wrote:
Sure, but Podman has smarts about currently used ranges and can use minimal ranges based on the image in use. If you use something like Ansible then it may or may not know about existing containers, and ranges in use, as well as would need to use 65k per container.
Oooooohhhh, that's the part I was missing. I thought podman just blindly used the range. Thanks! And "yes", we do need a blog about this, it feels like a useful thing many people are probably not aware of.
On 8/3/23 15:26, Chris Evich wrote:
On 8/3/23 06:20, Daniel Walsh wrote:
Sure, but Podman has smarts about currently used ranges and can use minimal ranges based on the image in use. If you use something like Ansible then it may or may not know about existing containers, and ranges in use, as well as would need to use 65k per container.
Oooooohhhh, that's the part I was missing. I thought podman just blindly used the range. Thanks! And "yes", we do need a blog about this, it feels like a useful thing many people are probably not aware of. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Rootful Podman looks for the user "containers" in /etc/subuid and /etc/subgid files and then divides the range of UIDs/GIDs in defined, containers/storage records the used UID ranges. With rootless, we sub-divide the users range.
On 8/4/23 08:17, Daniel Walsh wrote:
Rootful Podman looks for the user "containers" in /etc/subuid and /etc/subgid files and then divides the range of UIDs/GIDs in defined, containers/storage records the used UID ranges.
With rootless, we sub-divide the users range.
Right, and I see `podman pod create` also supports passing `--userns=auto`, so then all containers in that pod will have the same user namespace "view" for shared volumes. But now I'm questioning if podman really is "smart" enough to allocate the same rootless "sub-range" (from a single big entry) every time. Including across create/remove/create cycles for pods, containers, and volumes. Getting back to the suggestion for Mark's architecture (one user, managing tens of pods/apps, while maintaining "defense in depth" as much as possible). It would be really, really, really important for the pods and/or containers to always grab the same "sub-range" when using `--userns=auto`. Even if the pod or container is removed and re-created (for example, maybe some option needs changing). Though maybe (again) I'm misunderstanding how `--userns=auto` is suppose to work. It sounds like maybe the "sub-range" allocation is somehow persisted along with the shared volume data? So preserving the volume metadata then becomes incredibly important (unless you enjoy lots of manual-chown-labor). --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it.
On 8/4/23 11:16, Chris Evich wrote:
On 8/4/23 08:17, Daniel Walsh wrote:
Rootful Podman looks for the user "containers" in /etc/subuid and /etc/subgid files and then divides the range of UIDs/GIDs in defined, containers/storage records the used UID ranges.
With rootless, we sub-divide the users range.
Right, and I see `podman pod create` also supports passing `--userns=auto`, so then all containers in that pod will have the same user namespace "view" for shared volumes.
But now I'm questioning if podman really is "smart" enough to allocate the same rootless "sub-range" (from a single big entry) every time. Including across create/remove/create cycles for pods, containers, and volumes.
Getting back to the suggestion for Mark's architecture (one user, managing tens of pods/apps, while maintaining "defense in depth" as much as possible). It would be really, really, really important for the pods and/or containers to always grab the same "sub-range" when using `--userns=auto`. Even if the pod or container is removed and re-created (for example, maybe some option needs changing).
There is no guarantee that if you recreate the Pod/container that it will get the same user namespace. If you want to guarantee this, you would need to allocate the USERNS yourself. And isolate the range from the range that podman users for --userns=auto, if you are using these containers simultaneous. Podman is not that sophisticated when it picks its range. I believe it just looks sequentially through the list of unusers UIDs/GIDs and picks the next available range.
Though maybe (again) I'm misunderstanding how `--userns=auto` is suppose to work. It sounds like maybe the "sub-range" allocation is somehow persisted along with the shared volume data? So preserving the volume metadata then becomes incredibly important (unless you enjoy lots of manual-chown-labor).
--- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
participants (3)
-
Chris Evich -
Daniel Walsh -
Mark Raynsford