[Podman] Re: Podman Deployment Hardening Patterns

Andrew,      Thank you for this extensive proposal. Some thoughts inline... On Mon, Dec 14, 2020 at 9:26 AM Andrew G. Dunn <agd(a)epiphyte.org <mailto:agd@epiphyte.org>> wrote: Greetings, thanks for this awesome tool and growing community! We've been deploying podman, using systemd with podman directly (instances, and pods), or podman kube. We've been internally talking about a couple topics related to hardening and have been neglecting to find a place to initaite discussion, as it's a bit meta in nature. The mailing list looks like the right place! Everything proposed here comes down to personal preference, but the reason we wanted to share our discussion with the community is to explore what the sane defaults should be for users of podman. # rootless, rootless as non-root Brian Smith via this video [0] uses the terminology "rootles-podman-as-non- root". We understand that (likely niavely) to be "shifting twice":  - once from root on metal, to user on metal  - once more from root and user coliding to root in the container being remapped off user (using subuid/subgid) For those reading, check out Brian's videos, they are amazing. He allowed me to review them before he published them and they are great. Another follow up piece of content that might interesting you is this blog: https://www.redhat.com/en/blog/understanding-root-inside-and-outside-cont... <https://www.redhat.com/en/blog/understanding-root-inside-and-outside-cont... I "think" what you are getting at is that there are root inside and outside the user. It's best not to use root anywhere if possible - run podman as a user (ex fatherlinux), and run the process in the container as a user (ex sync). When you do this, you limit the damage that the podman user can do, as well as what a hacker can do. You are protecting the host from the user, and the containers from each other. Discussion here [1] shows that if you were to attempt to user systemd-sysuers or systemd-tmpfiles to package something (using podman) you'd not be able to set up the subuid/subgid mappings. Poettering goes on to point out that subuid/subgid as implemented has flaws [2]. We've been deploying as "rootless-podman-as-non-root" but have recently been considering removing the subuid/subgid configurations as the podman instances for us are already running under "system" users (e.g. `/sbin/nologin`). We can't seem to grasp the specific advantage when doing a "systems" deplyoment, and there is a distinct disadvantage when having to deal with file permissions that are uid/gid shifted (pressing one to use more permissive permissions than what would typically be necessary). We realize that "rootless-podman-as-non-root" is valuable for things like toolbox [3], where the example would be a non-root user wanting to run a container mapped once again off their namespaces (e.g. a browser or something of high risk). So, I can't comment too much on this, as I am not familiar with this systemd feature. That said, I know that the identity management team has been looking at centralized solution to manage subuid/subgids across clusters of hosts as well as different software on those hosts. # systemd unit hardening systemd itself has a _lot_ of hardening features available [4], one can make a unit wrapping podman and then examine it via `systemd-analyze security unit.service`. As podman has a `podman generate systemd` it'd be extremely interesting to have some discussion on how these features of systemd could be enumerated/used by default. This sounds quite interesting, and something I would be interested in as the product manager for Podman in RHEL. These are the kinds of Feature ideas that I'd love to discuss. # seccomp/eBPF/selinux There is already some documentation on generating seccomp profiles [5], as well as udica [6]. These seem to be very powerful tools to create instance specific isolation for deployments. We're very interested in these, but we're wondering how to practically apply these techniques for something that is a complex monolithic ontainer (e.g. gitlab). So far, at least in RHEL, we've provided these tools to users but pretty much left them to be configured for themselves. The main problem with seccomp is, the default rules have to either 1. be so loose as to not be very useful or 2. so tight that people will shut it off. Likely both Secomp and udica are best customized on a workload by workload basis, and the user of Podman is likely the SME for that workload more than the Podman team or in my case, RHEL. # Questions Does the podman community have a line to the systemd community to talk about leveraging subuid/subgid, is there a more systemd focused formalism for accomplishing this shift? For sure. What does the "shifting twice" accomplish for you when deploying a system style service? (as opposed to the other hardening options mentioned below) Will the podman community consider systemd to be a "first class" deplyoment, and split that style of deployments into "systems" and "users" where on "systems" we can go far deeper into the expected defaults/patterns (toolbox handling the "users" use case well)? Would someone working on selinux/udica consider a complex container use case (e.g. keycloak from RedHat itself, or gitlab as an upstream partner) for the generation of profiles? What are the patterns with generating profiles with udica? Would it be most reasonable to generate these profiles on a test system, generating a profile each time you instance the container, then deploying those profiles to production? Tom, I think this might be a great subject for the next Podman Community meeting? What do others think? It feels sufficiently complex enough to requires some synchronous communication? We're mainly just wanting to hear from folks who are deploying podman as to how they are using these tools, and what other tooling/techniques may be out there that we could be looking at. Thanks for considering the inquiry! [0]: http://www.youtube.com/watch?v=ZgXpWKgQclc&t=7m40s <http://www.youtube.com/watch?v=ZgXpWKgQclc&t=7m40s> [1]: https://github.com/systemd/systemd/issues/13717#issuecomment-711167021 <https://github.com/systemd/systemd/issues/13717#issuecomment-711167021> [2]: https://github.com/systemd/systemd/issues/13717#issuecomment-539476282 <https://github.com/systemd/systemd/issues/13717#issuecomment-539476282> [3]: https://github.com/containers/toolbox <https://github.com/containers/toolbox> [4]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html <https://www.freedesktop.org/software/systemd/man/systemd.exec.html> [5]: https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html <https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html> [6]: https://github.com/containers/udica <https://github.com/containers/udica> _______________________________________________ Podman mailing list -- podman(a)lists.podman.io <mailto:podman@lists.podman.io> To unsubscribe send an email to podman-leave(a)lists.podman.io <mailto:podman-leave@lists.podman.io> -- -- The Delicate Art of Product Management with Open Source:http://crunchtools.com/open-source-in-business-2020/ <http://crunchtools.com/open-source-in-business-2020/> -- Scott McCarty Product Management - Containers, Red Hat Enterprise Linux & OpenShift Email: smccarty(a)redhat.com <mailto:smccarty@redhat.com> Phone: 312-660-3535 Cell: 330-807-1043 Web: http://crunchtools.com <http://crunchtools.com> _______________________________________________ Podman mailing list -- podman(a)lists.podman.io To unsubscribe send an email to podman-leave(a)lists.podman.io