Andrew,      Thank you for this extensive proposal. Some thoughts inline... On Mon, Dec 14, 2020 at 9:26 AM Andrew G. Dunn <agd(a)epiphyte.org <mailto:agd@epiphyte.org>> wrote: Greetings, thanks for this awesome tool and growing community! We've been deploying podman, using systemd with podman directly (instances, and pods), or podman kube. We've been internally talking about a couple topics related to hardening and have been neglecting to find a place to initaite discussion, as it's a bit meta in nature. The mailing list looks like the right place! Everything proposed here comes down to personal preference, but the reason we wanted to share our discussion with the community is to explore what the sane defaults should be for users of podman. # rootless, rootless as non-root Brian Smith via this video [0] uses the terminology "rootles-podman-as-non- root". We understand that (likely niavely) to be "shifting twice":  - once from root on metal, to user on metal  - once more from root and user coliding to root in the container being remapped off user (using subuid/subgid) For those reading, check out Brian's videos, they are amazing. He allowed me to review them before he published them and they are great. Another follow up piece of content that might interesting you is this blog: https://www.redhat.com/en/blog/understanding-root-inside-and-outside-cont... <https://www.redhat.com/en/blog/understanding-root-inside-and-outside-cont... I "think" what you are getting at is that there are root inside and outside the user. It's best not to use root anywhere if possible - run podman as a user (ex fatherlinux), and run the process in the container as a user (ex sync). When you do this, you limit the damage that the podman user can do, as well as what a hacker can do. You are protecting the host from the user, and the containers from each other. Discussion here [1] shows that if you were to attempt to user systemd-sysuers or systemd-tmpfiles to package something (using podman) you'd not be able to set up the subuid/subgid mappings. Poettering goes on to point out that subuid/subgid as implemented has flaws [2]. We've been deploying as "rootless-podman-as-non-root" but have recently been considering removing the subuid/subgid configurations as the podman instances for us are already running under "system" users (e.g. `/sbin/nologin`). We can't seem to grasp the specific advantage when doing a "systems" deplyoment, and there is a distinct disadvantage when having to deal with file permissions that are uid/gid shifted (pressing one to use more permissive permissions than what would typically be necessary). We realize that "rootless-podman-as-non-root" is valuable for things like toolbox [3], where the example would be a non-root user wanting to run a container mapped once again off their namespaces (e.g. a browser or something of high risk). So, I can't comment too much on this, as I am not familiar with this systemd feature. That said, I know that the identity management team has been looking at centralized solution to manage subuid/subgids across clusters of hosts as well as different software on those hosts. # systemd unit hardening systemd itself has a _lot_ of hardening features available [4], one can make a unit wrapping podman and then examine it via `systemd-analyze security unit.service`. As podman has a `podman generate systemd` it'd be extremely interesting to have some discussion on how these features of systemd could be enumerated/used by default. This sounds quite interesting, and something I would be interested in as the product manager for Podman in RHEL. These are the kinds of Feature ideas that I'd love to discuss. # seccomp/eBPF/selinux There is already some documentation on generating seccomp profiles [5], as well as udica [6]. These seem to be very powerful tools to create instance specific isolation for deployments. We're very interested in these, but we're wondering how to practically apply these techniques for something that is a complex monolithic ontainer (e.g. gitlab). So far, at least in RHEL, we've provided these tools to users but pretty much left them to be configured for themselves. The main problem with seccomp is, the default rules have to either 1. be so loose as to not be very useful or 2. so tight that people will shut it off. Likely both Secomp and udica are best customized on a workload by workload basis, and the user of Podman is likely the SME for that workload more than the Podman team or in my case, RHEL. # Questions Does the podman community have a line to the systemd community to talk about leveraging subuid/subgid, is there a more systemd focused formalism for accomplishing this shift? For sure. What does the "shifting twice" accomplish for you when deploying a system style service? (as opposed to the other hardening options mentioned below) Will the podman community consider systemd to be a "first class" deplyoment, and split that style of deployments into "systems" and "users" where on "systems" we can go far deeper into the expected defaults/patterns (toolbox handling the "users" use case well)? Would someone working on selinux/udica consider a complex container use case (e.g. keycloak from RedHat itself, or gitlab as an upstream partner) for the generation of profiles? What are the patterns with generating profiles with udica? Would it be most reasonable to generate these profiles on a test system, generating a profile each time you instance the container, then deploying those profiles to production? Tom, I think this might be a great subject for the next Podman Community meeting? What do others think? It feels sufficiently complex enough to requires some synchronous communication? We're mainly just wanting to hear from folks who are deploying podman as to how they are using these tools, and what other tooling/techniques may be out there that we could be looking at. Thanks for considering the inquiry! [0]: http://www.youtube.com/watch?v=ZgXpWKgQclc&t=7m40s <http://www.youtube.com/watch?v=ZgXpWKgQclc&t=7m40s> [1]: https://github.com/systemd/systemd/issues/13717#issuecomment-711167021 <https://github.com/systemd/systemd/issues/13717#issuecomment-711167021> [2]: https://github.com/systemd/systemd/issues/13717#issuecomment-539476282 <https://github.com/systemd/systemd/issues/13717#issuecomment-539476282> [3]: https://github.com/containers/toolbox <https://github.com/containers/toolbox> [4]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html <https://www.freedesktop.org/software/systemd/man/systemd.exec.html> [5]: https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html <https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html> [6]: https://github.com/containers/udica <https://github.com/containers/udica> _______________________________________________ Podman mailing list -- podman(a)lists.podman.io <mailto:podman@lists.podman.io> To unsubscribe send an email to podman-leave(a)lists.podman.io <mailto:podman-leave@lists.podman.io> -- -- The Delicate Art of Product Management with Open Source:http://crunchtools.com/open-source-in-business-2020/ <http://crunchtools.com/open-source-in-business-2020/> -- Scott McCarty Product Management - Containers, Red Hat Enterprise Linux & OpenShift Email: smccarty(a)redhat.com <mailto:smccarty@redhat.com> Phone: 312-660-3535 Cell: 330-807-1043 Web: http://crunchtools.com <http://crunchtools.com> _______________________________________________ Podman mailing list -- podman(a)lists.podman.io To unsubscribe send an email to podman-leave(a)lists.podman.io

My inquiry was intended more along the lines of (for use of subuid/subgid specifically): - if you consider that subuid/subgid is "contentious" in its implementation + further, if you consider a purpose based user account for a service Do others think it an acceptable pattern to avoid rootless-podman-as-non-root for "systems" style deployment (e.g. the purpose based user account is `/sbin/nologin` and has restricted permissions). Again, realizing that rootless-podman-as-non-root is very useful for the toolbox use case.

I'd then asked about other hardening in cohort with the "fork in the road" of rootless-podman-as-non-root. It's a sort of value judgment discussion of: maybe you run with `--userns=keep-id` because you've layered in security profiling with udica. Mainly just wanting to start a mailing list dialog about how others are thinking of their deployments. On Tue, 2020-12-15 at 15:15 -0500, Daniel Walsh wrote: > Andrew I had a hard time following what you are asking, but I think you just > want to run a container with the UID of the user who launched it. > > podman run --userns=keep-id ... > > Does this for you > > /etc/subuid and /etc/subgid, provide a mechanism for users to have more then > one UID, and often to emulate root with namespaced capabilities. > > Most OCI container images from container registries have more then one > UID/GID within them. Often they have root and not root users.  Pulling one > of these images is not allowed by default unless the podman user has more > then one UID.  We did add a feature to storage.conf called > ignore_chown_errors that allows you to pull and install an image all as your > UID within your system, and then you can run it > with your uid, using the --userns=keep-id flag. > > > On 12/15/20 13:00, Scott McCarty wrote: > > Andrew, >      Thank you for this extensive proposal. Some thoughts inline... > > On Mon, Dec 14, 2020 at 9:26 AM Andrew G. Dunn <agd(a)epiphyte.org&gt; wrote: > >> Greetings, thanks for this awesome tool and growing community! >> >>  We've been deploying podman, using systemd with podman directly >> (instances, >>  and pods), or podman kube. We've been internally talking about a couple >> topics >>  related to hardening and have been neglecting to find a place to initaite >>  discussion, as it's a bit meta in nature. The mailing list looks like the >>  right place! >> >> >>  Everything proposed here comes down to personal preference, but the >> reason we >>  wanted to share our discussion with the community is to explore what the >> sane >>  defaults should be for users of podman. >> >>  # rootless, rootless as non-root >> >>  Brian Smith via this video [0] uses the terminology "rootles-podman-as- >> non- >>  root". We understand that (likely niavely) to be "shifting twice": >> >>   - once from root on metal, to user on metal >>   - once more from root and user coliding to root in the container being >>  remapped off user (using subuid/subgid) >> > For those reading, check out Brian's videos, they are amazing. He allowed me > to review them before he published them and they are great. Another follow > up piece of content that might interesting you is this blog: > > https://www.redhat.com/en/blog/understanding-root-inside-and-outside-cont... > > I "think" what you are getting at is that there are root inside and outside > the user. It's best not to use root anywhere if possible - run podman as a > user (ex fatherlinux), and run the process in the container as a user (ex > sync). When you do this, you limit the damage that the podman user can do, > as well as what a hacker can do. You are protecting the host from the user, > and the containers from each other. > > > >> >>  Discussion here [1] shows that if you were to attempt to user systemd- >> sysuers >>  or systemd-tmpfiles to package something (using podman) you'd not be able >> to >>  set up the subuid/subgid mappings. Poettering goes on to point out that >>  subuid/subgid as implemented has flaws [2]. >> >>  We've been deploying as "rootless-podman-as-non-root" but have recently >> been >>  considering removing the subuid/subgid configurations as the podman >> instances >>  for us are already running under "system" users (e.g. `/sbin/nologin`). >> We >>  can't seem to grasp the specific advantage when doing a "systems" >> deplyoment, >>  and there is a distinct disadvantage when having to deal with file >> permissions >>  that are uid/gid shifted (pressing one to use more permissive permissions >> than >>  what would typically be necessary). >> >>  We realize that "rootless-podman-as-non-root" is valuable for things like >>  toolbox [3], where the example would be a non-root user wanting to run a >>  container mapped once again off their namespaces (e.g. a browser or >> something >>  of high risk). >> > So, I can't comment too much on this, as I am not familiar with this systemd > feature. That said, I know that the identity management team has been > looking at centralized solution to manage subuid/subgids across clusters of > hosts as well as different software on those hosts. > > > >> >>  # systemd unit hardening >> >>  systemd itself has a _lot_ of hardening features available [4], one can >> make a >>  unit wrapping podman and then examine it via `systemd-analyze security >>  unit.service`. As podman has a `podman generate systemd` it'd be >> extremely >>  interesting to have some discussion on how these features of systemd >> could be >>  enumerated/used by default. >> > This sounds quite interesting, and something I would be interested in as the > product manager for Podman in RHEL. These are the kinds of Feature ideas > that I'd love to discuss. > > > >> >>  # seccomp/eBPF/selinux >> >>  There is already some documentation on generating seccomp profiles [5], >> as >>  well as udica [6]. These seem to be very powerful tools to create >> instance >>  specific isolation for deployments. We're very interested in these, but >> we're >>  wondering how to practically apply these techniques for something that is >> a >>  complex monolithic ontainer (e.g. gitlab). >> > So far, at least in RHEL, we've provided these tools to users but pretty > much left them to be configured for themselves. The main problem with > seccomp is, the default rules have to either 1. be so loose as to not be > very useful or 2. so tight that people will shut it off. > > Likely both Secomp and udica are best customized on a workload by workload > basis, and the user of Podman is likely the SME for that workload more than > the Podman team or in my case, RHEL. > > > >> >>  # Questions >> >>  Does the podman community have a line to the systemd community to talk >> about >>  leveraging subuid/subgid, is there a more systemd focused formalism for >>  accomplishing this shift? >> > For sure. > > > >> >>  What does the "shifting twice" accomplish for you when deploying a system >>  style service? (as opposed to the other hardening options mentioned >> below) >> >>  Will the podman community consider systemd to be a "first class" >> deplyoment, >>  and split that style of deployments into "systems" and "users" where on >>  "systems" we can go far deeper into the expected defaults/patterns >> (toolbox >>  handling the "users" use case well)? >> >>  Would someone working on selinux/udica consider a complex container use >> case >>  (e.g. keycloak from RedHat itself, or gitlab as an upstream partner) for >> the >>  generation of profiles? >> >>  What are the patterns with generating profiles with udica? Would it be >> most >>  reasonable to generate these profiles on a test system, generating a >> profile >>  each time you instance the container, then deploying those profiles to >>  production? >> > Tom, I think this might be a great subject for the next Podman Community > meeting? What do others think? It feels sufficiently complex enough to > requires some synchronous communication? > > > >> >>  We're mainly just wanting to hear from folks who are deploying podman as >> to >>  how they are using these tools, and what other tooling/techniques may be >> out >>  there that we could be looking at. Thanks for considering the inquiry! >> >>  [0]: http://www.youtube.com/watch?v=ZgXpWKgQclc&t=7m40s >>  [1]: >> https://github.com/systemd/systemd/issues/13717#issuecomment-711167021 >>  [2]: >> https://github.com/systemd/systemd/issues/13717#issuecomment-539476282 >>  [3]: https://github.com/containers/toolbox >>  [4]: https://www.freedesktop.org/software/systemd/man/systemd.exec.html >>  [5]: https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html >>  [6]: https://github.com/containers/udica >> >> >> >> >> >> >> >> >> >> >> >> >> >>  _______________________________________________ >>  Podman mailing list -- podman(a)lists.podman.io >>  To unsubscribe send an email to podman-leave(a)lists.podman.io >> > > >  -- > -- > The Delicate Art of Product Management with Open Source: > http://crunchtools.com/open-source-in-business-2020/ > -- > Scott McCarty > Product Management - Containers, Red Hat Enterprise Linux & OpenShift > Email: smccarty(a)redhat.com > Phone: 312-660-3535 > Cell: 330-807-1043 > Web: http://crunchtools.com > > > _______________________________________________ > Podman mailing list -- podman(a)lists.podman.io > To unsubscribe send an email to podman-leave(a)lists.podman.io > > _______________________________________________ > Podman mailing list -- podman(a)lists.podman.io > To unsubscribe send an email to podman-leave(a)lists.podman.io _______________________________________________ Podman mailing list -- podman(a)lists.podman.io To unsubscribe send an email to podman-leave(a)lists.podman.io