On 9/8/20 1:07 AM, Chintan from Rebhu wrote:

There were two recommended actions

1. |Add `--security-opt label=disable` while starting the container| 2. |Add `--group-add tty` while starting the container|

|The first one worked for me while running Envoyproxy 1.15.0. I would like to understand the security implications of this flag.|

Hello again, I cannot speak to the implications for Envoyproxy, but can offer some guidance on the specific questions. 1. This disables SELinux labeling on the host-side, for and resources container tries to use (like volume-mounts). It's generally not something we would recommend doing unless you have a very specific and well understood reason. For example, I use this option when I need to volume-mount a clone of the podman source repository from my home-directory, into a container to run some simple checks. Ordinarily this would fail unless I used the :z or :Z options on the mount. However, these are highly undesireable, since it will complicate future host-side manipulations of the files. Disabling SELinux labeling in this specific case isn't a problem, since the container is short-lived, and I can rely on version-control to track any unexpected changes. 2. This one is a lot more tricky to answer, since the issue you referenced has been fixed in crun. I think the answer really depends much more on exactly how your container is being run (and what the contained process(es) are attempting to do specifically). Based on the context of your other mail, it sounds like you're trying to execute some service in a rootless container, and it's attempting to drop privileges through su or runuser. If so, the answer is simple: don't do that :D If you're running rootless, the contained-user appears to contained-processes as root. However this is a lie and a trick. It's really just UID/GID mapped back to your user/group on the host. It doesn't actually have any consequential access beyond the user account, and especially when SELinux labeling is being enforced. If my assumptions are correct, what you likely need to do is NOT use `--security-opt label=disable'. Instead, volume mount from a dedicated directory on the host (outside of $HOME), and use the :Z or :z volume-mount options. Additionally, inside the rootless container, just run your service as "root". Forget about also using a container-user account through systemd, sudo, su, runuser, etc. It's not buying you any additional protections, just drastically and needlessly complicating the environment. Hopefully that helps. [*] This may not be entirely true, would really need someone more SELinux-knowledgeable and more complete runtime details to answer this fully. -- Chris Evich, RHCA III Senior Quality Assurance Engineer If it ain't broke, yain't tryin' hard nough.

Hello, everyone!! Thank you Chris, Matt, and Daniel for helping me understand the queries in my mind. Here is a pastebin of Access Vector Cache(AVC) output https://pastebin.com/c1w8AnA7 The contents were fetched using `|ausearch -m avc --start recent|`. On 08/09/20 6:32 pm, Chris Evich wrote:

On 9/8/20 12:48 AM, Chintan from Rebhu wrote:

...

The same error is not present when I switch from v1.15.0 to v1.14.4 of Envoyproxy.

I am out of my wits about this. Please tell me how I should find a solution.

We only use Podman in our infrastructure.

Hi,

First off, I don't mean to dismiss the scale of your frustrations or challenges here.  However, this list may not be the appropriate place for seeking infrastructure support for third-party interoperability (open-source or closed).

We're really narrowly focused on podman and it's intimately related container technologies, and improving the state thereof. If you're looking for end-to-end debugging and help, then a commercial support option would probably be best.

If it was something non-trivial then I would happily seek commercial support. I am just trying to run a command that works with Docker on Debian(checked a few moments ago) but fails on Podman. Podman is marketed as a drop-in replacement for Docker. So, one would expect that commands work out of box. And most generic applications do run fine. But there are a significant number of applications that need certain modifications to get things working with Podman. One grows appreciative of the finer details as they use Podman. But these finer details are poorly documented and highly fragmented all over the internet. And if applications keep not working and stay without resolution or without documentation then that will be a deterrent for Podman's wider community adoption. What do you suggest the community do to get applications running through Podman?

OTOH, I see you're follow-up mail, which does have some podman-specific questions we can probably help with.  I just want to be clear up-front, that this list is not an official/commercial means for overall/general system support.

In the first e-mail, I failed to mention that I ruled out issues in the containerfile provided by the Envoyproxy maintainers. There is no mention of a similar issue on their communication channels like Slack, GitHub issues and Google group. So, it was only logical to conclude that the issue is with Podman. -- Chintan Mishra

Hello!! On 08/09/20 11:28 pm, Daniel Walsh wrote:

What command are you using to launch the Envoyproxy container?

$ sudo podman run -v ./envoy.yaml:/etc/envoy/envoy.yaml --network test --name test-lz -d -p 10000:10000 envoyproxy/envoy:v1.15.0

The AVC's indicate the container is attempting to write to a fifo_file created by podman

# podman run --privileged -v /usr/bin/find:/usr/bin/find fedora find / -type p The issue is I am not getting any information about what fifo file is being blocked?

There is no output returned by the above command in rootful mode. Running the same command in rootless mode returns find: '/proc/tty/driver': Permission denied

Is this running rootful or rootless?

The AVC output in the pastebin was from the container running in rootful mode. I have tried running the container in both rootful and rootless mode. -- Chintan Mishra

On 9/8/20 13:06, Chintan from Rebhu wrote:

...

Hello, everyone!!

Thank you Chris, Matt, and Daniel for helping me understand the queries in my mind.

Here is a pastebin of Access Vector Cache(AVC) output https://pastebin.com/c1w8AnA7

The contents were fetched using `|ausearch -m avc --start recent|`.

On 08/09/20 6:32 pm, Chris Evich wrote:

...

On 9/8/20 12:48 AM, Chintan from Rebhu wrote:

...

The same error is not present when I switch from v1.15.0 to v1.14.4 of Envoyproxy.

I am out of my wits about this. Please tell me how I should find a solution.

We only use Podman in our infrastructure.

Hi,

First off, I don't mean to dismiss the scale of your frustrations or challenges here.  However, this list may not be the appropriate place for seeking infrastructure support for third-party interoperability (open-source or closed).

We're really narrowly focused on podman and it's intimately related container technologies, and improving the state thereof.  If you're looking for end-to-end debugging and help, then a commercial support option would probably be best.

If it was something non-trivial then I would happily seek commercial support. I am just trying to run a command that works with Docker on Debian(checked a few moments ago) but fails on Podman. Podman is marketed as a drop-in replacement for Docker. So, one would expect that commands work out of box. And most generic applications do run fine. But there are a significant number of applications that need certain modifications to get things working with Podman. One grows appreciative of the finer details as they use Podman. But these finer details are poorly documented and highly fragmented all over the internet. And if applications keep not working and stay without resolution or without documentation then that will be a deterrent for Podman's wider community adoption.

What do you suggest the community do to get applications running through Podman?

...

OTOH, I see you're follow-up mail, which does have some podman-specific questions we can probably help with.  I just want to be clear up-front, that this list is not an official/commercial means for overall/general system support.

In the first e-mail, I failed to mention that I ruled out issues in the containerfile provided by the Envoyproxy maintainers. There is no mention of a similar issue on their communication channels like Slack, GitHub issues and Google group. So, it was only logical to conclude that the issue is with Podman.

-- Chintan Mishra

_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io

_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io

Hello!! On 09/09/20 9:56 pm, apsoul@hotmail.com wrote:

I've had success using udica to craft the selinux policy that can then be applied to the container using the --security-opt flag.

See this blog post: https://fedoramagazine.org/use-udica-to-build-selinux-policy-for-containers/ Thank you apsoul for sharing a great resource.

The only place I've noticed udica falling short is running a pod of multiple containers that are all communicating with each other. In this instance I had to start the pod and monitor audit.log for avc denials and then manually update the udica generated .cil file.

It appears that this tool works with running containers. The Envoyproxy container gets killed even before the application within can start running. -- Chintan Mishra

_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io

Hello, apsoul!! I followed through with your recommendations and I could get the container to run. Thank you for taking the time to find a solution and helping me out. In my case, I had to append only one policy. I am highlighting the steps below for rootful container. On 11/09/20 3:16 am, apsoul@hotmail.com wrote:

Hi Chintan,

I got the envoy container running, but used the example config on envoy's Getting Started page: https://www.envoyproxy.io/docs/envoy/latest/start/start

Steps I followed:

1) Created a starter container 2) Created an selinux policy using udica 3) Started the container and monitored /var/log/audit/audit.log for denials 4) Updated the udica generated *.cil file everytime the container failed to start, until it finally started. This took 3 attempts, starting on the 4th.

Granted, not the most user friendly way of getting the container running, but I think contained from a security standpoint.

1) Starter Container

``` podman create --name envoy \ --label envoy=envoy \ -p 10000:10000 \ -v ${PWD}/envoy_example.yaml:/etc/envoy/envoy.yaml \ envoyproxy/envoy:v1.15.0 ```

2) Create Policy & remove starter container

``` podman inspect envoy > envoy_container.json udica -j envoy_container.json envoy semodule -i envoy.cil /usr/share/udica/templates/{base_container.cil,net_container.cil} podman rm envoy ```

3) Create Final Container, attempt to start and monitor audit.log

``` podman create --name envoy \ --security-opt label=type:envoy.process \ --label envoy=envoy \ -p 10000:10000 \ -v ${PWD}/envoy_example.yaml:/etc/envoy/envoy.yaml:Z \ envoyproxy/envoy:v1.15.0

podman start envoy ``` I followed all the steps till here.

Each time the container failed to start it would log a denial. After each denial I updated the *.cil policy file to allow the denial, and re-applied the policy with: `semodule -i envoy.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}` After running the container I started looking for AVC denials. The following denial didn't occur in my case. This could be due to rootful environment.

- Startup 1 ``` type=AVC msg=audit(1599771689.994:11166): avc: denied { read } for \ pid=1194200 comm="docker-entrypoi" path="/lib/x86_64-linux-gnu/libc-2.27.so" \ dev="dm-9" ino=71349652 scontext=system_u:system_r:envoy.process:s0:c76,c889 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0 ```

- Results in allow rule ``` (allow process default_t ( file ( read ))) ```

The following denial was there in the audit logs. ``` type=AVC msg=audit(1600000744.700:5261): avc:  denied  { setattr } for  pid=15927 comm="chown" name="" dev="pipefs" ino=96155 scontext=system_u:system_r:envoy.process:s0:c107,c940 tcontext=unconfined_u:system_r:container_runtime_t:s0 tclass=fifo_file permissive=0 type=AVC msg=audit(1600000744.700:5262): avc:  denied  { setattr } for  pid=15927 comm="chown" name="" dev="pipefs" ino=96156 scontext=system_u:system_r:envoy.process:s0:c107,c940 tcontext=unconfined_u:system_r:container_runtime_t:s0 tclass=fifo_file permissive=0 ``` In order to confirm that this is the only denial, container were started multiple times. And the two audit logs above were the only denials. These are similar to the logs below.

- Startup 2 ``` type=AVC msg=audit(1599772073.397:11205): avc: denied { setattr } for \ pid=1218170 comm="chown" name="" dev="pipefs" \ ino=6650489 scontext=system_u:system_r:envoy.process:s0:c76,c889 tcontext=unconfined_u:system_r:container_runtime_t:s0 tclass=fifo_file permissive=0 ```

- Results in allow rule ``` (allow process container_runtime_t ( fifo_file ( setattr ))) ```

Adding the policy rule above allowed the container to run flawlessly.

- Startup 3 ``` type=AVC msg=audit(1599772272.855:11237): avc: denied { name_bind } for \ pid=1232662 comm="envoy" src=9901 \ scontext=system_u:system_r:envoy.process:s0:c76,c889 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0 ```

- Results in allow rule ``` (allow process unreserved_port_t ( tcp_socket ( name_bind ))) ```

The final *.cil file looked like this:

```

The policy above was already present in the *.cil file generated in my environment. This could be due to rootful environment.

(block envoy (blockinherit container) (blockinherit restricted_net_container) (allow process process ( capability ( audit_write chown dac_override fowner fsetid kill mknod net_bind_service net_raw setfcap setgid setpcap setuid sys_chroot )))

(allow process usr_t ( dir ( open read getattr lock search ioctl add_name remove_name write ))) (allow process usr_t ( file ( getattr read write append ioctl lock map open create ))) (allow process usr_t ( sock_file ( getattr read write append open ))) (allow process default_t ( file ( read ))) (allow process container_runtime_t ( fifo_file ( setattr ))) (allow process unreserved_port_t ( tcp_socket ( name_bind ))) ) ```

Here is the final policy file in the rootful scenario. ``` (block envoy     (blockinherit container)     (blockinherit restricted_net_container)     (allow process process ( capability ( audit_write chown dac_override fowner fsetid kill mknod net_bind_service net_raw setfcap setgid setpcap setuid sys_chroot )))     (allow process container_runtime_t (fifo_file ( setattr )))     (allow process unreserved_port_t ( tcp_socket ( name_bind ))) ) ``` Thank you. PS: If you are reading this thread then here is a great resource for editing *.cil files: https://github.com/SELinuxProject/selinux-notebook/blob/main/src/cil_overvie... -- Chintan Mishra

Container started and a curl test was successful ``` # podman ps -a --filter "label=envoy=envoy" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 44f02a9f2ac3 docker.io/envoyproxy/envoy:v1.15.0 envoy -c /etc/env... About a minute ago Up About a minute ago 0.0.0.0:10000->10000/tcp envoy

# curl -I localhost:10000 HTTP/1.1 200 OK content-type: text/html; charset=ISO-8859-1 p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info." date: Thu, 10 Sep 2020 21:40:16 GMT server: envoy x-xss-protection: 0 x-frame-options: SAMEORIGIN expires: Thu, 10 Sep 2020 21:40:16 GMT cache-control: private set-cookie: 1P_JAR=2020-09-10-21; expires=Sat, 10-Oct-2020 21:40:16 GMT; path=/; domain=.google.com; Secure set-cookie: NID=204=Py9ONzAvLYe41BNU_HWe88th45fOsxWWmjbh6aodR2wroK8r7gY8blxHV54zG7deSKNmtOT66FQQnyPn8vpk_vb6CwE6ZH-_D3KQgNByttyF2qdUifuYnfzMlirQKv1aWejLrQPdTpt7WDjULDZDTlNpa9BIsvfA4dSShDrfgx4; expires=Fri, 12-Mar-2021 21:40:16 GMT; path=/; domain=.google.com; HttpOnly alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" x-envoy-upstream-service-time: 64 transfer-encoding: chunked ```

Hope this helps. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io

Hello!! This is the development configuration being used: https://pastebin.com/8J2Ev5Ys I have added some comments to help readers understand the config file. The configuration above uses three containers viz one Envoyproxy, one frontend, and one backend container. If needed then here is a simpler version of the same config that you can use with just Envoyproxy container and either one of the remaining containers: https://pastebin.com/biZCUPm7 Thank you. -- Chintan Mishra On 10/09/20 3:09 am, Daniel Walsh wrote:

On 9/9/20 08:24, Chintan from Rebhu wrote: Could you attache the envoy.yaml?

...

Hello!!

On 08/09/20 11:28 pm, Daniel Walsh wrote:

...

What command are you using to launch the Envoyproxy container?

$ sudo podman run -v ./envoy.yaml:/etc/envoy/envoy.yaml --network test --name test-lz -d -p 10000:10000 envoyproxy/envoy:v1.15.0

...

The AVC's indicate the container is attempting to write to a fifo_file created by podman

# podman run --privileged -v /usr/bin/find:/usr/bin/find fedora find / -type p The issue is I am not getting any information about what fifo file is being blocked?

There is no output returned by the above command in rootful mode.

Running the same command in rootless mode returns

find: '/proc/tty/driver': Permission denied

...

Is this running rootful or rootless?

The AVC output in the pastebin was from the container running in rootful mode.

I have tried running the container in both rootful and rootless mode.

-- Chintan Mishra

...

On 9/8/20 13:06, Chintan from Rebhu wrote:

...

Hello, everyone!!

Thank you Chris, Matt, and Daniel for helping me understand the queries in my mind.

Here is a pastebin of Access Vector Cache(AVC) output https://pastebin.com/c1w8AnA7

The contents were fetched using `|ausearch -m avc --start recent|`.

On 08/09/20 6:32 pm, Chris Evich wrote:

...

On 9/8/20 12:48 AM, Chintan from Rebhu wrote:

...

The same error is not present when I switch from v1.15.0 to v1.14.4 of Envoyproxy.

I am out of my wits about this. Please tell me how I should find a solution.

We only use Podman in our infrastructure.

Hi,

First off, I don't mean to dismiss the scale of your frustrations or challenges here.  However, this list may not be the appropriate place for seeking infrastructure support for third-party interoperability (open-source or closed).

We're really narrowly focused on podman and it's intimately related container technologies, and improving the state thereof.  If you're looking for end-to-end debugging and help, then a commercial support option would probably be best.

If it was something non-trivial then I would happily seek commercial support. I am just trying to run a command that works with Docker on Debian(checked a few moments ago) but fails on Podman. Podman is marketed as a drop-in replacement for Docker. So, one would expect that commands work out of box. And most generic applications do run fine. But there are a significant number of applications that need certain modifications to get things working with Podman. One grows appreciative of the finer details as they use Podman. But these finer details are poorly documented and highly fragmented all over the internet. And if applications keep not working and stay without resolution or without documentation then that will be a deterrent for Podman's wider community adoption.

What do you suggest the community do to get applications running through Podman?

...

OTOH, I see you're follow-up mail, which does have some podman-specific questions we can probably help with.  I just want to be clear up-front, that this list is not an official/commercial means for overall/general system support.

In the first e-mail, I failed to mention that I ruled out issues in the containerfile provided by the Envoyproxy maintainers. There is no mention of a similar issue on their communication channels like Slack, GitHub issues and Google group. So, it was only logical to conclude that the issue is with Podman.

-- Chintan Mishra

_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io

_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io

_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io

_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io