Re: Networking / CNI and firewalld (iptables)
(moving thread back to list, excuse the lack of trimming) On 5/26/20 1:18 PM, Felder, Christian wrote:
On 26. May 2020, at 19:43, Gordon Messmer <gordon.messmer@gmail.com <mailto:gordon.messmer@gmail.com>> wrote:
On 5/26/20 1:51 AM, Felder, Christian wrote:
When using podman run -p … DNAT rules in the forward chain are automatically created for allowing traffic to the container/pod.
I think you might be mixing up two different things. When I run "podman run -p" I see a new rule in the PREROUTING chain of the "nat" table. I don't see any rules in the FORWARD chain of the "filter" table.
Sorry. You’re right and I mixed things up. Indeed there is a new rule in PREROUTING and this is the rule which bypasses the INPUT chain. I cannot explicitly configure ports on the INPUT chain as the packets are forwarded to the CNI-HOSTPORT-DNAT target directly.
Because you're doing DNAT, you should be looking at the FORWARD chain, not the INPUT chain. As far as I can tell, testing on my system, the FORWARD chain requires an explicit rule to allow external access to container-published ports. I do have a system that doesn't require an explicit rule, because the destination network is part of firewalld's "trusted" zone. It seems likely that you also have a rule that is allowing forwarded traffic. Can you post the complete output of "iptables -L -nv" and "iptables -L -nv -t nat" somewhere that we can view them, if you don't see the rule now? (ip6tables if you're testing connections from an external host over IPv6, of course)
There are no DNAT rules in the FORWARD chain.
I mixed this up with adding an explicit rule to the top of the FORWARD chain as described in this firewalld issue regarding docker. https://github.com/firewalld/firewalld/issues/556
e.g. firewall-cmd --direct --add-passthrough ipv4 -I FORWARD 1 -s 10.88.2.0/24 -p tcp --dport 636 -j DROP
but this wouldn’t help either. Let’s forget about that ;-)
Unfortunately this bypasses the input chain which is usually used to explicitly allowing external traffic for a specific service/port. Using podman run -p … the port is world-wide accessible though.
That isn't true on my system, as far as I can tell.
One solution is to just bind to the loopback interface using -p 127.0.0.1:XXX:XXX which will ensure that the port is just available on the host system but on the other hand this does not allow using ssh tunnelling for authorised external access.
Why wouldn't it allow ssh tunneling?
You’re right I can tunnel traffic to it, e.g. using
ssh -M -S ~/.ssh/ssh-ldap3 -f root@centos-8 -L 6636:localhost:636 -N
which would allow me to access the container from my system on port 6636 which runs on port 636 on the host centos-8.
My conclusion:
It’s probably the easiest option to publish to loopback if I wan’t to limit access just to the host whereas publishing to all interfaces would still allow external traffic as well without explicitly configuring the firewall because of the PREROUTING mechanism. I can still ssh-tunnel traffic to that service when binding to loopback.
Imho people may not be aware of the INPUT chain being bypassed.
On 26. May 2020, at 22:59, Gordon Messmer <gordon.messmer@gmail.com> wrote:
(moving thread back to list, excuse the lack of trimming)
On 5/26/20 1:18 PM, Felder, Christian wrote:
On 26. May 2020, at 19:43, Gordon Messmer <gordon.messmer@gmail.com <mailto:gordon.messmer@gmail.com> <mailto:gordon.messmer@gmail.com <mailto:gordon.messmer@gmail.com>>> wrote:
On 5/26/20 1:51 AM, Felder, Christian wrote:
When using podman run -p … DNAT rules in the forward chain are automatically created for allowing traffic to the container/pod.
I think you might be mixing up two different things. When I run "podman run -p" I see a new rule in the PREROUTING chain of the "nat" table. I don't see any rules in the FORWARD chain of the "filter" table.
Sorry. You’re right and I mixed things up. Indeed there is a new rule in PREROUTING and this is the rule which bypasses the INPUT chain. I cannot explicitly configure ports on the INPUT chain as the packets are forwarded to the CNI-HOSTPORT-DNAT target directly.
Because you're doing DNAT, you should be looking at the FORWARD chain, not the INPUT chain. As far as I can tell, testing on my system, the FORWARD chain requires an explicit rule to allow external access to container-published ports.
I do have a system that doesn't require an explicit rule, because the destination network is part of firewalld's "trusted" zone. It seems likely that you also have a rule that is allowing forwarded traffic. Can you post the complete output of "iptables -L -nv" and "iptables -L -nv -t nat" somewhere that we can view them, if you don't see the rule now? (ip6tables if you're testing connections from an external host over IPv6, of course)
$ iptables -L -nv Chain INPUT (policy ACCEPT 632 packets, 93455 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 0 0 ACCEPT udp -- cni-podman0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 Chain FORWARD (policy ACCEPT 9 packets, 488 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT 525 packets, 90646 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68 $ iptables -L -nv -t nat Chain PREROUTING (policy ACCEPT 138 packets, 96492 bytes) pkts bytes target prot opt in out source destination 137 95750 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT 1 packets, 64 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 37 packets, 2827 bytes) pkts bytes target prot opt in out source destination 2 159 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24 0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255 0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 0 0 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 0 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24 33 2513 CNI-HOSTPORT-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd requiring masquerade */ 0 0 CNI-89e741c1b1372b76c949874b all -- * * 10.88.2.32 0.0.0.0/0 /* name: "podman" id: "0b6380bee2428b325486817d5c805c2ddfb32724bc4ca74ed24bc52e3974c9cf" */ Chain OUTPUT (policy ACCEPT 35 packets, 2699 bytes) pkts bytes target prot opt in out source destination 0 0 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain CNI-HOSTPORT-SETMARK (2 references) pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd masquerade mark */ MARK or 0x2000 Chain CNI-HOSTPORT-MASQ (1 references) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000 Chain CNI-HOSTPORT-DNAT (2 references) pkts bytes target prot opt in out source destination 2 128 CNI-DN-89e741c1b1372b76c9498 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "podman" id: "0b6380bee2428b325486817d5c805c2ddfb32724bc4ca74ed24bc52e3974c9cf" */ multiport dports 636 Chain CNI-89e741c1b1372b76c949874b (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.2.0/24 /* name: "podman" id: "0b6380bee2428b325486817d5c805c2ddfb32724bc4ca74ed24bc52e3974c9cf" */ 0 0 MASQUERADE all -- * * 0.0.0.0/0 !224.0.0.0/4 /* name: "podman" id: "0b6380bee2428b325486817d5c805c2ddfb32724bc4ca74ed24bc52e3974c9cf" */ Chain CNI-DN-89e741c1b1372b76c9498 (1 references) pkts bytes target prot opt in out source destination 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.88.2.32 0.0.0.0/0 tcp dpt:636 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:636 2 128 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:636 to:10.88.2.32:6636 $ firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: enp0s5 sources: services: cockpit dhcpv6-client ssh ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules:
There are no DNAT rules in the FORWARD chain.
I mixed this up with adding an explicit rule to the top of the FORWARD chain as described in this firewalld issue regarding docker. https://github.com/firewalld/firewalld/issues/556
e.g. firewall-cmd --direct --add-passthrough ipv4 -I FORWARD 1 -s 10.88.2.0/24 -p tcp --dport 636 -j DROP
but this wouldn’t help either. Let’s forget about that ;-)
Unfortunately this bypasses the input chain which is usually used to explicitly allowing external traffic for a specific service/port. Using podman run -p … the port is world-wide accessible though.
That isn't true on my system, as far as I can tell.
One solution is to just bind to the loopback interface using -p 127.0.0.1:XXX:XXX which will ensure that the port is just available on the host system but on the other hand this does not allow using ssh tunnelling for authorised external access.
Why wouldn't it allow ssh tunneling?
You’re right I can tunnel traffic to it, e.g. using
ssh -M -S ~/.ssh/ssh-ldap3 -f root@centos-8 -L 6636:localhost:636 -N
which would allow me to access the container from my system on port 6636 which runs on port 636 on the host centos-8.
My conclusion:
It’s probably the easiest option to publish to loopback if I wan’t to limit access just to the host whereas publishing to all interfaces would still allow external traffic as well without explicitly configuring the firewall because of the PREROUTING mechanism. I can still ssh-tunnel traffic to that service when binding to loopback.
Imho people may not be aware of the INPUT chain being bypassed.
_______________________________________________ Podman mailing list -- podman@lists.podman.io <mailto:podman@lists.podman.io> To unsubscribe send an email to podman-leave@lists.podman.io <mailto:podman-leave@lists.podman.io>
On 5/26/20 2:15 PM, Felder, Christian wrote:
Sorry. You’re right and I mixed things up. Indeed there is a new rule in PREROUTING and this is the rule which bypasses the INPUT chain. I cannot explicitly configure ports on the INPUT chain as the packets are forwarded to the CNI-HOSTPORT-DNAT target directly.
$ iptables -L -nv Chain FORWARD (policy ACCEPT 9 packets, 488 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Your FORWARD rules only govern traffic that crosses the "virbr0" interface specifically, so anything that doesn't uses the default "ACCEPT" policy. Set your policy to DENY, or add a REJECT rule that matches all packets at the end of that chain.
On 26. May 2020, at 23:28, Gordon Messmer <gordon.messmer@gmail.com> wrote:
On 5/26/20 2:15 PM, Felder, Christian wrote:
Sorry. You’re right and I mixed things up. Indeed there is a new rule in PREROUTING and this is the rule which bypasses the INPUT chain. I cannot explicitly configure ports on the INPUT chain as the packets are forwarded to the CNI-HOSTPORT-DNAT target directly.
$ iptables -L -nv Chain FORWARD (policy ACCEPT 9 packets, 488 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Your FORWARD rules only govern traffic that crosses the "virbr0" interface specifically, so anything that doesn't uses the default "ACCEPT" policy.
Set your policy to DENY, or add a REJECT rule that matches all packets at the end of that chain.
Thanks for pointing that out. When changing the default policy to DROP (using iptables -P FORWARD DROP) the traffic is blocked as expected. Using iptables is deprecated in RHEL/CentOS 8 and unfortunately the default policy for firewalld is hard-wired to ACCEPT as it has another concept of using zones which I tried without success in this use-case so far. Note: The CNI is added to trusted zone by default. I think I’ll stick to the approach not publishing any services to all interfaces which should not be exposed world-wide. When using this internally I can still bind to loopback (127.0.0.1) and use ssh-tunnelling to get access to that service as intended. Thus I don’t need to fiddle too much with the firewalld configs in this scenario. Thanks a lot for your great advice. I hope this is beneficial for someone else stumbling across this.
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 5/28/20 2:19 AM, Felder, Christian wrote:
Using iptables is deprecated in RHEL/CentOS 8 and unfortunately the default policy for firewalld is hard-wired to ACCEPT as it has another concept of using zones which I tried without success in this use-case so far. Note: The CNI is added to trusted zone by default.
I haven't moved to CentOS 8 yet, but that's definitely something I'll need to look at when I do. All of the versions of firewalld I've used end the FORWARD chain with a REJECT rule that isn't limited to any specific interface. Unless this is intentional behavior related to the CNI interface being trusted, it seems like a bug.
participants (2)
-
Felder, Christian -
Gordon Messmer