Best practices for restricting container network access
Hello all, I was wondering what was the current recommended way to restrict network access of containers? I'm trying to setup a web service via 'podman compose' ; I like most of my web services (proxied through Nginx) to only have access to the lo interface, and while for other services this is done via a systemd unit setting I'm having trouble finding the equivalent for podman. The following rules seem to work (I can also use -s/-d and specify the whole IP range used by containers): iptables -t filter -I NETAVARK_FORWARD -i podman2 ! -o lo,podman2 -j DROP iptables -t filter -I NETAVARK_FORWARD -o podman2 ! -i lo,podman2 -j DROP However, I'm not sure when the NETAVARK_FORWARD table is created (should I even use this table?), and the podman2 interface also does not exist before the network is created (when running 'podman compose up'). Is there a way to run these commands when the containers are brought up, like some kind of pre-up script? Is there a better way of achieving what I'm trying to do? Thanks for your help, François-Xavier
Hello François-Xavier, I can't speak to best practices or broadly acceptable recommendations, but below is what I would try. For each container that doesn't need to talk to any other container, set `network_mode: none` for that container's entry in the `services:` top-level element. Please note that I haven't personally tried combining `network_mode: none` with `ports:` within the same container, so if that doesn't work as I described that may be my fault. For each container that needs to talk to another container, define a network in the `networks:` top-level element with `internal: true` set for it. In each container's entry in the `services:` top-level element, include that network in `networks:`. In either case, these should prevent the default behaviour of attaching containers to the default network, which is what normally gives broad outbound network access to containers. No iptables commands should be needed. If you only want the containers' exposed ports to be accessible on the machine running the containers, specify 127.0.0.1 in addition to the port(s) themselves in each container's `ports:` section (syntax here: https://github.com/compose-spec/compose-spec/blob/master/05-services.md#port...). For syntax and extra options, I would look here: https://github.com/compose-spec/compose-spec/blob/master/spec.md Cheers, -Keith On Wednesday, July 10th, 2024 at 7:46 PM, François-Xavier Thomas <fx.thomas@gmail.com> wrote:
Hello all,
I was wondering what was the current recommended way to restrict network access of containers?
I'm trying to setup a web service via 'podman compose' ; I like most of my web services (proxied through Nginx) to only have access to the lo interface, and while for other services this is done via a systemd unit setting I'm having trouble finding the equivalent for podman.
The following rules seem to work (I can also use -s/-d and specify the whole IP range used by containers):
iptables -t filter -I NETAVARK_FORWARD -i podman2 ! -o lo,podman2 -j DROP iptables -t filter -I NETAVARK_FORWARD -o podman2 ! -i lo,podman2 -j DROP
However, I'm not sure when the NETAVARK_FORWARD table is created (should I even use this table?), and the podman2 interface also does not exist before the network is created (when running 'podman compose up').
Is there a way to run these commands when the containers are brought up, like some kind of pre-up script? Is there a better way of achieving what I'm trying to do?
Thanks for your help, François-Xavier _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Hi Keith! Thanks for taking the time to explore this with me!
For each container that doesn't need to talk to any other container, set `network_mode: none` for that container's entry in the `services:` top-level element. Please note that I haven't personally tried combining `network_mode: none` with `ports:` within the same container, so if that doesn't work as I described that may be my fault.
Generally I'm not too interested in blocking containers entirely - I'm interested in web app containers that are either on the other side of the proxy themselves, or communicating with the container that is.
For each container that needs to talk to another container, define a network in the `networks:` top-level element with `internal: true` set for it. In each container's entry in the `services:` top-level element, include that network in `networks:`.
When I do that, DNS name resolution stops working: I cannot nslookup <other container> e.g. a database from within a web server container, which is useful for avoiding having to hardcode IPs. Any way to help with that? I think the podman resolver (Netavark) is listening on the configured IP in /etc/resolv.conf (10.89.0.1:53 for example) but trying to talk to it times out.
If you only want the containers' exposed ports to be accessible on the machine running the containers, specify 127.0.0.1 in addition to the port(s) themselves in each container's `ports:` section (syntax here: https://github.com/compose-spec/compose-spec/blob/master/05-services.md#port...).
Yup, I already do that.
Hello all, I did some more investigation. TL;DR: Internal networks should work for my purposes as Keith said (while still being able to resolve container names via DNS), but apparently that doesn't work on my host due to either a bug or unsupported configuration.
For each container that needs to talk to another container, define a network in the `networks:` top-level element with `internal: true` set for it. In each container's entry in the `services:` top-level element, include that network in `networks:`.
When I do that, DNS name resolution stops working: I cannot nslookup <other container> e.g. a database from within a web server container, which is useful for avoiding having to hardcode IPs. Any way to help with that?
The following rules (I copied part of what's present when internal: false) are enough to get DNS working in containers with internal: true: iptables -I INPUT 1 -s 10.89.0.0/24 -d 10.89.0.0/24 -p udp --dport 20053 -j ACCEPT iptables -t nat -I PREROUTING 1 -m addrtype --dst-type LOCAL -p udp -d 10.89.0.0/24 --dport 53 -j DNAT --to-destination 10.89.0.1:20053 iptables -t nat -I OUTPUT 1 -m addrtype --dst-type LOCAL -p udp -d 10.89.0.0/24 --dport 53 -j DNAT --to-destination 10.89.0.1:20053 Container-to-container networking (directly via their IP) was already working even without these rules - they only affect DNS resolution. On my machine I set the dns_bind_port to 20053 in /etc/containers/containers.conf (because a DNS resolver is already listening on port 53 on the host and prevents aardvark-dns from running on the default port), but it turns out that: * My default policy for the INPUT and FORWARD iptables chains is DROP, managed by ufw, which blocks access to the resolver. The first rule lets containers contact the resolver at the non-standard 20053 port, but not 53 (because it's not actually listening at that port). * The last 2 rules redirect regular port 53 DNS requests from inside containers to the resolver listening at port 20053, making DNS finally work. I don't know why they're missing with internal: true. I looked at [0] in detail and it does not mention anything to explain these two gotchas. Is there any chance they could be added in the documentation/FAQ, and is [0] the best place for it? Or is it something that could be fixed/improved in Podman itself? In the second bullet point in particular a non-default DNS port is simply not working at all in an internal network because the DNAT rule is missing. In the first point I feel that Podman/netavark should cooperate with (or at least warn about connectivity issues potentially caused by) existing firewall rules and default chain policies, but I would understand that supporting all possible (mis-)configurations would be a daunting task! Thanks a lot for your insights, François-Xavier --- [0] https://github.com/containers/podman/blob/main/docs/tutorials/basic_networki...
Hello all, I hope you don't mind me sharing my results here, at least this might help other people in the same situation as me - but if anybody has a comment on how things are supposed to be working I would be glad! Re: container-to-container DNS issues -------------------------------------
TL;DR: Internal networks should work for my purposes as Keith said (while still being able to resolve container names via DNS), but apparently that doesn't work on my host due to either a bug or unsupported configuration.
I'm happy to say I solved this! In the end the DNS issues were all my fault - my local DNS resolver was configured to run on 0.0.0.0 (all interfaces), and it took me a while to figure out that it would silently prevent container-to-container name resolution from working because it *also* automatically listened on the Podman virtual interfaces before aardvark-dns had a chance to start. I changed it to only run on the external and loopback interfaces, and now container-to-container DNS works just fine with the default port. It looks like there were two separate things in Podman that made understanding all this more difficult: - there is no warning message when aardvark-dns can't start because the port is already taken by the host (that would have made the issue very obvious) - internal networks don't generate DNAT rules when dns_port is set to anything other than 53 ; containers can access the DNS resolver on the non-standard port just fine and have /etc/resolv.conf configured to the correct IP, but the resolv.conf mechanism cannot (to my knowledge) use a different port and thus DNS fails in practice Re: restricting container access -------------------------------- Now back to my original goal.
For each container that needs to talk to another container, define a network in the `networks:` top-level element with `internal: true` set for it. In each container's entry in the `services:` top-level element, include that network in `networks:`.
If you only want the containers' exposed ports to be accessible on the machine running the containers, specify 127.0.0.1 in addition to the port(s) themselves in each container's `ports:` section (syntax here: https://github.com/compose-spec/compose-spec/blob/master/05-services.md#port...).
I retried Keith's advice, here is the current behavior I'm seeing (with ufw setup to default deny incoming and forwarded connections, and no extra rules related to containers - just my regular rules for services on the host): - With internal: false, I can communicate (ping, mapped ports and exposed ports) with all the containers from the hsot, and they can also access external services on the internet or on the local network. - With internal: true, I can ping the containers from the host, but the rest of the communications are blocked, including mapped ports. The last part sounds weird to me, is that the expected behavior or is it maybe another misconfiguration on my part? I was expecting port mapping to be an allowlist, something like "internal networks have no communications, except those that are explicitly allowed one by one. I unfortunately haven't found any online resources that tell me how internal networks are supposed to work in detail, other than saying (paraphrasing) "internal networks are internal"... The podman compose commands also let me map a port just fine but doesn't tell me it's not going to work because the network is internal. It sounds that so far my best bet would be: * Don't use internal networks (because I want port mapping to work) * Add extra iptables/ufw rules that default-denies all traffic outside of the podman network except the one I want (still working on that one... making it appear on boot without conflicting with the netavark rules is tricky) I'll keep you posted when I find a workable solution. Take care, FX
Hello all,
I hope you don't mind me sharing my results here, at least this might help other people in the same situation as me - but if anybody has a comment on how things are supposed to be working I would be glad!
Re: container-to-container DNS issues -------------------------------------
TL;DR: Internal networks should work for my purposes as Keith said (while still being able to resolve container names via DNS), but apparently that doesn't work on my host due to either a bug or unsupported configuration.
I'm happy to say I solved this!
In the end the DNS issues were all my fault - my local DNS resolver was configured to run on 0.0.0.0 (all interfaces), and it took me a while to figure out that it would silently prevent container-to-container name resolution from working because it *also* automatically listened on the Podman virtual interfaces before aardvark-dns had a chance to start.
I changed it to only run on the external and loopback interfaces, and now container-to-container DNS works just fine with the default port.
It looks like there were two separate things in Podman that made understanding all this more difficult:
- there is no warning message when aardvark-dns can't start because the port is already taken by the host (that would have made the issue very obvious) I fixed this very recently in netavark/aardvark-dns v1.12.0, so this already fixed.
- internal networks don't generate DNAT rules when dns_port is set to anything other than 53 ; containers can access the DNS resolver on the non-standard port just fine and have /etc/resolv.conf configured to the correct IP, but the resolv.conf mechanism cannot (to my knowledge) use a different port and thus DNS fails in practice Correct this is a problem, please file a bug on the netavark repo about it. This is similar to https://github.com/containers/podman/issues/22807. Right now internal networks do nothing with the host firewall I think we must reevaluate
Hi François-Xavier, replies inline. On 04/08/2024 16:16, François-Xavier Thomas wrote: that design decision.
Re: restricting container access --------------------------------
Now back to my original goal.
For each container that needs to talk to another container, define a network in the `networks:` top-level element with `internal: true` set for it. In each container's entry in the `services:` top-level element, include that network in `networks:`.
If you only want the containers' exposed ports to be accessible on the machine running the containers, specify 127.0.0.1 in addition to the port(s) themselves in each container's `ports:` section (syntax here: https://github.com/compose-spec/compose-spec/blob/master/05-services.md#port...).
I retried Keith's advice, here is the current behavior I'm seeing (with ufw setup to default deny incoming and forwarded connections, and no extra rules related to containers - just my regular rules for services on the host):
- With internal: false, I can communicate (ping, mapped ports and exposed ports) with all the containers from the hsot, and they can also access external services on the internet or on the local network.
- With internal: true, I can ping the containers from the host, but the rest of the communications are blocked, including mapped ports.
The last part sounds weird to me, is that the expected behavior or is it maybe another misconfiguration on my part?
This is expected with our current design see my point above how we do nothing with the firewall int he internal case thus no port DNAT rules are added as well. However note that this will actually work when running rootless podman today as it uses a user space forwarder.
I was expecting port mapping to be an allowlist, something like "internal networks have no communications, except those that are explicitly allowed one by one.
I unfortunately haven't found any online resources that tell me how internal networks are supposed to work in detail, other than saying (paraphrasing) "internal networks are internal"... The podman compose commands also let me map a port just fine but doesn't tell me it's not going to work because the network is internal.
It sounds that so far my best bet would be:
* Don't use internal networks (because I want port mapping to work) * Add extra iptables/ufw rules that default-denies all traffic outside of the podman network except the one I want (still working on that one... making it appear on boot without conflicting with the netavark rules is tricky)
Yes this is the difficult part. I think we should likely expose some netavark-user chains where users could hook into to further restrict traffic at there will. Contributions welcome, https://github.com/containers/netavark/issues/705.
I'll keep you posted when I find a workable solution.
Take care, FX _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
-- Paul Holzinger
Hello Paul, Thanks for the answers, that helps a lot - I haven't really used containers since more than 5 years and that space evolves quickly so I wasn't sure if I was missing something! On 05/08/2024 19:28, Paul Holzinger wrote:
- there is no warning message when aardvark-dns can't start because the port is already taken by the host (that would have made the issue very obvious) I fixed this very recently in netavark/aardvark-dns v1.12.0, so this already fixed.
Great news, thank you!
- internal networks don't generate DNAT rules when dns_port is set to anything other than 53 ; containers can access the DNS resolver on the non-standard port just fine and have /etc/resolv.conf configured to the correct IP, but the resolv.conf mechanism cannot (to my knowledge) use a different port and thus DNS fails in practice Correct this is a problem, please file a bug on the netavark repo about it. This is similar to https://github.com/containers/podman/issues/22807. Right now internal networks do nothing with the host firewall I think we must reevaluate that design decision.
Created: https://github.com/containers/netavark/issues/1051
The last part sounds weird to me, is that the expected behavior or is it maybe another misconfiguration on my part? This is expected with our current design see my point above how we do nothing with the firewall int he internal case thus no port DNAT rules are added as well. However note that this will actually work when running rootless podman today as it uses a user space forwarder.
Good to know even if for the moment I'm mostly using rootful. In general I felt that the "Basic networking" document was both too detailed and not detailed enough when I tried to understand all this. Would you welcome suggestions/attempts at creating small PRs for the documentation, or is that something that's best done by the dev team? What's the best location, the podman repo?
On 10/08/2024 18:58, François-Xavier Thomas wrote:
Hello Paul,
Thanks for the answers, that helps a lot - I haven't really used containers since more than 5 years and that space evolves quickly so I wasn't sure if I was missing something!
On 05/08/2024 19:28, Paul Holzinger wrote:
- there is no warning message when aardvark-dns can't start because the port is already taken by the host (that would have made the issue very obvious) I fixed this very recently in netavark/aardvark-dns v1.12.0, so this already fixed.
Great news, thank you!
- internal networks don't generate DNAT rules when dns_port is set to anything other than 53 ; containers can access the DNS resolver on the non-standard port just fine and have /etc/resolv.conf configured to the correct IP, but the resolv.conf mechanism cannot (to my knowledge) use a different port and thus DNS fails in practice Correct this is a problem, please file a bug on the netavark repo about it. This is similar to https://github.com/containers/podman/issues/22807. Right now internal networks do nothing with the host firewall I think we must reevaluate that design decision.
Created: https://github.com/containers/netavark/issues/1051 Thanks.
The last part sounds weird to me, is that the expected behavior or is it maybe another misconfiguration on my part? This is expected with our current design see my point above how we do nothing with the firewall int he internal case thus no port DNAT rules are added as well. However note that this will actually work when running rootless podman today as it uses a user space forwarder.
Good to know even if for the moment I'm mostly using rootful.
In general I felt that the "Basic networking" document was both too detailed and not detailed enough when I tried to understand all this.
Would you welcome suggestions/attempts at creating small PRs for the documentation, or is that something that's best done by the dev team? What's the best location, the podman repo?
External contributions are always welcome. The Podman repo is the best place, i.e. here is the source of the current "Basic networking" Guide: https://github.com/containers/podman/blob/main/docs/tutorials/basic_networki...
participants (3)
-
François-Xavier Thomas -
Keith H -
Paul Holzinger