9:17 a.m.
Hello all,
I was wondering what was the current recommended way to restrict network
access of containers?
I'm trying to setup a web service via 'podman compose' ; I like most of
my web services (proxied through Nginx) to only have access to the lo
interface, and while for other services this is done via a systemd unit
setting I'm having trouble finding the equivalent for podman.
The following rules seem to work (I can also use -s/-d and specify the
whole IP range used by containers):
iptables -t filter -I NETAVARK_FORWARD -i podman2 ! -o lo,podman2 -j DROP
iptables -t filter -I NETAVARK_FORWARD -o podman2 ! -i lo,podman2 -j DROP
However, I'm not sure when the NETAVARK_FORWARD table is created (should
I even use this table?), and the podman2 interface also does not exist
before the network is created (when running 'podman compose up').
Is there a way to run these commands when the containers are brought up,
like some kind of pre-up script? Is there a better way of achieving what
I'm trying to do?
Thanks for your help,
François-Xavier
11:52 a.m.
Hello François-Xavier,
I can't speak to best practices or broadly acceptable recommendations, but below is
what I would try.
For each container that doesn't need to talk to any other container, set
`network_mode: none` for that container's entry in the `services:` top-level element.
Please note that I haven't personally tried combining `network_mode: none` with
`ports:` within the same container, so if that doesn't work as I described that may be
my fault.
For each container that needs to talk to another container, define a network in the
`networks:` top-level element with `internal: true` set for it. In each container's
entry in the `services:` top-level element, include that network in `networks:`.
In either case, these should prevent the default behaviour of attaching containers to the
default network, which is what normally gives broad outbound network access to containers.
No iptables commands should be needed.
If you only want the containers' exposed ports to be accessible on the machine running
the containers, specify 127.0.0.1 in addition to the port(s) themselves in each
container's `ports:` section (syntax here:
https://github.com/compose-spec/compose-spec/blob/master/05-services.md#p...).
For syntax and extra options, I would look here:
https://github.com/compose-spec/compose-spec/blob/master/spec.md
Cheers,
-Keith
On Wednesday, July 10th, 2024 at 7:46 PM, François-Xavier Thomas
<fx.thomas(a)gmail.com> wrote:
>
>
> Hello all,
>
> I was wondering what was the current recommended way to restrict network
> access of containers?
>
> I'm trying to setup a web service via 'podman compose' ; I like most of
> my web services (proxied through Nginx) to only have access to the lo
> interface, and while for other services this is done via a systemd unit
> setting I'm having trouble finding the equivalent for podman.
>
> The following rules seem to work (I can also use -s/-d and specify the
> whole IP range used by containers):
>
> iptables -t filter -I NETAVARK_FORWARD -i podman2 ! -o lo,podman2 -j DROP
> iptables -t filter -I NETAVARK_FORWARD -o podman2 ! -i lo,podman2 -j DROP
>
> However, I'm not sure when the NETAVARK_FORWARD table is created (should
> I even use this table?), and the podman2 interface also does not exist
> before the network is created (when running 'podman compose up').
>
> Is there a way to run these commands when the containers are brought up,
> like some kind of pre-up script? Is there a better way of achieving what
> I'm trying to do?
>
> Thanks for your help,
> François-Xavier
> _______________________________________________
> Podman mailing list -- podman(a)lists.podman.io
> To unsubscribe send an email to podman-leave(a)lists.podman.io
11:40 a.m.
Hi Keith!
Thanks for taking the time to explore this with me!
For each container that doesn't need to talk to any other
container, set `network_mode: none` for that container's entry in the `services:`
top-level element. Please note that I haven't personally tried combining
`network_mode: none` with `ports:` within the same container, so if that doesn't work
as I described that may be my fault.
Generally I'm not too interested in blocking containers entirely - I'm
interested in web app containers that are either on the other side of
the proxy themselves, or communicating with the container that is.
For each container that needs to talk to another container, define a
network in the `networks:` top-level element with `internal: true` set for it. In each
container's entry in the `services:` top-level element, include that network in
`networks:`.
When I do that, DNS name resolution stops working: I cannot nslookup
<other container> e.g. a database from within a web server container,
which is useful for avoiding having to hardcode IPs. Any way to help
with that?
I think the podman resolver (Netavark) is listening on the configured IP
in /etc/resolv.conf (10.89.0.1:53 for example) but trying to talk to it
times out.
If you only want the containers' exposed ports to be accessible
on the machine running the containers, specify 127.0.0.1 in addition to the port(s)
themselves in each container's `ports:` section (syntax here:
https://github.com/compose-spec/compose-spec/blob/master/05-services.md#p...).
Yup, I already do that.
2:57 p.m.
Hello all,
I did some more investigation.
TL;DR: Internal networks should work for my purposes as Keith said
(while still being able to resolve container names via DNS), but
apparently that doesn't work on my host due to either a bug or
unsupported configuration.
> For each container that needs to talk to another container,
define a
> network in the `networks:` top-level element with `internal: true` set
> for it. In each container's entry in the `services:` top-level
> element, include that network in `networks:`.
When I do that, DNS name resolution stops working: I cannot nslookup
<other container> e.g. a database from within a web server container,
which is useful for avoiding having to hardcode IPs. Any way to help
with that?
The following rules (I copied part of what's present when internal:
false) are enough to get DNS working in containers with internal: true:
iptables -I INPUT 1 -s 10.89.0.0/24 -d 10.89.0.0/24 -p udp --dport
20053 -j ACCEPT
iptables -t nat -I PREROUTING 1 -m addrtype --dst-type LOCAL -p udp
-d 10.89.0.0/24 --dport 53 -j DNAT --to-destination 10.89.0.1:20053
iptables -t nat -I OUTPUT 1 -m addrtype --dst-type LOCAL -p udp -d
10.89.0.0/24 --dport 53 -j DNAT --to-destination 10.89.0.1:20053
Container-to-container networking (directly via their IP) was already
working even without these rules - they only affect DNS resolution.
On my machine I set the dns_bind_port to 20053 in
/etc/containers/containers.conf (because a DNS resolver is already
listening on port 53 on the host and prevents aardvark-dns from running
on the default port), but it turns out that:
* My default policy for the INPUT and FORWARD iptables chains is DROP,
managed by ufw, which blocks access to the resolver. The first rule lets
containers contact the resolver at the non-standard 20053 port, but not
53 (because it's not actually listening at that port).
* The last 2 rules redirect regular port 53 DNS requests from inside
containers to the resolver listening at port 20053, making DNS finally work.
I don't know why they're missing with internal: true. I looked at [0] in
detail and it does not mention anything to explain these two gotchas. Is
there any chance they could be added in the documentation/FAQ, and is
[0] the best place for it?
Or is it something that could be fixed/improved in Podman itself? In the
second bullet point in particular a non-default DNS port is simply not
working at all in an internal network because the DNAT rule is missing.
In the first point I feel that Podman/netavark should cooperate with (or
at least warn about connectivity issues potentially caused by) existing
firewall rules and default chain policies, but I would understand that
supporting all possible (mis-)configurations would be a daunting task!
Thanks a lot for your insights,
François-Xavier
---
[0]
https://github.com/containers/podman/blob/main/docs/tutorials/basic_netwo...
9:16 a.m.
Hello all,
I hope you don't mind me sharing my results here, at least this might
help other people in the same situation as me - but if anybody has a
comment on how things are supposed to be working I would be glad!
Re: container-to-container DNS issues
-------------------------------------
TL;DR: Internal networks should work for my purposes as Keith said
(while still being able to resolve container names via DNS), but
apparently that doesn't work on my host due to either a bug or
unsupported configuration.
I'm happy to say I solved this!
In the end the DNS issues were all my fault - my local DNS resolver was
configured to run on 0.0.0.0 (all interfaces), and it took me a while to
figure out that it would silently prevent container-to-container name
resolution from working because it *also* automatically listened on the
Podman virtual interfaces before aardvark-dns had a chance to start.
I changed it to only run on the external and loopback interfaces, and
now container-to-container DNS works just fine with the default port.
It looks like there were two separate things in Podman that made
understanding all this more difficult:
- there is no warning message when aardvark-dns can't start because the
port is already taken by the host (that would have made the issue very
obvious)
- internal networks don't generate DNAT rules when dns_port is set to
anything other than 53 ; containers can access the DNS resolver on the
non-standard port just fine and have /etc/resolv.conf configured to the
correct IP, but the resolv.conf mechanism cannot (to my knowledge) use a
different port and thus DNS fails in practice
Re: restricting container access
--------------------------------
Now back to my original goal.
For each container that needs to talk to another container, define a
network in the `networks:` top-level element with `internal: true` set
for it. In each container's entry in the `services:` top-level element,
include that network in `networks:`.
If you only want the containers' exposed ports to be accessible
on
the machine running the containers, specify 127.0.0.1 in addition to the
port(s) themselves in each container's `ports:` section (syntax here:
https://github.com/compose-spec/compose-spec/blob/master/05-services.md#p...).
I retried Keith's advice, here is the current behavior I'm seeing (with
ufw setup to default deny incoming and forwarded connections, and no
extra rules related to containers - just my regular rules for services
on the host):
- With internal: false, I can communicate (ping, mapped ports and
exposed ports) with all the containers from the hsot, and they can also
access external services on the internet or on the local network.
- With internal: true, I can ping the containers from the host, but the
rest of the communications are blocked, including mapped ports.
The last part sounds weird to me, is that the expected behavior or is it
maybe another misconfiguration on my part?
I was expecting port mapping to be an allowlist, something like
"internal networks have no communications, except those that are
explicitly allowed one by one.
I unfortunately haven't found any online resources that tell me how
internal networks are supposed to work in detail, other than saying
(paraphrasing) "internal networks are internal"... The podman compose
commands also let me map a port just fine but doesn't tell me it's not
going to work because the network is internal.
It sounds that so far my best bet would be:
* Don't use internal networks (because I want port mapping to work)
* Add extra iptables/ufw rules that default-denies all traffic outside
of the podman network except the one I want (still working on that
one... making it appear on boot without conflicting with the netavark
rules is tricky)
I'll keep you posted when I find a workable solution.
Take care,
FX
12:28 p.m.
New subject: Best practices for restricting container network access
Hi François-Xavier,
replies inline.
On 04/08/2024 16:16, François-Xavier Thomas wrote:
Hello all,
I hope you don't mind me sharing my results here, at least this might
help other people in the same situation as me - but if anybody has a
comment on how things are supposed to be working I would be glad!
Re: container-to-container DNS issues
-------------------------------------
> TL;DR: Internal networks should work for my purposes as Keith said
(while still being able to resolve container names via DNS), but
apparently that doesn't work on my host due to either a bug or
unsupported configuration.
I'm happy to say I solved this!
In the end the DNS issues were all my fault - my local DNS resolver
was configured to run on 0.0.0.0 (all interfaces), and it took me a
while to figure out that it would silently prevent
container-to-container name resolution from working because it *also*
automatically listened on the Podman virtual interfaces before
aardvark-dns had a chance to start.
I changed it to only run on the external and loopback interfaces, and
now container-to-container DNS works just fine with the default port.
It looks like there were two separate things in Podman that made
understanding all this more difficult:
- there is no warning message when aardvark-dns can't start because
the port is already taken by the host (that would have made the issue
very obvious)
I fixed this very recently in netavark/aardvark-dns v1.12.0, so this
already fixed.
- internal networks don't generate DNAT rules when dns_port is set to
anything other than 53 ; containers can access the DNS resolver on the
non-standard port just fine and have /etc/resolv.conf configured to
the correct IP, but the resolv.conf mechanism cannot (to my knowledge)
use a different port and thus DNS fails in practice
Correct this is a problem,
please file a bug on the netavark repo about
it. This is similar to
https://github.com/containers/podman/issues/22807. Right now internal
networks do nothing with the host firewall I think we must reevaluate
that design decision.
Re: restricting container access
--------------------------------
Now back to my original goal.
> For each container that needs to talk to another container, define a
network in the `networks:` top-level element with `internal: true` set
for it. In each container's entry in the `services:` top-level
element, include that network in `networks:`.
> If you only want the containers' exposed ports to be accessible on
the machine running the containers, specify 127.0.0.1 in addition to
the port(s) themselves in each container's `ports:` section (syntax
here:
https://github.com/compose-spec/compose-spec/blob/master/05-services.md#p...).
I retried Keith's advice, here is the current behavior I'm seeing
(with ufw setup to default deny incoming and forwarded connections,
and no extra rules related to containers - just my regular rules for
services on the host):
- With internal: false, I can communicate (ping, mapped ports and
exposed ports) with all the containers from the hsot, and they can
also access external services on the internet or on the local network.
- With internal: true, I can ping the containers from the host, but
the rest of the communications are blocked, including mapped ports.
The last part sounds weird to me, is that the expected behavior or is
it maybe another misconfiguration on my part?
This is expected with our current
design see my point above how we do
nothing with the firewall int he internal case thus no port DNAT rules
are added as well.
However note that this will actually work when running rootless podman
today as it uses a user space forwarder.
I was expecting port mapping to be an allowlist, something like
"internal networks have no communications, except those that are
explicitly allowed one by one.
I unfortunately haven't found any online resources that tell me how
internal networks are supposed to work in detail, other than saying
(paraphrasing) "internal networks are internal"... The podman compose
commands also let me map a port just fine but doesn't tell me it's not
going to work because the network is internal.
It sounds that so far my best bet would be:
* Don't use internal networks (because I want port mapping to work)
* Add extra iptables/ufw rules that default-denies all traffic outside
of the podman network except the one I want (still working on that
one... making it appear on boot without conflicting with the netavark
rules is tricky)
Yes this is the difficult part. I think we should likely expose
some
netavark-user chains where users could hook into to further restrict
traffic at there will. Contributions welcome,
https://github.com/containers/netavark/issues/705.
I'll keep you posted when I find a workable solution.
Take care,
FX
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
--
Paul Holzinger
11:58 a.m.
New subject: Best practices for restricting container network access
Hello Paul,
Thanks for the answers, that helps a lot - I haven't really used
containers since more than 5 years and that space evolves quickly so I
wasn't sure if I was missing something!
On 05/08/2024 19:28, Paul Holzinger wrote:
> - there is no warning message when aardvark-dns can't start
because
> the port is already taken by the host (that would have made the issue
> very obvious)
I fixed this very recently in netavark/aardvark-dns v1.12.0, so this
already fixed.
Great news, thank you!
> - internal networks don't generate DNAT rules when dns_port
is set to
> anything other than 53 ; containers can access the DNS resolver on the
> non-standard port just fine and have /etc/resolv.conf configured to
> the correct IP, but the resolv.conf mechanism cannot (to my knowledge)
> use a different port and thus DNS fails in practice
Correct this is a problem, please file a bug on the netavark repo about
it. This is similar to
https://github.com/containers/podman/issues/22807. Right now internal
networks do nothing with the host firewall I think we must reevaluate
that design decision.
Created: https://github.com/containers/netavark/issues/1051
> The last part sounds weird to me, is that the expected behavior
or is
> it maybe another misconfiguration on my part?
This is expected with our current design see my point above how we do
nothing with the firewall int he internal case thus no port DNAT rules
are added as well.
However note that this will actually work when running rootless podman
today as it uses a user space forwarder.
Good to know even if for the moment I'm mostly using rootful.
In general I felt that the "Basic networking" document was both too
detailed and not detailed enough when I tried to understand all this.
Would you welcome suggestions/attempts at creating small PRs for the
documentation, or is that something that's best done by the dev team?
What's the best location, the podman repo?
4 a.m.
New subject: Best practices for restricting container network access
On 10/08/2024 18:58, François-Xavier Thomas wrote:
Hello Paul,
Thanks for the answers, that helps a lot - I haven't really used
containers since more than 5 years and that space evolves quickly so I
wasn't sure if I was missing something!
On 05/08/2024 19:28, Paul Holzinger wrote:
>> - there is no warning message when aardvark-dns can't start because
>> the port is already taken by the host (that would have made the
>> issue very obvious)
> I fixed this very recently in netavark/aardvark-dns v1.12.0, so this
> already fixed.
Great news, thank you!
>> - internal networks don't generate DNAT rules when dns_port is set
>> to anything other than 53 ; containers can access the DNS resolver
>> on the non-standard port just fine and have /etc/resolv.conf
>> configured to the correct IP, but the resolv.conf mechanism cannot
>> (to my knowledge) use a different port and thus DNS fails in practice
> Correct this is a problem, please file a bug on the netavark repo
> about it. This is similar to
> https://github.com/containers/podman/issues/22807. Right now internal
> networks do nothing with the host firewall I think we must reevaluate
> that design decision.
Created: https://github.com/containers/netavark/issues/1051
Thanks.
>> The last part sounds weird to me, is that the expected behavior or
>> is it maybe another misconfiguration on my part?
> This is expected with our current design see my point above how we do
> nothing with the firewall int he internal case thus no port DNAT
> rules are added as well.
> However note that this will actually work when running rootless
> podman today as it uses a user space forwarder.
Good to know even if for the moment I'm mostly using rootful.
In general I felt that the "Basic networking" document was both too
detailed and not detailed enough when I tried to understand all this.
Would you welcome suggestions/attempts at creating small PRs for the
documentation, or is that something that's best done by the dev team?
What's the best location, the podman repo?
External contributions are always welcome. The Podman repo is the best
place, i.e. here is the source of the current "Basic networking" Guide:
https://github.com/containers/podman/blob/main/docs/tutorials/basic_netwo...
101
days inactive
138
days old
7 comments
3 participants
participants (3)
-
François-Xavier Thomas -
Keith H -
Paul Holzinger