broken networking for published ports
Hi, I must have jinxed myself when I emailed this list a few days ago about how well Podman had been working for me. Earlier today I let Gnome Software Center update my Fedora 33 system. After the update grafana alerted me about an unreachable service and I confirmed that both of my podman services had fallen off the network. Podman runs two sets of containers on this machine: * a Prometheus pod containing several containers for prometheus, grafana, and nginx; the pod publishes port 443/tcp on the host ("-p 443") * a CoreDNS container; this container exposes port 53/udp and 9153/tcp ("-p 10.100.10.5:53:53/udp -p 9153") and suddenly none of these ports were accessible over network or even locally on the host. After some fumbling I realized that some of the ports weren't being published like they used to: # podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fa71bff884bc docker.io/coredns/coredns:latest -conf /root/Coref... 4 seconds ago Up 4 seconds ago 0.0.0.0:34595->9153/tcp coredns f034c62577a2 docker.io/prom/prometheus:latest --config.file=/et... 12 hours ago Up 12 hours ago 0.0.0.0:37683->443/tcp prometheus You can see that podman is listening on 34595 instead of 9153. This port seems to be randomly assigned each time I restart the container. I can workaround the above TCP issue by specifying both src and dest ports, e.g. "-p 9153:9153". I scanned the recent release notes, open github issues, and some docs but can't understand why "-p 9153" suddenly stopped working like it had been. Any ideas? The bigger problem is that the UDP port for DNS is completely broken. I intentionally publish 53 to a specific IP so that CoreDNS only handles lookups from the external interface but "-p 10.100.10.5:53:53" doesn't work anymore: # dig @10.100.10.5 coredns.io ... ;; connection timed out; no servers could be reached and I don't see any evidence of the UDP mapping at all in podman or netstat: # netstat -aun | grep 10.100.10.5 udp 0 0 10.100.10.5:68 10.100.10.1:67 ESTABLISHED udp 0 0 10.100.10.5:41443 172.217.10.227:443 ESTABLISHED udp 0 0 10.100.10.5:58091 142.250.64.106:443 ESTABLISHED udp 0 0 10.100.10.5:46088 142.250.64.110:443 ESTABLISHED udp 0 0 10.100.10.5:58834 172.217.197.189:443 ESTABLISHED # podman port -a | grep -v tcp # I'm not 100% either of these commands would be expected to show the UDP mapping. But DNS lookups are broken and I don't know how to fix this. I'm not sure what was upgraded earlier today that might have caused this behavior change. I also haven't seen any relevant errors in any of the obvious logs. # podman --version podman version 2.2.1 Any help would be appreciated! Thx, brian
Hi, I created https://github.com/containers/podman/issues/8741 and https://github.com/containers/podman/issues/8742 for greater visibility into this strange behavior. Thanks, brian On Sun, Dec 13, 2020 at 12:03 AM Brian Fallik <bfallik@gmail.com> wrote:
Hi,
I must have jinxed myself when I emailed this list a few days ago about how well Podman had been working for me. Earlier today I let Gnome Software Center update my Fedora 33 system. After the update grafana alerted me about an unreachable service and I confirmed that both of my podman services had fallen off the network.
Podman runs two sets of containers on this machine: * a Prometheus pod containing several containers for prometheus, grafana, and nginx; the pod publishes port 443/tcp on the host ("-p 443") * a CoreDNS container; this container exposes port 53/udp and 9153/tcp ("-p 10.100.10.5:53:53/udp -p 9153") and suddenly none of these ports were accessible over network or even locally on the host.
After some fumbling I realized that some of the ports weren't being published like they used to: # podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fa71bff884bc docker.io/coredns/coredns:latest -conf /root/Coref... 4 seconds ago Up 4 seconds ago 0.0.0.0:34595->9153/tcp coredns f034c62577a2 docker.io/prom/prometheus:latest --config.file=/et... 12 hours ago Up 12 hours ago 0.0.0.0:37683->443/tcp prometheus You can see that podman is listening on 34595 instead of 9153. This port seems to be randomly assigned each time I restart the container.
I can workaround the above TCP issue by specifying both src and dest ports, e.g. "-p 9153:9153". I scanned the recent release notes, open github issues, and some docs but can't understand why "-p 9153" suddenly stopped working like it had been. Any ideas?
The bigger problem is that the UDP port for DNS is completely broken. I intentionally publish 53 to a specific IP so that CoreDNS only handles lookups from the external interface but "-p 10.100.10.5:53:53" doesn't work anymore: # dig @10.100.10.5 coredns.io ... ;; connection timed out; no servers could be reached and I don't see any evidence of the UDP mapping at all in podman or netstat: # netstat -aun | grep 10.100.10.5 udp 0 0 10.100.10.5:68 10.100.10.1:67 ESTABLISHED udp 0 0 10.100.10.5:41443 172.217.10.227:443 ESTABLISHED udp 0 0 10.100.10.5:58091 142.250.64.106:443 ESTABLISHED udp 0 0 10.100.10.5:46088 142.250.64.110:443 ESTABLISHED udp 0 0 10.100.10.5:58834 172.217.197.189:443 ESTABLISHED # podman port -a | grep -v tcp # I'm not 100% either of these commands would be expected to show the UDP mapping. But DNS lookups are broken and I don't know how to fix this.
I'm not sure what was upgraded earlier today that might have caused this behavior change. I also haven't seen any relevant errors in any of the obvious logs. # podman --version podman version 2.2.1
Any help would be appreciated!
Thx, brian
participants (1)
-
Brian Fallik