Hi,
I created
https://github.com/containers/podman/issues/8741 and
https://github.com/containers/podman/issues/8742 for greater visibility
into this strange behavior.
Thanks,
brian
On Sun, Dec 13, 2020 at 12:03 AM Brian Fallik <bfallik(a)gmail.com> wrote:
Hi,
I must have jinxed myself when I emailed this list a few days ago about
how well Podman had been working for me. Earlier today I let Gnome Software
Center update my Fedora 33 system. After the update grafana alerted me
about an unreachable service and I confirmed that both of my podman
services had fallen off the network.
Podman runs two sets of containers on this machine:
* a Prometheus pod containing several containers for prometheus, grafana,
and nginx; the pod publishes port 443/tcp on the host ("-p 443")
* a CoreDNS container; this container exposes port 53/udp and 9153/tcp
("-p 10.100.10.5:53:53/udp -p 9153")
and suddenly none of these ports were accessible over network or even
locally on the host.
After some fumbling I realized that some of the ports weren't being
published like they used to:
# podman ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
fa71bff884bc docker.io/coredns/coredns:latest -conf
/root/Coref... 4 seconds ago Up 4 seconds ago 0.0.0.0:34595->9153/tcp
coredns
f034c62577a2 docker.io/prom/prometheus:latest
--config.file=/et... 12 hours ago Up 12 hours ago 0.0.0.0:37683->443/tcp
prometheus
You can see that podman is listening on 34595 instead of 9153. This port
seems to be randomly assigned each time I restart the container.
I can workaround the above TCP issue by specifying both src and dest
ports, e.g. "-p 9153:9153". I scanned the recent release notes, open github
issues, and some docs but can't understand why "-p 9153" suddenly stopped
working like it had been. Any ideas?
The bigger problem is that the UDP port for DNS is completely broken. I
intentionally publish 53 to a specific IP so that CoreDNS only handles
lookups from the external interface but "-p 10.100.10.5:53:53" doesn't
work anymore:
# dig @10.100.10.5 coredns.io
...
;; connection timed out; no servers could be reached
and I don't see any evidence of the UDP mapping at all in podman or
netstat:
# netstat -aun | grep 10.100.10.5
udp 0 0 10.100.10.5:68 10.100.10.1:67
ESTABLISHED
udp 0 0 10.100.10.5:41443 172.217.10.227:443
ESTABLISHED
udp 0 0 10.100.10.5:58091 142.250.64.106:443
ESTABLISHED
udp 0 0 10.100.10.5:46088 142.250.64.110:443
ESTABLISHED
udp 0 0 10.100.10.5:58834 172.217.197.189:443
ESTABLISHED
# podman port -a | grep -v tcp
#
I'm not 100% either of these commands would be expected to show the UDP
mapping. But DNS lookups are broken and I don't know how to fix this.
I'm not sure what was upgraded earlier today that might have caused this
behavior change. I also haven't seen any relevant errors in any of the
obvious logs.
# podman --version
podman version 2.2.1
Any help would be appreciated!
Thx,
brian