12:03 a.m.
Hi,
I must have jinxed myself when I emailed this list a few days ago about how
well Podman had been working for me. Earlier today I let Gnome Software
Center update my Fedora 33 system. After the update grafana alerted me
about an unreachable service and I confirmed that both of my podman
services had fallen off the network.
Podman runs two sets of containers on this machine:
* a Prometheus pod containing several containers for prometheus, grafana,
and nginx; the pod publishes port 443/tcp on the host ("-p 443")
* a CoreDNS container; this container exposes port 53/udp and 9153/tcp
("-p 10.100.10.5:53:53/udp -p 9153")
and suddenly none of these ports were accessible over network or even
locally on the host.
After some fumbling I realized that some of the ports weren't being
published like they used to:
# podman ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
fa71bff884bc docker.io/coredns/coredns:latest -conf
/root/Coref... 4 seconds ago Up 4 seconds ago 0.0.0.0:34595->9153/tcp
coredns
f034c62577a2 docker.io/prom/prometheus:latest
--config.file=/et... 12 hours ago Up 12 hours ago 0.0.0.0:37683->443/tcp
prometheus
You can see that podman is listening on 34595 instead of 9153. This port
seems to be randomly assigned each time I restart the container.
I can workaround the above TCP issue by specifying both src and dest ports,
e.g. "-p 9153:9153". I scanned the recent release notes, open github
issues, and some docs but can't understand why "-p 9153" suddenly stopped
working like it had been. Any ideas?
The bigger problem is that the UDP port for DNS is completely broken. I
intentionally publish 53 to a specific IP so that CoreDNS only handles
lookups from the external interface but "-p 10.100.10.5:53:53" doesn't work
anymore:
# dig @10.100.10.5 coredns.io
...
;; connection timed out; no servers could be reached
and I don't see any evidence of the UDP mapping at all in podman or netstat:
# netstat -aun | grep 10.100.10.5
udp 0 0 10.100.10.5:68 10.100.10.1:67
ESTABLISHED
udp 0 0 10.100.10.5:41443 172.217.10.227:443
ESTABLISHED
udp 0 0 10.100.10.5:58091 142.250.64.106:443
ESTABLISHED
udp 0 0 10.100.10.5:46088 142.250.64.110:443
ESTABLISHED
udp 0 0 10.100.10.5:58834 172.217.197.189:443
ESTABLISHED
# podman port -a | grep -v tcp
#
I'm not 100% either of these commands would be expected to show the UDP
mapping. But DNS lookups are broken and I don't know how to fix this.
I'm not sure what was upgraded earlier today that might have caused this
behavior change. I also haven't seen any relevant errors in any of the
obvious logs.
# podman --version
podman version 2.2.1
Any help would be appreciated!
Thx,
brian