Hi,

I must have jinxed myself when I emailed this list a few days ago about how well Podman had been working for me. Earlier today I let Gnome Software Center update my Fedora 33 system. After the update grafana alerted me about an unreachable service and I confirmed that both of my podman services had fallen off the network.

Podman runs two sets of containers on this machine:

 * a Prometheus pod containing several containers for prometheus, grafana, and nginx; the pod publishes port 443/tcp on the host ("-p 443")

 * a CoreDNS container; this container exposes port 53/udp and 9153/tcp ("-p 10.100.10.5:53:53/udp -p 9153")

and suddenly none of these ports were accessible over network or even locally on the host.

After some fumbling I realized that some of the ports weren't being published like they used to:

  # podman ps
  CONTAINER ID  IMAGE                                     COMMAND               CREATED        STATUS            PORTS                    NAMES
  fa71bff884bc  docker.io/coredns/coredns:latest          -conf /root/Coref...  4 seconds ago  Up 4 seconds ago  0.0.0.0:34595->9153/tcp  coredns
  f034c62577a2  docker.io/prom/prometheus:latest          --config.file=/et...  12 hours ago   Up 12 hours ago   0.0.0.0:37683->443/tcp   prometheus

You can see that podman is listening on 34595 instead of 9153. This port seems to be randomly assigned each time I restart the container.

I can workaround the above TCP issue by specifying both src and dest ports, e.g. "-p 9153:9153". I scanned the recent release notes, open github issues, and some docs but can't understand why "-p 9153" suddenly stopped working like it had been. Any ideas?

The bigger problem is that the UDP port for DNS is completely broken. I intentionally publish 53 to a specific IP so that CoreDNS only handles lookups from the external interface but "-p 10.100.10.5:53:53" doesn't work anymore:

  # dig @10.100.10.5 coredns.io

  ...

  ;; connection timed out; no servers could be reached

and I don't see any evidence of the UDP mapping at all in podman or netstat:

  # netstat -aun | grep 10.100.10.5
  udp        0      0 10.100.10.5:68          10.100.10.1:67          ESTABLISHED
  udp        0      0 10.100.10.5:41443       172.217.10.227:443      ESTABLISHED
  udp        0      0 10.100.10.5:58091       142.250.64.106:443      ESTABLISHED
  udp        0      0 10.100.10.5:46088       142.250.64.110:443      ESTABLISHED
  udp        0      0 10.100.10.5:58834       172.217.197.189:443     ESTABLISHED
  # podman port -a | grep -v tcp

  #

I'm not 100% either of these commands would be expected to show the UDP mapping. But DNS lookups are broken and I don't know how to fix this.

I'm not sure what was upgraded earlier today that might have caused this behavior change. I also haven't seen any relevant errors in any of the obvious logs.

  # podman --version
  podman version 2.2.1

Any help would be appreciated!

Thx,

brian