Can't maintain connection to container's listening port
by Ranbir
Hello,
I have a rootless container running postgrey on a Rocky Linux 8 server.
Besides the fact I had problems getting the container running rootless,
which I overcame, the new issue is that connections to the exposed port
are established and then immediately dropped. I can't figure out why
this is happening.
Here's postgrey listening inside the container:
[containers@bigsecret ~]$ podman exec -ti postgrey ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:10023 0.0.0.0:*
I can connect to the port inside of the container and the connection
stays up until I cancel it:
[containers@bigsecret ~]$podman exec -ti postgrey telnet localhost 10023
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
If I try to telnet to the port from the container host using the host's
routable IP or from different server, I get a "Connection closed by
foreign host." message immediately after the connection is established.
I have systemd enabled in the container. I can control the postgrey
daemon with systemd and systemd doesn't report any errors when I check
the daemon's status.
I don't see any selinux denials. I tried turning off enforcement anyway
and saw no change. I did see language errors being logged by postgrey,
so I installed the missing RPMs in the running container (I'm just
testing things out with this container), which got rid of those errors.
But, that didn't change the connection weirdness.
Any ideas what the problem could be? The pod and container definitions
are below.
pod
{
"Id": "a9292128fc778c6287e80ff71d5e2ee1320b3395dc48a7e31af1db77cc7f695a",
"Name": "smtp",
"Created": "2021-11-25T12:58:55.447833371-05:00",
"CreateCommand": [
"podman",
"pod",
"create",
"--name",
"smtp",
"--publish",
"1.2.3.4:10023:10023",
"--publish",
"1.2.3.4:1587:587",
"--publish",
"1.2.3.4:1783:783",
"--publish",
"1.2.3.4:1025:25"
],
"State": "Running",
"Hostname": "smtp",
"CreateCgroup": true,
"CgroupParent": "user.slice",
"CgroupPath": "user.slice/user-libpod_pod_a9292128fc778c6287e80ff71d5e2ee1320b3395dc48a7e31af1db77cc7f695a.slice",
"CreateInfra": true,
"InfraContainerID": "a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4",
"InfraConfig": {
"PortBindings": {
"10023/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "10023"
}
],
"25/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1025"
}
],
"587/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1587"
}
],
"783/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1783"
}
]
},
"HostNetwork": false,
"StaticIP": "",
"StaticMAC": "",
"NoManageResolvConf": false,
"DNSServer": null,
"DNSSearch": null,
"DNSOption": null,
"NoManageHosts": false,
"HostAdd": null,
"Networks": null,
"NetworkOptions": null
},
"SharedNamespaces": [
"ipc",
"net",
"uts"
],
"NumContainers": 2,
"Containers": [
{
"Id": "a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4",
"Name": "a9292128fc77-infra",
"State": "running"
},
{
"Id": "f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57",
"Name": "postgrey",
"State": "running"
}
]
}
container
[
{
"Id": "f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57",
"Created": "2021-12-05T00:18:28.942285862-05:00",
"Path": "/usr/sbin/init",
"Args": [
"/usr/sbin/init"
],
"State": {
"OciVersion": "1.0.2-dev",
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6047,
"ConmonPid": 6031,
"ExitCode": 0,
"Error": "",
"StartedAt": "2021-12-22T14:32:26.339653403-05:00",
"FinishedAt": "2021-12-22T14:27:28.403171029-05:00",
"Healthcheck": {
"Status": "",
"FailingStreak": 0,
"Log": null
}
},
"Image": "9aefd5346e1f34b16a096b52575cc249b14a9a56664c6e1f2113ad3ef449c025",
"ImageName": "localhost/postgrey-v0.0.3:latest",
"Rootfs": "",
"Pod": "a9292128fc778c6287e80ff71d5e2ee1320b3395dc48a7e31af1db77cc7f695a",
"ResolvConfPath": "/tmp/podman-run-1000/containers/overlay-containers/a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4/userdata/resolv.conf",
"HostnamePath": "/tmp/podman-run-1000/containers/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata/hostname",
"HostsPath": "/tmp/podman-run-1000/containers/overlay-containers/a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4/userdata/hosts",
"StaticDir": "/srv/containers/storage/1000/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata",
"OCIConfigPath": "/srv/containers/storage/1000/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata/config.json",
"OCIRuntime": "runc",
"ConmonPidFile": "/tmp/podman-run-1000/containers/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata/conmon.pid",
"PidFile": "/tmp/podman-run-1000/containers/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata/pidfile",
"Name": "postgrey",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:container_file_t:s0:c654,c974",
"ProcessLabel": "system_u:system_r:container_init_t:s0:c654,c974",
"AppArmorProfile": "",
"EffectiveCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"BoundingCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/srv/containers/storage/1000/overlay/2a356f237c2fe380f476133e0939553512ac8167ff7cbb2338d9405090528f7e/diff:/srv/containers/storage/1000/overlay/04f97fe38f3ca40a0d4a7ee7f6da4276ab30746e05c360975bd2e3569afde128/diff:/srv/containers/storage/1000/overlay/4d50441def2b07f8fcd48aad187815089621ddccf2384180db0c28c5272889f8/diff:/srv/containers/storage/1000/overlay/7933807b1a3f6ecbc852d38f269984065dfb57d49ddf40fdea70dfe66a6c6b14/diff:/srv/containers/storage/1000/overlay/1855256707116c0c229fec2d3a60bce4a11fdfc8b0bffa9663c84e69ec326160/diff",
"MergedDir": "/srv/containers/storage/1000/overlay/113bb9169c33b29659143e14363c6a8fc07a7cd6a8ffc72697337a83200db18e/merged",
"UpperDir": "/srv/containers/storage/1000/overlay/113bb9169c33b29659143e14363c6a8fc07a7cd6a8ffc72697337a83200db18e/diff",
"WorkDir": "/srv/containers/storage/1000/overlay/113bb9169c33b29659143e14363c6a8fc07a7cd6a8ffc72697337a83200db18e/work"
}
},
"Mounts": [
{
"Type": "volume",
"Name": "postgrey",
"Source": "/srv/containers/storage/1000/volumes/postgrey/_data",
"Destination": "/var/spool/postfix/postgrey",
"Driver": "local",
"Mode": "Z",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "5e82bc179344af8710114ca61f84dbfe7a8866c8aac5fab6bcef70e6cba6df76",
"Source": "/srv/containers/storage/1000/volumes/5e82bc179344af8710114ca61f84dbfe7a8866c8aac5fab6bcef70e6cba6df76/_data",
"Destination": "/sys/fs/cgroup",
"Driver": "local",
"Mode": "",
"Options": [
"nodev",
"exec",
"nosuid",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
}
],
"Dependencies": [
"a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4"
],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"10023/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "10023"
}
],
"25/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1025"
}
],
"587/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1587"
}
],
"783/tcp": [
{
"HostIp": "1.2.3.4",
"HostPort": "1783"
}
]
},
"SandboxKey": "/run/user/1000/netns/cni-a2c22e7a-f19f-8320-fe77-9d44a822154d"
},
"ExitCommand": [
"/usr/bin/podman",
"--root",
"/srv/containers/storage/1000",
"--runroot",
"/tmp/podman-run-1000/containers",
"--log-level",
"warning",
"--cgroup-manager",
"systemd",
"--tmpdir",
"/tmp/run-1000/libpod/tmp",
"--runtime",
"runc",
"--storage-driver",
"overlay",
"--storage-opt",
"overlay.mount_program=/usr/bin/fuse-overlayfs",
"--events-backend",
"file",
"container",
"cleanup",
"f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57"
],
"Namespace": "",
"IsInfra": false,
"Config": {
"Hostname": "smtp",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"container=docker",
"HOME=/root",
"HOSTNAME=smtp"
],
"Cmd": [
"/usr/sbin/init"
],
"Image": "localhost/postgrey-v0.0.3:latest",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": {
"io.buildah.version": "1.21.3",
"org.label-schema.build-date": "20210620",
"org.label-schema.license": "BSD-3-Clause",
"org.label-schema.name": "Rocky Linux Base Image",
"org.label-schema.schema-version": "1.0",
"org.label-schema.vendor": "Rocky Enterprise Software Foundation",
"org.opencontainers.image.created": "2021-06-20 00:00:00+01:00",
"org.opencontainers.image.licenses": "BSD-3-Clause",
"org.opencontainers.image.title": "Rocky Linux Base Image",
"org.opencontainers.image.vendor": "Rocky Enterprise Software Foundation"
},
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.ContainerType": "container",
"io.kubernetes.cri-o.Created": "2021-12-05T00:18:28.942285862-05:00",
"io.kubernetes.cri-o.SandboxID": "smtp",
"io.kubernetes.cri-o.TTY": "true",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "FALSE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.stopSignal": "37"
},
"StopSignal": 37,
"CreateCommand": [
"podman",
"run",
"-d",
"-t",
"--name",
"postgrey",
"--pod",
"smtp",
"--volume",
"postgrey:/var/spool/postfix/postgrey:Z",
"postgrey-v0.0.3"
],
"SystemdMode": true,
"Umask": "0022",
"Timeout": 0,
"StopTimeout": 10
},
"HostConfig": {
"Binds": [
"postgrey:/var/spool/postfix/postgrey:Z,rw,rprivate,nosuid,nodev,rbind",
"5e82bc179344af8710114ca61f84dbfe7a8866c8aac5fab6bcef70e6cba6df76:/sys/fs/cgroup:rprivate,rw,nodev,exec,nosuid,rbind"
],
"CgroupManager": "systemd",
"CgroupMode": "private",
"ContainerIDFile": "",
"LogConfig": {
"Type": "k8s-file",
"Config": null,
"Path": "/srv/containers/storage/1000/overlay-containers/f32c676da8eb38f3e45bb8670e0d8330707fa3dfc216238e4f73bbe638d85a57/userdata/ctr.log",
"Tag": "",
"Size": "0B"
},
"NetworkMode": "container:a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [
"CAP_AUDIT_WRITE",
"CAP_MKNOD"
],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "container:a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [],
"Tmpfs": {},
"UTSMode": "container:a75ed34c8117daaff8be1e9060c07478b6894d4d06a93c963142d8b3de95b0a4",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "user.slice/user-libpod_pod_a9292128fc778c6287e80ff71d5e2ee1320b3395dc48a7e31af1db77cc7f695a.slice",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"CgroupConf": null
}
}
]
--
Ranbir
2 years, 11 months
TLS Handshake Error and Exit 125 error when pulling multiple images with Podman
by Christopher.Miller＠gd-ms.com
Not sure if anyone else has seen this. We use Podman to pull multiple images from a Quay registry for building software.
During this process, an image won't get pulled. We then see a TLS Handshake errors along with an exit-125 errors too.
Wondering if anyone has experienced this.
Thanks
2 years, 11 months
what replaces the -- link
by Michael Biarnes Kiefer
Hi,
I am new to podman.
I have this problem:
The Docker option *--link* is legacy and doesn't work any-more.
Using podman right now I saw that this option should be replaced
somehow by *--network
*or *--pod*.
The thing is I tried several things with --network or --pod but somehow two
different containers running (from two different images) can't communicate
to each other:
Before (when Docker --link still was available):
$ podman images
REPOSITORY TAG
IMAGE ID CREATED SIZE
localhost/kiegroup/kie-server-showcase
7.63.0.Final 7de306f0d498 20 minutes ago 1.03 GB
localhost/kiegroup/business-central-workbench-showcase 7.63.0.Final
446a00d242c2 10 days ago 1.11 GB
podman run -p 8083:8080 -p 8003:8001 -i -t --name business-central-wb
kiegroup/business-central-workbench-showcase:7.63.0.Final (runs the
446a00d242c2 container)
podman run -p 8088:8080 -i -t --name kieserver-wb --link
business-central-wb:kie-wb kiegroup/kie-server-showcase:7.63.0.Final (run
7de306f0d498 container linked to 446a00d242c2)
Both Containers could communicate and share different vars.
I tried to create a network and run both containers with --network
<createdNetwork> or I created a pod and run both containers with --pod
<created pod> option, but somehow
the running container doesn't share the needed values. I do podman inspect
and look to container values - and they are there as expected.
What do I wrong?
podman version 3.4.4
OS: Fedora 34
ThanX
Michael
--
Michael Biarnés Kiefer
Release engineer
EMEA ENG Product-Middleware
Red Hat SL Barcelona Spain <https://www.redhat.com>
<https://www.redhat.com>
2 years, 11 months
