February 2023 - Podman - Podman List Archives

Overriding run root "/tmp/containers-user-1253/containers" with "/home/myuser/rundir/containers" from database

by GHui Wu

I set "export XDG_RUNTIME_DIR=", so it will use /tmp/containers-user-1253/containers. But sometimes it'll use /home/myuser/rundir/containers. In debug log, there is "Overriding run root "/tmp/containers-user-1253/containers" with "/home/myuser/rundir/containers" from database". Why is this happened. time="2023-02-14T16:05:51+08:00" level=debug msg="Initializing boltdb state at /tmp/1253/share/containers/storage/libpod/bolt_state.db" time="2023-02-14T16:05:51+08:00" level=debug msg="Using conmon from $PATH: "/home/myuser/.local/container/bin/conmon"" time="2023-02-14T16:05:51+08:00" level=debug msg="Initializing boltdb state at /tmp/1253/share/containers/storage/libpod/bolt_state.db" time="2023-02-14T16:05:51+08:00" level=debug msg="Overriding run root "/tmp/containers-user-1253/containers" with "/home/myuser/rundir/containers" from database" time="2023-02-14T16:05:51+08:00" level=debug msg="Using graph driver overlay" time="2023-02-14T16:05:51+08:00" level=debug msg="Using graph root /tmp/1253/share/containers/storage" time="2023-02-14T16:05:51+08:00" level=debug msg="Using run root /home/myuser/rundir/containers" time="2023-02-14T16:05:51+08:00" level=debug msg="Using static dir /tmp/1253/share/containers/storage/libpod" time="2023-02-14T16:05:51+08:00" level=debug msg="Using tmp dir /tmp/podman-run-1253/libpod/tmp" time="2023-02-14T16:05:51+08:00" level=debug msg="Using volume path /tmp/1253/share/containers/storage/volumes" time="2023-02-14T16:05:51+08:00" level=info msg="podman filtering at log level trace"

3 weeks, 2 days

3
5
0 / 0

Error: container creation timeout: internal libpod error

by GHui Wu

When I run the following command to start container, there is error "Error: container creation timeout: internal libpod error". podman --log-level=debug run --rm --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host --privileged -v ./Dmyjob:/Dmyjob:z -w /Dmyjob abacus:3.1.0 ./myjob.sh The error log as following. Error: container creation timeout: internal libpod error Error: container creation timeout: internal libpod error Error: container creation timeout: internal libpod error Error: container creation timeout: internal libpod error time="2023-01-24T11:15:16+08:00" level=warning msg="cannot toggle freezer: cgroups not configured for container" time="2023-01-24T11:15:16+08:00" level=warning msg="cannot toggle freezer: cgroups not configured for container" time="2023-01-24T11:15:16+08:00" level=warning msg="lstat : no such file or directory" time="2023-01-24T11:15:16+08:00" level=warning msg="Failed to remove cgroup (will retry)" error="rmdir /sys/fs/cgroup/cpuset/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f: device or resource busy" time="2023-01-24T11:15:16+08:00" level=warning msg="Failed to remove cgroup (will retry)" error="rmdir /sys/fs/cgroup/memory/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f: device or resource busy" time="2023-01-24T11:15:16+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/cpuset/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f: device or resource busy" time="2023-01-24T11:15:16+08:00" level=error msg="Failed to remove cgroup" error="rmdir /sys/fs/cgroup/memory/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f: device or resource busy" time="2023-01-24T11:15:16+08:00" level=error msg="Failed to remove paths: map[cpuset:/sys/fs/cgroup/cpuset/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f memory:/sys/fs/cgroup/memory/slurm/uid_7809/job_2186496/step_3/7be1f8e2a2af45e2fdff36b7a845a122986699a623891f6b13f73e331d6ea96f]" time="2023-01-24T11:15:16+08:00" level=error msg="container not running" time="2023-01-24T11:15:16+08:00" level=error msg="forwarding signal 15 to container 3adc0ebf2a42635b72d586e359422038bf1a05b10e908a22c8409dbe2868bd9b: error sending signal to container 3adc0ebf2a42635b72d586e359422038bf1a05b10e908a22c8409dbe2868bd9b: `/public3/home/scb6724/.local/container/sbin/runc kill 3adc0ebf2a42635b72d586e359422038bf1a05b10e908a22c8409dbe2868bd9b 15` failed: exit status 1"

4 weeks

1
0
0 / 0

Error: failed to connect to container's attach socket: /xxxxx/attach: no such file or directory

by GHui Wu

I start podman as following command. podman --log-level=debug run --rm --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host --privileged -v ./Dmyjob:/Dmyjob:z -w /Dmyjob abacus:3.1.0 ./myjob.sh But sometimes it output error "Error: failed to connect to container's attach socket: /xxxxx/attach: no such file or directory". time="2023-02-24T15:56:41+08:00" level=debug msg="Successfully cleaned up container ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4" time="2023-02-24T15:56:41+08:00" level=error msg="Storage for container ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4 has been removed" time="2023-02-24T15:56:41+08:00" level=debug msg="Removing all exec sessions for container ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4" time="2023-02-24T15:56:41+08:00" level=debug msg="Container ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4 storage is already unmounted, skipping..." time="2023-02-24T15:56:41+08:00" level=info msg="Storage for container ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4 already removed" time="2023-02-24T15:56:41+08:00" level=debug msg="ExitCode msg: \"failed to connect to container's attach socket: /export/myuser/11727/1253/share/containers/storage/overlay-containers/ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4/userdata/attach: no such file or directory\"" Error: failed to connect to container's attach socket: /export/myuser/11727/1253/share/containers/storage/overlay-containers/ab2d81ec95dd0025a588ad016fb2b79db8c4e1617f0b808f756a3d25d4fca2a4/userdata/attach: no such file or directory

4 weeks

1
0
0 / 0

runc start b714ab8d87af579aea314ee36964e49a807ab5025099ff3b3e140df715bf914a failed: exit status 1

by GHui Wu

I start podman as following command. podman --log-level=debug run --rm --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host --privileged -v ./Dmyjob:/Dmyjob:z -w /Dmyjob abacus:3.1.0 ./myjob.sh But sometimes it output error "runc start xxxxx failed: exit status 1". time="2023-02-24T14:29:21+08:00" level=error msg="remove /export/myuser/14234/podman-run-1253/runc/b714ab8d87af579aea314ee36964e49a807ab5025099ff3b3e140df715bf914a/exec.fifo: no such file or directory" time="2023-02-24T14:29:21+08:00" level=error msg="remove /export/myuser/14234/podman-run-1253/runc/58a04ffff99f6769136a36676ff69e3631f29e8353fc155b4a211c6a496ab58d/exec.fifo: no such file or directory" time="2023-02-24T14:29:21+08:00" level=debug msg="unable to remove container b714ab8d87af579aea314ee36964e49a807ab5025099ff3b3e140df715bf914a after failing to start and attach to it" time="2023-02-24T14:29:21+08:00" level=debug msg="unable to remove container 58a04ffff99f6769136a36676ff69e3631f29e8353fc155b4a211c6a496ab58d after failing to start and attach to it" time="2023-02-24T14:29:21+08:00" level=debug msg="ExitCode msg: \"`/export/home/myuser/.local/container/sbin/runc start b714ab8d87af579aea314ee36964e49a807ab5025099ff3b3e140df715bf914a` failed: exit status 1\"" time="2023-02-24T14:29:21+08:00" level=debug msg="ExitCode msg: \"`/export/home/myuser/.local/container/sbin/runc start 58a04ffff99f6769136a36676ff69e3631f29e8353fc155b4a211c6a496ab58d` failed: exit status 1\"" Error: `/export/home/myuser/.local/container/sbin/runc start 58a04ffff99f6769136a36676ff69e3631f29e8353fc155b4a211c6a496ab58d` failed: exit status 1 Error: `/export/home/myuser/.local/container/sbin/runc start b714ab8d87af579aea314ee36964e49a807ab5025099ff3b3e140df715bf914a` failed: exit status 1

4 weeks

1
0
0 / 0

Container Plumbing Days Conference - March 22-23, 2023, 9:00 -> 15:00 EST (UTC-4)

by Tom Sweeney

Hi All, On March 22-23, 2023, from 9:00 a.m. to 3:00 p.m. EST (UTC-4), the Container Plumbing Days Conference will be running virtually. It's free to join, and there are a number of great talks. The schedule is posted, and registration is now open at: https://containerplumbing.org/register <https://t.co/9XwPcKZPGi> Hope to see you there! t

4 weeks

1
0
0 / 0

ERRO[0000] stat /tmp/22359: no such file or directory

by GHui Wu

I have set "XDG_RUNTIME_DIR=/tmp/22359". But when I run "podman images", there is error "ERRO[0000] stat /tmp/22359: no such file or directory".

4 weeks, 1 day

3
4
0 / 0

Problems with routing in rootless podman

by Henrik Jacobsson

Hello. We are running our application in rootless podman. After some random time (a couple of hours - a couple of weeks), we lose the network connectivity into the container. Everything seems to work fine from inside the container to the rest of the world (yum/dnf, ping, curl), but it looks like the routing stops working when someone calls from the outside. I set up a netcat listener (nc -lv), and called it on localhost (worked fine) and on the tap-interface (long delays if the packet ever returned). I also set up a tcpdump in a third screen – output below. bash-4.4$ podman --version podman version 4.2.0 bash-4.4$ uname -a Linux podman-container 5.4.17-2136.315.5.el8uek.x86_64 #2 SMP Wed Dec 21 19:38:18 PST 2022 x86_64 x86_64 x86_64 GNU/Linux bash-4.4$ cat /etc/os-release NAME="Oracle Linux Server" VERSION="8.7" ID="ol" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Oracle Linux Server 8.7" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:oracle:linux:8:7:server" HOME_URL="https://linux.oracle.com/" BUG_REPORT_URL="https://bugzilla.oracle.com/" ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8" ORACLE_BUGZILLA_PRODUCT_VERSION=8.7 ORACLE_SUPPORT_PRODUCT="Oracle Linux" ORACLE_SUPPORT_PRODUCT_VERSION=8.7 # Testing communication using 'localhost' inside the container - works as expected [root@NC-Test_podman-container /]# nc -lv 10370 Listening on 0.0.0.0 10370 Connection received on localhost 47218 ping from server ping from client [root@NC-Test_podman-container /]# nc -v localhost 10370 nc: connect to localhost (::1) port 10370 (tcp) failed: Connection refused Connection to localhost (127.0.0.1) 10370 port [tcp/*] succeeded! ping from server ping from client # Testing communication using hostname - "some" packets arrives, but only after a random delay of about 30-600 seconds [root@NC-Test_podman-container /]# nc -lv 10370 Listening on 0.0.0.0 10370 server Connection received on podman-container 59258 client [root@NC-Test_podman-container /]# nc -v podman-container 10370 Connection to podman-container (10.11.12.102) 10370 port [tcp/*] succeeded! client server [root@NC-Test_podman-container base_domain]# tcpdump -vv -X host podman-container and port 10370 dropped privs to tcpdump tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:41:49.080602 IP (tos 0x0, ttl 64, id 61404, offset 0, flags [DF], proto TCP (6), length 47) podman-container.56372 > podman-container-oob.10370: Flags [P.], cksum 0xdb21 (correct), seq 2129174302:2129174309, ack 1071210498, win 65480, length 7 0x0000: 4500 002f efdc 4000 4006 7df1 0a00 0264 E../..@.@.}....d 0x0010: 0a31 b666 dc34 2882 7ee8 9f1e 3fd9 6002 .1.f.4(.~...?.`. 0x0020: 5018 ffc8 db21 0000 636c 6965 6e74 0a P....!..client. 12:41:49.080783 IP (tos 0x0, ttl 64, id 48821, offset 0, flags [none], proto TCP (6), length 40) podman-container-oob.10370 > podman-container.56372: Flags [.], cksum 0x2039 (correct), seq 1, ack 7, win 65535, length 0 0x0000: 4500 0028 beb5 0000 4006 ef1f 0a31 b666 E..(....@....1.f 0x0010: 0a00 0264 2882 dc34 3fd9 6002 7ee8 9f25 ...d(..4?.`.~..% 0x0020: 5010 ffff 2039 0000 P....9.. 12:42:28.673431 IP (tos 0x0, ttl 64, id 49091, offset 0, flags [none], proto TCP (6), length 40) podman-container-oob.10370 > podman-container.51394: Flags [F.], cksum 0xf92e (correct), seq 946730519, ack 2284994989, win 65535, length 0 0x0000: 4500 0028 bfc3 0000 4006 ee11 0a31 b666 E..(....@....1.f 0x0010: 0a00 0264 2882 c8c2 386d f617 8832 41ad ...d(...8m...2A. 0x0020: 5011 ffff f92e 0000 P....... 12:42:28.673436 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) podman-container.51394 > podman-container-oob.10370: Flags [R], cksum 0x27c1 (correct), seq 2284994989, win 0, length 0 0x0000: 4500 0028 0000 4000 4006 6dd5 0a00 0264 E..(..@.@.m....d 0x0010: 0a31 b666 c8c2 2882 8832 41ad 0000 0000 .1.f..(..2A..... 0x0020: 5004 0000 27c1 0000 P...'... 12:44:28.693154 IP (tos 0x0, ttl 64, id 49943, offset 0, flags [none], proto TCP (6), length 47) podman-container-oob.10370 > podman-container.56372: Flags [P.], cksum 0xcadb (correct), seq 1:8, ack 7, win 65535, length 7 0x0000: 4500 002f c317 0000 4006 eab6 0a31 b666 E../....@....1.f 0x0010: 0a00 0264 2882 dc34 3fd9 6002 7ee8 9f25 ...d(..4?.`.~..% 0x0020: 5018 ffff cadb 0000 7365 7276 6572 0a P.......server. 12:44:28.693174 IP (tos 0x0, ttl 64, id 61405, offset 0, flags [DF], proto TCP (6), length 40) podman-container.56372 > podman-container-oob.10370: Flags [.], cksum 0x2070 (correct), seq 7, ack 8, win 65473, length 0 0x0000: 4500 0028 efdd 4000 4006 7df7 0a00 0264 E..(..@.@.}....d 0x0010: 0a31 b666 dc34 2882 7ee8 9f25 3fd9 6009 .1.f.4(.~..%?.`. 0x0020: 5010 ffc1 2070 0000 P....p.. Kind regards //Henrik

1 month

2
1
0 / 0

Error: OCI runtime error: runc: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF

by GHui Wu

When I execute "podman run --rm --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host --privileged -w /workspace xxximage ./xxx.sh", sometimes there is error. Error: Storage for container 53a7dd0ae112ebf4d1f24c9c9cc06f0925c684c872770c7ea65f87762094f2df has been removed" Error: OCI runtime error: runc: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF

1 month

1
0
0 / 0

How to run podman-compose with --root flag?

by Koodavalli, Subhodh (Subhodh K N)

Hi, Goal: The goal is to be able to run podman-compose and point it to different storage locations. Example: podman-compose up -root /app1/containers/storage -d -f docker-compose-1.yml podman-compose up -root /app2/containers/storage -d -f docker-compose-2.yml podman-compose up -root /app3/containers/storage -d -f docker-compose-3.yml I noticed that only the podman command has the -root flag. How do I go about achieving this behavior using compose? Thanks, Subhodh K N

1 month

1
0
0 / 0

Storage for container c1a8d39e5276909827451b1f53c000b5c644a3266824416a55bc381a3d84abc1 has been removed

by GHui Wu

When I run "podman run --rm --env-host --userns=keep-id --network=host --pid=host --ipc=host --privileged xxximage xxx.sh", there is output the following error. time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container c1a8d39e5276909827451b1f53c000b5c644a3266824416a55bc381a3d84abc1 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container c2a4a75804a56bba5c10c90d60a35bb21a7f55ee3ffa9e078bc672817e7d533c has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container f41301358da8715e9443c075e7e59a73d85c5188c1dbccf7e2d7722fd059b43d has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container b4e4d087ba7726bda471e1b912efa9a64aedaf39bb2fa3e41bdc915488a40542 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 798a62751f66f10bbf4e44fa83d00a7e2b78d8036045271009cec1245d6f0464 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 935464db3cab0840b9eafb98035aae29912dc531a12d626fd97804b1575c1e8c has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 8dcf505273da9f8a487993bb80a676e070784a896667aa47980ee3b75725212a has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 23abed966bf082f454bf926cbe2af386e5698d3a1e9ca961c70fa66c0a0d2dab has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 6275e312a951db5e091c5648ffa3f2e5b88b0a55b696d6b83c55bf6144e953b1 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container d9760b91ed141e12dc326d1f2920af8a84d85cb080b7b9fedafcec61444b43ee has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 389f6e50edf5f487b8386d182b89521bda58aaab9d6565409ad576597df6f001 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container ec8d4d7a2a09f4d217f13ae15bc3d594e406dde65b6cd25de7f7803167679624 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container a13e0f3d5e94c70c08a475f27cb9becf84495d796e0aa6365ce81dd2dbbdf0f7 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container edab35249b42a319512b88b6a5df2e16b513a2ba0ccb02354a94db50080a21f1 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container d6d853edf39618831cf3a1558c9dd1cdc6e9c8a366f3fc1c34b9ec00ace8cab8 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container 4fd8d85ba941fa0b3d32322fd9601791e2337603abf07b32aa540843cfe30811 has been removed" time="2023-02-15T10:59:55+08:00" level=error msg="Storage for container b81b9c20ab89d7063cd2bcca9eff156f6dbe3923b273607b6470ef232b8647ec has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 888c4f5fe0fa25f659f791b61c26c48a1c225cc06b0305fdb23d81dcf1810c84 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 20b8412411b019eb9a002a8c65477ebf793409a87b88df5fb539035b096593ff has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 34d30f095dd26943c9ef4bc0d7ec733c97eb0ae9af2a1bbf0476882c74b4945d has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 6e5a5218bb705b1448bed6732374ac1188dd66266bb427efef3d3c03397968d3 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 65eabb6181d89b027c9c4a1b36121fc3bd09bd463935a529be87a58c952f972e has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 139e049b7dc7bc1ff73aae79ebed110659981907e7de4ec45f952928339f724f has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 2152ff8e0adbbf04309219b6482eec1b62a50390da54a5bfe31212c20b29a9e5 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 60219f2cf9fc8ebf6130150f84385cec76ae47b8c40f0121621ebf0285be2b28 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container f50eea42d5a84f3d4967d9fefb9c6943060bcd9fa131bf28d4fffc6316ced1bc has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 2de2d56e8c6b58532d6d1f04b8ec87156f32ded1679ea9ddf0db90e8f0bd9da3 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container a37e787e54bb729139cc32ce635bc32fa36fca1afb17faffae8179042ab468bd has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 064e09108a357948539d11f6d5a771d0f5e3298c0240e0a59198eb447ae5a897 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container c5ee48c7bce0a4adb531b141f2b4c6f8d1caf603b3af1da1df73a5b6616c6c69 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 3a49cde1a7089672934c23f14117b9f1b84dfa561df1abac50460148221ad953 has been removed" time="2023-02-15T10:59:56+08:00" level=error msg="Storage for container 51e60a1704152607aaf9ed4b3d043042665f1bb33a1b6aa1fd79a18bb418e3b5 has been removed"

1 month

2
2
0 / 0

2023

2022

2021

2020

2019

Podman February 2023