What causes 'invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process'?
Hello all, I see the message 'invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process' quite often upon executing podman (rootless). In the net I could only find that this can happen after a host reboot. But in my case there is no reboot, so I am curious wich conditions actually have to be met for this message to appear. Yes, after "podman system migrate", I can use podman again, but it might occur just a minute later, or a day. The only observation I had besides the error message is, that after that "podman system migrate" I still see a couple of containers in state "Stopping". With those, podman rm -f always complains that it cannot kill the proces, but at least it can remove the containers. I'm curious if I could prevent that "invalid internal status" completely (because it disturbs my automation), but then I would need somebody explaining to me why this does actually happen. Any hints? -dirk
On 7/12/24 09:13, Dirk Försterling via Podman wrote:
Hello all,
I see the message 'invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process' quite often upon executing podman (rootless).
In the net I could only find that this can happen after a host reboot. But in my case there is no reboot, so I am curious wich conditions actually have to be met for this message to appear. Yes, after "podman system migrate", I can use podman again, but it might occur just a minute later, or a day. The only observation I had besides the error message is, that after that "podman system migrate" I still see a couple of containers in state "Stopping". With those, podman rm -f always complains that it cannot kill the proces, but at least it can remove the containers.
I'm curious if I could prevent that "invalid internal status" completely (because it disturbs my automation), but then I would need somebody explaining to me why this does actually happen. Any hints?
-dirk _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
podman system migrate recreates the default rootless user's user namespace. This looks like something is collapsing for some reason. Changing of /etc/subuid and /etc/subgid can require a system migrate. What version of Podman are you using? Output of podman info would be helpful. Usually questions like this are best handled within a github issue.
Thanks for your reply. I wasn't aware that this might be a podman issue. I just wanted to know the possible circumstances/reasons that can cause that behavior, so I can see how I can stop it. The podman version is the RHEL8 packaged version (see podman info output below). The podman system migrate requirement pops up randomly with no reboot and no modification of /etc/subuid or /etc subgid. I just want to understand what else podman has to detect in order to issue this message and refuse further action. -dirk podman info host: arch: amd64 buildahVersion: 1.33.8 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.10-1.module+el8.10.0+21962+8143777b.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: 8f1552a71b76511fb9b4a96282513d17686c56fc' cpuUtilization: idlePercent: 97.87 systemPercent: 0.61 userPercent: 1.52 cpus: 2 databaseBackend: boltdb distribution: distribution: rhel version: "8.10" eventLogger: file freeLocks: 2048 hostname: pipeline-agent idMappings: gidmap: - container_id: 0 host_id: 1003 size: 1 - container_id: 1 host_id: 231072 size: 65536 uidmap: - container_id: 0 host_id: 1002 size: 1 - container_id: 1 host_id: 231072 size: 65536 kernel: 4.18.0-553.8.1.el8_10.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 406753280 memTotal: 8071684096 networkBackend: cni networkBackendInfo: backend: cni dns: package: podman-plugins-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64 path: /usr/libexec/cni/dnsname version: |- CNI dnsname plugin version: 1.4.0-dev commit: unknown CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0 package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21962+8143777b.x86_64 path: /usr/libexec/cni ociRuntime: name: runc package: runc-1.1.12-1.module+el8.10.0+21974+acd2159c.x86_64 path: /usr/bin/runc version: |- runc version 1.1.12 spec: 1.0.2-dev go: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198) libseccomp: 2.5.2 os: linux pasta: executable: "" package: "" version: "" remoteSocket: exists: false path: /run/user/1002/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.3-1.module+el8.10.0+21962+8143777b.x86_64 version: |- slirp4netns version 1.2.3 commit: c22fde291bb35b354e6ca44d13be181c76a0a432 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.2 swapFree: 0 swapTotal: 0 uptime: 301h 24m 36.00s (Approximately 12.54 days) variant: "" plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.access.redhat.com - registry.redhat.io - docker.io store: configFile: /home/t01a01/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /home/t01a01/.local/share/containers/storage graphRootAllocated: 33275510784 graphRootUsed: 28726689792 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "false" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 8 runRoot: /tmp/containers-user-1002/containers transientStore: false volumePath: /home/t01a01/.local/share/containers/storage/volumes version: APIVersion: 4.9.4-rhel Built: 1718703254 BuiltTime: Tue Jun 18 09:34:14 2024 GitCommit: "" GoVersion: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198) Os: linux OsArch: linux/amd64 Version: 4.9.4-rhel [t01a01@pipeline-agent:dev ~]$ podman info | sed s/t01a01/podman01/g host: arch: amd64 buildahVersion: 1.33.8 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.10-1.module+el8.10.0+21962+8143777b.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: 8f1552a71b76511fb9b4a96282513d17686c56fc' cpuUtilization: idlePercent: 97.87 systemPercent: 0.61 userPercent: 1.52 cpus: 2 databaseBackend: boltdb distribution: distribution: rhel version: "8.10" eventLogger: file freeLocks: 2048 hostname: pipeline-agent idMappings: gidmap: - container_id: 0 host_id: 1003 size: 1 - container_id: 1 host_id: 231072 size: 65536 uidmap: - container_id: 0 host_id: 1002 size: 1 - container_id: 1 host_id: 231072 size: 65536 kernel: 4.18.0-553.8.1.el8_10.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 407351296 memTotal: 8071684096 networkBackend: cni networkBackendInfo: backend: cni dns: package: podman-plugins-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64 path: /usr/libexec/cni/dnsname version: |- CNI dnsname plugin version: 1.4.0-dev commit: unknown CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0 package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21962+8143777b.x86_64 path: /usr/libexec/cni ociRuntime: name: runc package: runc-1.1.12-1.module+el8.10.0+21974+acd2159c.x86_64 path: /usr/bin/runc version: |- runc version 1.1.12 spec: 1.0.2-dev go: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198) libseccomp: 2.5.2 os: linux pasta: executable: "" package: "" version: "" remoteSocket: exists: false path: /run/user/1002/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.3-1.module+el8.10.0+21962+8143777b.x86_64 version: |- slirp4netns version 1.2.3 commit: c22fde291bb35b354e6ca44d13be181c76a0a432 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.2 swapFree: 0 swapTotal: 0 uptime: 301h 24m 56.00s (Approximately 12.54 days) variant: "" plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.access.redhat.com - registry.redhat.io - docker.io store: configFile: /home/podman01/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /home/podman01/.local/share/containers/storage graphRootAllocated: 33275510784 graphRootUsed: 28726689792 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "false" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 8 runRoot: /tmp/containers-user-1002/containers transientStore: false volumePath: /home/podman01/.local/share/containers/storage/volumes version: APIVersion: 4.9.4-rhel Built: 1718703254 BuiltTime: Tue Jun 18 09:34:14 2024 GitCommit: "" GoVersion: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198) Os: linux OsArch: linux/amd64 Version: 4.9.4-rhel
funny, now you see my discarded attempt at anonymizing the user, because copy+paste sucks. I'm sorry for the excess content caused by this. Shouldn't have tried anyway. I didn't find a cause of the system migrate requirement yet, but a workaround that at least keeps the automation running: User cron job that checks every minute if podman responds with the "system migrate" requirement, and if so, execution of podman system migrate.
participants (2)
-
Daniel Walsh -
Dirk Försterling