What causes 'invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process'?
8:13 a.m.
Hello all,
I see the message 'invalid internal status, try resetting the pause process with
"podman system migrate": could not find any running process: no such
process' quite often upon executing podman (rootless).
In the net I could only find that this can happen after a host reboot. But in my case
there is no reboot, so I am curious wich conditions actually have to be met for this
message to appear. Yes, after "podman system migrate", I can use podman again,
but it might occur just a minute later, or a day. The only observation I had besides the
error message is, that after that "podman system migrate" I still see a couple
of containers in state "Stopping". With those, podman rm -f always complains
that it cannot kill the proces, but at least it can remove the containers.
I'm curious if I could prevent that "invalid internal status" completely
(because it disturbs my automation), but then I would need somebody explaining to me why
this does actually happen. Any hints?
-dirk
8:57 a.m.
New subject: What causes 'invalid internal status, try resetting the pause process with "podman system migrate": could not find any running process: no such process'?
On 7/12/24 09:13, Dirk Försterling via Podman wrote:
Hello all,
I see the message 'invalid internal status, try resetting the pause process with
"podman system migrate": could not find any running process: no such
process' quite often upon executing podman (rootless).
In the net I could only find that this can happen after a host reboot. But in my case
there is no reboot, so I am curious wich conditions actually have to be met for this
message to appear. Yes, after "podman system migrate", I can use podman again,
but it might occur just a minute later, or a day. The only observation I had besides the
error message is, that after that "podman system migrate" I still see a couple
of containers in state "Stopping". With those, podman rm -f always complains
that it cannot kill the proces, but at least it can remove the containers.
I'm curious if I could prevent that "invalid internal status" completely
(because it disturbs my automation), but then I would need somebody explaining to me why
this does actually happen. Any hints?
-dirk
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
podman system migrate recreates the default rootless user's user
namespace. This looks like something is collapsing for some reason.
Changing of /etc/subuid and /etc/subgid can require a system migrate.
What version of Podman are you using? Output of podman info would be
helpful. Usually questions like this are best handled within a github
issue.
9:52 a.m.
Thanks for your reply.
I wasn't aware that this might be a podman issue. I just wanted to know the possible
circumstances/reasons that can cause that behavior, so I can see how I can stop it.
The podman version is the RHEL8 packaged version (see podman info output below). The
podman system migrate requirement pops up randomly with no reboot and no modification of
/etc/subuid or /etc subgid. I just want to understand what else podman has to detect in
order to issue this message and refuse further action.
-dirk
podman info
host:
arch: amd64
buildahVersion: 1.33.8
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.10-1.module+el8.10.0+21962+8143777b.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit:
8f1552a71b76511fb9b4a96282513d17686c56fc'
cpuUtilization:
idlePercent: 97.87
systemPercent: 0.61
userPercent: 1.52
cpus: 2
databaseBackend: boltdb
distribution:
distribution: rhel
version: "8.10"
eventLogger: file
freeLocks: 2048
hostname: pipeline-agent
idMappings:
gidmap:
- container_id: 0
host_id: 1003
size: 1
- container_id: 1
host_id: 231072
size: 65536
uidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 231072
size: 65536
kernel: 4.18.0-553.8.1.el8_10.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 406753280
memTotal: 8071684096
networkBackend: cni
networkBackendInfo:
backend: cni
dns:
package: podman-plugins-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64
path: /usr/libexec/cni/dnsname
version: |-
CNI dnsname plugin
version: 1.4.0-dev
commit: unknown
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21962+8143777b.x86_64
path: /usr/libexec/cni
ociRuntime:
name: runc
package: runc-1.1.12-1.module+el8.10.0+21974+acd2159c.x86_64
path: /usr/bin/runc
version: |-
runc version 1.1.12
spec: 1.0.2-dev
go: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
libseccomp: 2.5.2
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: false
path: /run/user/1002/podman/podman.sock
security:
apparmorEnabled: false
capabilities:
CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.3-1.module+el8.10.0+21962+8143777b.x86_64
version: |-
slirp4netns version 1.2.3
commit: c22fde291bb35b354e6ca44d13be181c76a0a432
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 301h 24m 36.00s (Approximately 12.54 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /home/t01a01/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/t01a01/.local/share/containers/storage
graphRootAllocated: 33275510784
graphRootUsed: 28726689792
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 8
runRoot: /tmp/containers-user-1002/containers
transientStore: false
volumePath: /home/t01a01/.local/share/containers/storage/volumes
version:
APIVersion: 4.9.4-rhel
Built: 1718703254
BuiltTime: Tue Jun 18 09:34:14 2024
GitCommit: ""
GoVersion: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
Os: linux
OsArch: linux/amd64
Version: 4.9.4-rhel
[t01a01@pipeline-agent:dev ~]$ podman info | sed s/t01a01/podman01/g
host:
arch: amd64
buildahVersion: 1.33.8
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.10-1.module+el8.10.0+21962+8143777b.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit:
8f1552a71b76511fb9b4a96282513d17686c56fc'
cpuUtilization:
idlePercent: 97.87
systemPercent: 0.61
userPercent: 1.52
cpus: 2
databaseBackend: boltdb
distribution:
distribution: rhel
version: "8.10"
eventLogger: file
freeLocks: 2048
hostname: pipeline-agent
idMappings:
gidmap:
- container_id: 0
host_id: 1003
size: 1
- container_id: 1
host_id: 231072
size: 65536
uidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 231072
size: 65536
kernel: 4.18.0-553.8.1.el8_10.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 407351296
memTotal: 8071684096
networkBackend: cni
networkBackendInfo:
backend: cni
dns:
package: podman-plugins-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64
path: /usr/libexec/cni/dnsname
version: |-
CNI dnsname plugin
version: 1.4.0-dev
commit: unknown
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21962+8143777b.x86_64
path: /usr/libexec/cni
ociRuntime:
name: runc
package: runc-1.1.12-1.module+el8.10.0+21974+acd2159c.x86_64
path: /usr/bin/runc
version: |-
runc version 1.1.12
spec: 1.0.2-dev
go: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
libseccomp: 2.5.2
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: false
path: /run/user/1002/podman/podman.sock
security:
apparmorEnabled: false
capabilities:
CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.3-1.module+el8.10.0+21962+8143777b.x86_64
version: |-
slirp4netns version 1.2.3
commit: c22fde291bb35b354e6ca44d13be181c76a0a432
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 301h 24m 56.00s (Approximately 12.54 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /home/podman01/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/podman01/.local/share/containers/storage
graphRootAllocated: 33275510784
graphRootUsed: 28726689792
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 8
runRoot: /tmp/containers-user-1002/containers
transientStore: false
volumePath: /home/podman01/.local/share/containers/storage/volumes
version:
APIVersion: 4.9.4-rhel
Built: 1718703254
BuiltTime: Tue Jun 18 09:34:14 2024
GitCommit: ""
GoVersion: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
Os: linux
OsArch: linux/amd64
Version: 4.9.4-rhel
7:24 a.m.
funny, now you see my discarded attempt at anonymizing the user, because copy+paste sucks.
I'm sorry for the excess content caused by this. Shouldn't have tried anyway.
I didn't find a cause of the system migrate requirement yet, but a workaround that at
least keeps the automation running: User cron job that checks every minute if podman
responds with the "system migrate" requirement, and if so, execution of podman
system migrate.
120
days inactive
125
days old
3 comments
2 participants
participants (2)
-
Daniel Walsh -
Dirk Försterling