RunRoot & mistaken IDs
Hi guys. I experience this: -> $ podman images WARN[0000] RunRoot is pointing to a path (/run/user/1007/containers) which is not writable. Most likely podman will fail. Error: creating events dirs: mkdir /run/user/1007: permission denied -> $ id uid=2001(podmania) gid=2001(podmania) groups=2001(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 I think it might have something to do with the fact that I changed UID for the user, but why would this be? How troubleshoot & fix it, ideally without system reboot? many thanks, L.
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it. Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice. Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though. --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If it ain't broke, your hammer isn't wide 'nough.
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
---
On 28/03/2023 21:00, Chris Evich wrote: thanks, that was that delay, yes, bit annoying If 'usermod' was in mass/often use.
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist: -> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument -> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 Where/what does it persist/insist on that old, non-existent UID - would anybody know? many thanks, L.
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so. -> $ echo $XDG_RUNTIME_DIR /run/user/1107400004
On 1/29/24 08:52, lejeczek via Podman wrote:
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so.
-> $ echo $XDG_RUNTIME_DIR /run/user/1107400004 _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Ok you probably need to do a `podman system reset` since you changed the ownership of the homedir and the UID of the process running Podman. Podman recorded the previous settings in its database.
On 29/01/2024 15:55, Daniel Walsh wrote:
On 1/29/24 08:52, lejeczek via Podman wrote:
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote:
I think it might have something to do with the fact that I changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so.
-> $ echo $XDG_RUNTIME_DIR /run/user/1107400004 _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Ok you probably need to do a `podman system reset` since you changed the ownership of the homedir and the UID of the process running Podman. Podman recorded the previous settings in its database. _______________________________________________
Doing anything as the user, seems not as viable option. -> $ podman system reset WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument forcibly: -> $ rm -fr /home.sysop/podmania/.local/share/containers/* helps, kind of, for very next issue is: -> $ podman system reset ERRO[0000] cannot find UID/GID for user podmania: cannot read subids - check rootless mode in man pages. WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user WARNING! This will remove: ... I presumed - incorrectly? - that (these days) subordinate UIDs should work when: -> $ authselect current Profile ID: sssd Enabled features: - with-sudo - with-subid or am I missing something? p.s./btw - is it just me or Centos is getting increasingly clunky, really?
On 1/29/24 10:21, lejeczek via Podman wrote:
On 29/01/2024 15:55, Daniel Walsh wrote:
On 1/29/24 08:52, lejeczek via Podman wrote:
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote:
On 3/28/23 09:06, lejeczek via Podman wrote: > I think it might have something to do with the fact that I > changed UID for the user
The files under /run/user/$UID are typically managed by systemd-logind. I've noticed sometimes there's a delay between logging out and the files being cleaned up. Try logging out for a minute or three and see if that fixes it.
Also, if you have lingering enabled for the user, it may take a restart of particular the user.slice.
Lastly, I'm not certain, but you (as root) may be able to `systemctl reload systemd-logind`. That's a total guess though.
Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so.
-> $ echo $XDG_RUNTIME_DIR /run/user/1107400004 _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Ok you probably need to do a `podman system reset` since you changed the ownership of the homedir and the UID of the process running Podman. Podman recorded the previous settings in its database. _______________________________________________
Doing anything as the user, seems not as viable option.
-> $ podman system reset WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
forcibly: -> $ rm -fr /home.sysop/podmania/.local/share/containers/* helps, kind of, for very next issue is:
-> $ podman system reset ERRO[0000] cannot find UID/GID for user podmania: cannot read subids - check rootless mode in man pages. WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user WARNING! This will remove: ...
I presumed - incorrectly? - that (these days) subordinate UIDs should work when: -> $ authselect current Profile ID: sssd Enabled features: - with-sudo - with-subid
or am I missing something?
p.s./btw - is it just me or Centos is getting increasingly clunky, really? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Don't know if the remote /etc/subuid and /etc/subgid is working correctly. Is there a test program to list the contents of subuid? getent subuid USER?
There is a tool in Fedora: getsubids $ man -k getsubids getsubids (1) - get the subordinate id ranges for a user $ grep $USER: /etc/subuid test:100000:65536 $ getsubids $USER 0: test 100000 65536 $ grep $USER: /etc/subgid test:200000:65536 $ getsubids -g $USER 0: test 200000 65536 $ rpm -qf /usr/bin/getsubids shadow-utils-subid-4.14.0-2.fc40.aarch64 $ On Mon, Jan 29, 2024 at 5:05 PM Daniel Walsh <dwalsh@redhat.com> wrote:
On 1/29/24 10:21, lejeczek via Podman wrote:
On 29/01/2024 15:55, Daniel Walsh wrote:
On 1/29/24 08:52, lejeczek via Podman wrote:
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote: > On 3/28/23 09:06, lejeczek via Podman wrote: >> I think it might have something to do with the fact that I >> changed UID for the user > > The files under /run/user/$UID are typically managed by > systemd-logind. I've noticed sometimes there's a delay between > logging out and the files being cleaned up. Try logging out for > a minute or three and see if that fixes it. > > Also, if you have lingering enabled for the user, it may take a > restart of particular the user.slice. > > Lastly, I'm not certain, but you (as root) may be able to > `systemctl reload systemd-logind`. That's a total guess though. > > Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so.
-> $ echo $XDG_RUNTIME_DIR /run/user/1107400004 _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Ok you probably need to do a `podman system reset` since you changed the ownership of the homedir and the UID of the process running Podman. Podman recorded the previous settings in its database. _______________________________________________
Doing anything as the user, seems not as viable option.
-> $ podman system reset WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
forcibly: -> $ rm -fr /home.sysop/podmania/.local/share/containers/* helps, kind of, for very next issue is:
-> $ podman system reset ERRO[0000] cannot find UID/GID for user podmania: cannot read subids - check rootless mode in man pages. WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user WARNING! This will remove: ...
I presumed - incorrectly? - that (these days) subordinate UIDs should work when: -> $ authselect current Profile ID: sssd Enabled features: - with-sudo - with-subid
or am I missing something?
p.s./btw - is it just me or Centos is getting increasingly clunky, really? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Don't know if the remote /etc/subuid and /etc/subgid is working correctly.
Is there a test program to list the contents of subuid?
getent subuid USER? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 29/01/2024 16:27, Daniel Walsh wrote:
On 1/29/24 10:21, lejeczek via Podman wrote:
On 29/01/2024 15:55, Daniel Walsh wrote:
On 1/29/24 08:52, lejeczek via Podman wrote:
On 29/01/2024 12:04, Daniel Walsh wrote:
On 1/29/24 02:35, lejeczek via Podman wrote:
On 28/03/2023 21:00, Chris Evich wrote: > On 3/28/23 09:06, lejeczek via Podman wrote: >> I think it might have something to do with the fact >> that I changed UID for the user > > The files under /run/user/$UID are typically managed > by systemd-logind. I've noticed sometimes there's a > delay between logging out and the files being > cleaned up. Try logging out for a minute or three > and see if that fixes it. > > Also, if you have lingering enabled for the user, it > may take a restart of particular the user.slice. > > Lastly, I'm not certain, but you (as root) may be > able to `systemctl reload systemd-logind`. That's a > total guess though. > > Those parts seem very clunky - at least in up-to-date Centos 9 stream - I have removed a user and re/created that user in IdM and.. even after full & healthy OS reboot, containers/podman insist:
-> $ podman container ls -a WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
-> $ id uid=1107400004(podmania) gid=1107400004(podmania) groups=1107400004(podmania) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Where/what does it persist/insist on that old, non-existent UID - would anybody know?
many thanks, L. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Do you have XDG_RUNTIME_DIR pointing at it?
Nope, I don't think so.
-> $ echo $XDG_RUNTIME_DIR /run/user/1107400004 _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Ok you probably need to do a `podman system reset` since you changed the ownership of the homedir and the UID of the process running Podman. Podman recorded the previous settings in its database. _______________________________________________
Doing anything as the user, seems not as viable option.
-> $ podman system reset WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argument
forcibly: -> $ rm -fr /home.sysop/podmania/.local/share/containers/* helps, kind of, for very next issue is:
-> $ podman system reset ERRO[0000] cannot find UID/GID for user podmania: cannot read subids - check rootless mode in man pages. WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user WARNING! This will remove: ...
I presumed - incorrectly? - that (these days) subordinate UIDs should work when: -> $ authselect current Profile ID: sssd Enabled features: - with-sudo - with-subid
or am I missing something?
p.s./btw - is it just me or Centos is getting increasingly clunky, really? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Don't know if the remote /etc/subuid and /etc/subgid is working correctly.
Is there a test program to list the contents of subuid?
getent subuid USER? _______________________________________________
I think it's all good - perhaps me getting clunky - sub ids work. If you hit above "issue" - read up on IdM/FreeIPA - seems that _subordintate_ IDs are not "fully" implemented there yet(?) There is a tool: -> $ /usr/libexec/ipa/ipa-subids --group=containers Found 2 user(s) without subordinate ids Processing user 'podmania' (1/2) Processing user 'appownia' (2/2) Updated 2 user(s) The ipa-subids command was successful
after longest break, I went back to that misbehaving system - in case I never replied, to share a "fix" -> $ podman system reset -f WARN[0000] RunRoot is pointing to a path (/run/user/2001/containers) which is not writable. Most likely podman will fail. Error: default OCI runtime "crun" not found: invalid argumen -> rm -f ~/.local/share/containers/storage/db.sql -> $ podman system reset -f -> echo $? -> 0
participants (4)
-
Chris Evich -
Daniel Walsh -
Erik Sjölund -
lejeczek