RunRoot & mistaken IDs
by lejeczek
Hi guys.
I experience this:
-> $ podman images
WARN[0000] RunRoot is pointing to a path
(/run/user/1007/containers) which is not writable. Most
likely podman will fail.
Error: creating events dirs: mkdir /run/user/1007:
permission denied
-> $ id
uid=2001(podmania) gid=2001(podmania) groups=2001(podmania)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
I think it might have something to do with the fact that I
changed UID for the user, but why would this be?
How troubleshoot & fix it, ideally without system reboot?
many thanks, L.
2 months, 2 weeks
mqueue msg_max in rootless container
by Michael Ivanov
Hallo!
I'm trying to run my application in podman rootless container and I stumble
on following problem: my program needs /proc/sys/fs/mqueue/msg_max to be at
least 256, but in running container this value is just 10. When I try to
specify this parameter while running the image (--sysctl 'fs.mqueue.msg_max=256')
I get the following error:
Error: open /proc/sys/fs/mqueue/msg_max: Permission denied: OCI permission denied
and container is not created.
My host where container is being run has this parameter set to 256. How can I
expose current host setting for msg_max to my container?
Best regards,
--
\ / | |
(OvO) | Михаил Иванов |
(^^^) | |
\^/ | E-mail: ivans(a)isle.spb.ru |
^ ^ | |
4 months, 3 weeks
Rootless container startup failure at bootup, launches fine manually
by jklaiho＠iki.fi
We've got a Ubuntu 22.04 server running a rootless container as a systemd user service on Podman 4.4.2, using Quadlet. Podman and its dependencies are installed from binary releases on GitHub and/or from source.
Last night the server required a reboot due to some security updates, but the service/container failed to come back up after the boot. In the morning, I was able to manually start it up with `systemctl --user start cms_backend` with no changes to any configuration in the meantime.
Here's the full journalctl output for this unit from last night after the reboot (timestamps removed for brevity):
systemd[746]: cms_backend.service: unit configures an IP firewall, but not running as root.
systemd[746]: (This warning is only shown for the first unit using IP firewalling.)
systemd[746]: Starting CMS Backend...
cms_backend[787]: time="2023-05-25T03:01:58+03:00" level=error msg="Refreshing container 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405: retrieving temporary directory for container 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405: no such container"
podman[787]: 2023-05-25 03:01:59.000295722 +0300 EEST m=+0.107418901 system refresh
cms_backend[787]: time="2023-05-25T03:01:59+03:00" level=warning msg="Found incomplete layer \"b767c3f7da27350a451d15f1e972cdd874ea0e6f4c38fad1f644686f0831786f\", deleting it"
cms_backend[787]: time="2023-05-25T03:01:59+03:00" level=error msg="Free container lock: no such file or directory"
podman[787]: 2023-05-25 03:01:59.037410032 +0300 EEST m=+0.144533221 container remove 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405 (image=<redacted>:latest, name=cms_backend, com.<redacted>.cms.commit_sha=99d36804, com.<redacted>.cms.component=server, com.<redacted>.cms.pipeline_id=25445, PODMAN_SYSTEMD_UNIT=cms_backend.service)
cms_backend[787]: Error: remove /run/user/1001/cms_backend.cid: no such file or directory
podman[787]: 2023-05-25 03:01:59.00150107 +0300 EEST m=+0.108624239 image pull <redacted>:latest
systemd[746]: cms_backend.service: Main process exited, code=exited, status=125/n/a
systemd[746]: cms_backend.service: Failed with result 'exit-code'.
systemd[746]: Failed to start CMS Backend.
Here's the Quadlet generator we use:
[Unit]
Description=CMS Backend
Wants=network-online.target
After=network-online.target
[Container]
Image=<redacted>:latest
ContainerName=cms_backend
Exec=/bin/bash -c "pip install -q -e . \
&& python project/manage.py migrate -v 1 \
&& python project/manage.py tailwind install \
&& python project/manage.py tailwind build \
&& python project/manage.py collectstatic --noinput \
&& python -Wd project/manage.py check \
&& exec gunicorn --workers 8 --threads 4 --worker-class gthread --worker-tmp-dir /dev/shm --error-logfile - --bind 0.0.0.0:8000 --pythonpath project cms_site.wsgi:application
EnvironmentFile=/home/cms/.cms_backend.env
RemapUsers=manual
RemapUid=0:0:1
RemapUid=100:1:1
RemapGid=0:0:1
RemapGid=65534:1:1
Network=pasta:-t,auto,-T,auto
# TODO change to native format after Podman 4.5 upgrade
PodmanArgs=--log-driver=journald \
--mount type=bind,source=/var/log/cms,target=/logs \
--mount type=bind,source=/srv/cms/staticfiles,target=/staticfiles
[Install]
WantedBy=default.target
- - - - -
The container itself is a lightly customized official Python 3.11 image running a Django application, practically identical to others that we're running with no similar issues. The entry point of the image is ["/usr/bin/dumb-init", "--"]. Like I said, the service came up manually just fine several hours after the initial failure, so it doesn't seem likely that this is a problem in our application code.
I can't decipher the error messages, can someone?
10 months, 3 weeks
--dns=ipaddr - no effect of it
by lejeczek
Hi guys.
--dns=none renders what expected but with an actual server,
say: --dns=10.3.1.200
resolve.conf seems to be the one of the host's, as if --dns
did not happen.
Can anybody else say that is the case? Am I missing something?
I'm on Centos 9 stream with all bits up-to-date.
many thanks, L.
10 months, 3 weeks
Podman Desktop unable to setup Docker Compose
by Mehdi Haghgoo
Hi,
On mac, I can simply press the Compose hint on the status bar for Podman Desktop and it sets up docker-compose.
On Linux, however, when I press the hint, I get the following error:
"Please add the compose wrapper bin folder to your PATH environment variable. Value is /home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin. The script /home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin/compose will setup for you the DOCKER_HOST environment variable."
I tried adding the mentioned value to the PATH in ~/.bash_profile and logging out and logging in, but it does not work. I always get the same error.
Looks like I can directly run "/home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin/compose up" to start a compose project, which is interesting.
Or, I might be able to go the manual way (as before) of setting DOCKER_HOST to point to the Podman socket, but why doesn't the Podman Desktop way not work?
Am I doing something wrong?
Mehdi/
10 months, 4 weeks
How does podman set rootfs ownership to root when using --userns keep-id ?
by Fabio
Hi all,
I'm trying to understand some of the internals of namespace-based Linux
containers and I'm kindly asking you for help.
When launching `podman run -it --rm -v ~/Downloads:/dwn
docker.io/library/ubuntu /bin/bash`, the inside user is root. That is
expected, and without any surprise the /proc/self/uid_map is:
0 1000 1
1 100000 65536
When launching `podman run -it --rm -v ~/Downloads:/dwn --userns keep-id
docker.io/library/ubuntu /bin/bash` instead, the /proc/self/uid_map is:
0 1 1000
1000 0 1
1001 1001 64536
If I'm understanding it well, in the latter case there is a double
mapping: to keep host UID and GID, podman fires two user namespaces,
where the inner namespace maps its IDs the outer namespace, which
finally maps to the host (that is, 1000 -> 0 -> 1000 again).
The mechanism I don't get is how podman manages to make the rootfs owned
by root inside the inner namespace, while assigning volumes to the
unprivileged inner user:
dr-xr-xr-x. 1 root root 18 May 4 06:33 .
dr-xr-xr-x. 1 root root 18 May 4 06:33 ..
lrwxrwxrwx. 1 root root 7 Mar 8 02:05 bin -> usr/bin
drwxr-xr-x. 1 root root 0 Apr 18 2022 boot
[...]
drwxr-xr-x. 1 myuser 1000 2.1K May 3 15:07 dwn
What is the algorithm here? I have a feeling there is some clever
combination of syscalls here I don't get. When I tried to reproduce this
double namespace situation, the rootfs of the inner namespace was all
owned by 1000, not 0.
Thank you so so much for your time if you're willing to help me,
Fabio.
11 months
Podman Community Cabal Meeting, Thur, May 18, 2023, at 11:00 am EDT (UTC-5)
by Tom Sweeney
Hi All,
The next Podman Community Cabal Meeting is happening a week from now, on
Thur, May 18, 2023, at 11:00 am EDT (UTC-5). We'll be talking
containers, splitting storage layers, and more! Agenda with video
conference link: https://hackmd.io/gQCfskDuRLm7iOsWgH2yrg
Hope to see you there!
t
11 months, 2 weeks
