Rootless container startup failure at bootup, launches fine manually
by jklaiho＠iki.fi
We've got a Ubuntu 22.04 server running a rootless container as a systemd user service on Podman 4.4.2, using Quadlet. Podman and its dependencies are installed from binary releases on GitHub and/or from source.
Last night the server required a reboot due to some security updates, but the service/container failed to come back up after the boot. In the morning, I was able to manually start it up with `systemctl --user start cms_backend` with no changes to any configuration in the meantime.
Here's the full journalctl output for this unit from last night after the reboot (timestamps removed for brevity):
systemd[746]: cms_backend.service: unit configures an IP firewall, but not running as root.
systemd[746]: (This warning is only shown for the first unit using IP firewalling.)
systemd[746]: Starting CMS Backend...
cms_backend[787]: time="2023-05-25T03:01:58+03:00" level=error msg="Refreshing container 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405: retrieving temporary directory for container 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405: no such container"
podman[787]: 2023-05-25 03:01:59.000295722 +0300 EEST m=+0.107418901 system refresh
cms_backend[787]: time="2023-05-25T03:01:59+03:00" level=warning msg="Found incomplete layer \"b767c3f7da27350a451d15f1e972cdd874ea0e6f4c38fad1f644686f0831786f\", deleting it"
cms_backend[787]: time="2023-05-25T03:01:59+03:00" level=error msg="Free container lock: no such file or directory"
podman[787]: 2023-05-25 03:01:59.037410032 +0300 EEST m=+0.144533221 container remove 8f03c9c90e6f8aab02344284ba760fe8ddbf52becb7ba95c383fb80c3bd04405 (image=<redacted>:latest, name=cms_backend, com.<redacted>.cms.commit_sha=99d36804, com.<redacted>.cms.component=server, com.<redacted>.cms.pipeline_id=25445, PODMAN_SYSTEMD_UNIT=cms_backend.service)
cms_backend[787]: Error: remove /run/user/1001/cms_backend.cid: no such file or directory
podman[787]: 2023-05-25 03:01:59.00150107 +0300 EEST m=+0.108624239 image pull <redacted>:latest
systemd[746]: cms_backend.service: Main process exited, code=exited, status=125/n/a
systemd[746]: cms_backend.service: Failed with result 'exit-code'.
systemd[746]: Failed to start CMS Backend.
Here's the Quadlet generator we use:
[Unit]
Description=CMS Backend
Wants=network-online.target
After=network-online.target
[Container]
Image=<redacted>:latest
ContainerName=cms_backend
Exec=/bin/bash -c "pip install -q -e . \
&& python project/manage.py migrate -v 1 \
&& python project/manage.py tailwind install \
&& python project/manage.py tailwind build \
&& python project/manage.py collectstatic --noinput \
&& python -Wd project/manage.py check \
&& exec gunicorn --workers 8 --threads 4 --worker-class gthread --worker-tmp-dir /dev/shm --error-logfile - --bind 0.0.0.0:8000 --pythonpath project cms_site.wsgi:application
EnvironmentFile=/home/cms/.cms_backend.env
RemapUsers=manual
RemapUid=0:0:1
RemapUid=100:1:1
RemapGid=0:0:1
RemapGid=65534:1:1
Network=pasta:-t,auto,-T,auto
# TODO change to native format after Podman 4.5 upgrade
PodmanArgs=--log-driver=journald \
--mount type=bind,source=/var/log/cms,target=/logs \
--mount type=bind,source=/srv/cms/staticfiles,target=/staticfiles
[Install]
WantedBy=default.target
- - - - -
The container itself is a lightly customized official Python 3.11 image running a Django application, practically identical to others that we're running with no similar issues. The entry point of the image is ["/usr/bin/dumb-init", "--"]. Like I said, the service came up manually just fine several hours after the initial failure, so it doesn't seem likely that this is a problem in our application code.
I can't decipher the error messages, can someone?
3 months, 3 weeks
--dns=ipaddr - no effect of it
by lejeczek
Hi guys.
--dns=none renders what expected but with an actual server,
say: --dns=10.3.1.200
resolve.conf seems to be the one of the host's, as if --dns
did not happen.
Can anybody else say that is the case? Am I missing something?
I'm on Centos 9 stream with all bits up-to-date.
many thanks, L.
3 months, 3 weeks
Podman Desktop unable to setup Docker Compose
by Mehdi Haghgoo
Hi,
On mac, I can simply press the Compose hint on the status bar for Podman Desktop and it sets up docker-compose.
On Linux, however, when I press the hint, I get the following error:
"Please add the compose wrapper bin folder to your PATH environment variable. Value is /home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin. The script /home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin/compose will setup for you the DOCKER_HOST environment variable."
I tried adding the mentioned value to the PATH in ~/.bash_profile and logging out and logging in, but it does not work. I always get the same error.
Looks like I can directly run "/home/mehdi/.local/share/containers/podman-desktop/extensions-storage/compose/bin/compose up" to start a compose project, which is interesting.
Or, I might be able to go the manual way (as before) of setting DOCKER_HOST to point to the Podman socket, but why doesn't the Podman Desktop way not work?
Am I doing something wrong?
Mehdi/
3 months, 4 weeks
How does podman set rootfs ownership to root when using --userns keep-id ?
by Fabio
Hi all,
I'm trying to understand some of the internals of namespace-based Linux
containers and I'm kindly asking you for help.
When launching `podman run -it --rm -v ~/Downloads:/dwn
docker.io/library/ubuntu /bin/bash`, the inside user is root. That is
expected, and without any surprise the /proc/self/uid_map is:
0 1000 1
1 100000 65536
When launching `podman run -it --rm -v ~/Downloads:/dwn --userns keep-id
docker.io/library/ubuntu /bin/bash` instead, the /proc/self/uid_map is:
0 1 1000
1000 0 1
1001 1001 64536
If I'm understanding it well, in the latter case there is a double
mapping: to keep host UID and GID, podman fires two user namespaces,
where the inner namespace maps its IDs the outer namespace, which
finally maps to the host (that is, 1000 -> 0 -> 1000 again).
The mechanism I don't get is how podman manages to make the rootfs owned
by root inside the inner namespace, while assigning volumes to the
unprivileged inner user:
dr-xr-xr-x. 1 root root 18 May 4 06:33 .
dr-xr-xr-x. 1 root root 18 May 4 06:33 ..
lrwxrwxrwx. 1 root root 7 Mar 8 02:05 bin -> usr/bin
drwxr-xr-x. 1 root root 0 Apr 18 2022 boot
[...]
drwxr-xr-x. 1 myuser 1000 2.1K May 3 15:07 dwn
What is the algorithm here? I have a feeling there is some clever
combination of syscalls here I don't get. When I tried to reproduce this
double namespace situation, the rootfs of the inner namespace was all
owned by 1000, not 0.
Thank you so so much for your time if you're willing to help me,
Fabio.
4 months
Podman Community Cabal Meeting, Thur, May 18, 2023, at 11:00 am EDT (UTC-5)
by Tom Sweeney
Hi All,
The next Podman Community Cabal Meeting is happening a week from now, on
Thur, May 18, 2023, at 11:00 am EDT (UTC-5). We'll be talking
containers, splitting storage layers, and more! Agenda with video
conference link: https://hackmd.io/gQCfskDuRLm7iOsWgH2yrg
Hope to see you there!
t
4 months, 2 weeks
Networking between KVM and containers
by sven＠svenschwermer.de
Hi,
I have a host running Debian Bullseye (Podman v3.0.1). On that host, I run an OPNsense VM via KVM. The goal is to create a (virtual) network connection between that VM and one or more Podman containers.
So far, I have created a dedicated bridge network for the VM via this network definition:
<network connections='1'>
<name>services</name>
<uuid>884d7543-91b0-4752-93b7-7efc6633d733</uuid>
<bridge name='virbr1' stp='on' delay='0'/>
<mac address='52:54:00:78:f8:79'/>
<ip address='192.168.50.1' netmask='255.255.255.0'>
</ip>
</network>
I then created this network for Podman:
[
{
"cniVersion": "0.4.0",
"name": "services",
"plugins": [
{
"ipam": {
"gateway": "192.168.50.2",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"subnet": "192.168.50.0/24",
"type": "host-local"
},
"master": "virbr1",
"type": "macvlan"
}
]
}
]
The container is is started like so:
podman run --network=services --ip=192.168.50.10 [...]
This does work, however, it doesn't seem ideal. Is there a better way to achieve networking between VM and containers? Is there a way to make Podman actually configure networking by making DHCP requests (to the OPNsense VM)? That way, DNS would be configured properly as well.
Any pointers are welcome 😄
Thanks, Sven
4 months, 2 weeks
'system reset' makes things weird - ?
by lejeczek
Hi guys.
I'm seeing something strange and I hoped experts/devel would
comment, on:
-> $ podman network ls
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
-> $ podman system reset --force
-> $ podman network ls
WARN[0000] Error validating CNI config file
/etc/cni/net.d/87-podman-bridge.conflist: [failed to find
plugin "bridge" in path [/usr/local/libexec/cni
/usr/libexec/cni /usr/local/lib/cni /usr/lib/cni
/opt/cni/bin] failed to find plugin "portmap" in path
[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni
/usr/lib/cni /opt/cni/bin] failed to find plugin "firewall"
in path [/usr/local/libexec/cni /usr/libexec/cni
/usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find
plugin "tuning" in path [/usr/local/libexec/cni
/usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
then I ssh-log out & in, things seem okey:
-> $ podman network ls
NETWORK ID NAME DRIVER
2f259bab93aa podman bridge
and if I 'reset' then in circles it goes.
what is actually happening here?
Is system setup/installation missing something?
many thanks, L.
4 months, 2 weeks
