2:30 a.m.
Hello everyone,
**Disclaimer: This is a long e-mail.**
I am on NixOS (23.05), using the podman binary provided by the
distribution package. There are several issues that I am facing but
the issue that I want resolved is that _I want rootless Podman
containers started at boot_.
I won't get much into NixOS other than what is needed (i.e. no
advocacy for NixOS). NixOS, being a distribution with reproducible
builds, has a different method of storing binaries. Instead of
binaries living in `/usr/bin`, binaries actually live in
`/nix/store/<hash>-pkg-ver/bin`. Thereafter, the binaries are linked
into `/run/current-system/sw/bin`. My `PATH` (from a login shell)
looks like the following:
```
[pratham@sentinel] $ echo $PATH
/home/pratham/.local/bin:/home/pratham/bin:/run/wrappers/bin:/home/pratham/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin:/home/pratham/.nix-profile/bin:/etc/profiles/per-user/pratham/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
```
NixOS, being an OS that you can build with configuration files (i.e.
almost zero bash code to install; except for formatting and mounting),
there exists a way to declare your Podman containers like you do in a
compose.yaml and those containers will automatically be created as a
systemd service [0]. This is great! But those service files are placed
in `/etc/systemd/user`. This has an issue: the Podman container now
runs as root. I checked this by **logging in as root** and checking
the output of `podman ps` (not just `sudo podman ps`). If I wanted
rootful containers, I wouldn't be using Podman...
So, for the time being, I have resorted to writing a systemd unit file
by hand (which is stored in `$HOME/.config/systemd/user`). But the
path `/run/current-system/sw/bin` is missing from the unit's PATH. No
biggie, I can just add it using the following line under the
`[Service]` section:
```
Environment="PATH=/run/current-system/sw/bin:$PATH"
```
(This is a temporary hack and is strongly advised against, but I did
this as a troubleshooting measure, not as a solution.)
But the service fails with the following log entries in journalctl:
```
Jul 11 10:46:47 sentinel podman[36673]:
time="2023-07-11T10:46:47+05:30" level=error msg="running
`/run/current-system/sw/bin/newuidmap 36686 0 1000 1 1 10000 65536`:
newuidmap: write to uid_map failed: Operation not permitted\n"
Jul 11 10:46:47 sentinel podman[36673]: Error: cannot set up namespace
using "/run/current-system/sw/bin/newuidmap": should have setuid or
have filecaps setuid: exit status 1
Jul 11 10:46:47 sentinel systemd[1317]: testing-env.service: Main
process exited, code=exited, status=125/n/a
```
I never encountered this error on Fedora or RHEL. While experimenting,
I noticed one thing: **If I run _any_ Podman command (even `podman
ps`) from my _login shell_ and then restart the Podman container's
systemd service, the service runs cleanly.**
From the _Why can't I use sudo with rootless Podman_ article [1]:
One of the core reasons Podman requires a temporary files directory
is for detecting if the system has rebooted. After a reboot, all containers are no longer
running, all container filesystems are unmounted, and all network interfaces need to be
recreated (among many other things). Podman needs to update its database to reflect this
and perform some per-boot setup to ensure it is ready to launch containers. This is called
"refreshing the state."
This is necessary because Podman is not a daemon. Each Podman command is run as a new
process and doesn't initially know what state containers are in. You can look in the
database for an accurate picture of all your current containers and their states.
Refreshing the state after a reboot is essential to making sure this picture continues to
be accurate.
To perform the refresh, you need a reliable way of detecting a system reboot, and early
in development, the Podman team settled on using a sentinel file on a tmpfs filesystem. A
tmpfs is an in-memory filesystem that is not saved after a reboot—every time the system
starts, a tmpfs mount will be empty. By checking for the existence of a file on such a
filesystem and creating it if it does not exist, Podman can know if it's the first
time it has run since the system rebooted.
The problem becomes in determining where you should put your temporary files directory.
The obvious answer is /tmp, but this is not guaranteed to be a tmpfs filesystem (though it
often is). Instead, by default, Podman will use /run, which is guaranteed to be a tmpfs.
Unfortunately, /run is only writable by root, so rootless Podman must look elsewhere. Our
team settled on the /run/user/$UID directories, a per-user temporary files directory.
This means that Podman needs some sort of "initialization" when the
system has rebooted. Apparently, due to NixOS' nature, this
"initialization" doesn't occur when Podman is invoked from a systemd
service (something is missing but I can't figure out _what_). So I
rebooted and setup an `inotifywait` job (logged in as `root`--not with
the `sudo` prefix--with the command `inotifywait /run/user/1000/
--recursive --monitor`; `XDG_RUNTIME_DIR` for user `pratham` is
`/run/user/1000`) and ran `podman ps` when I was logged in as user
`pratham`. It generated the following output:
```
/run/user/1000/ ATTRIB,ISDIR libpod
/run/user/1000/libpod/ ATTRIB,ISDIR
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ CREATE pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ OPEN pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MODIFY pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MOVED_FROM pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MOVED_TO pause.pid
/run/user/1000/ ATTRIB,ISDIR libpod
/run/user/1000/libpod/ ATTRIB,ISDIR
/run/user/1000/containers/ CREATE,ISDIR overlay
/run/user/1000/containers/ OPEN,ISDIR overlay
/run/user/1000/containers/ ACCESS,ISDIR overlay
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay
/run/user/1000/containers/overlay/ CREATE overlay-true
/run/user/1000/containers/overlay/ OPEN overlay-true
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE overlay-true
/run/user/1000/containers/overlay/ OPEN overlay-true
/run/user/1000/containers/overlay/ CLOSE_NOWRITE,CLOSE overlay-true
/run/user/1000/containers/overlay/ CREATE metacopy()-false
/run/user/1000/containers/overlay/ OPEN metacopy()-false
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE metacopy()-false
/run/user/1000/containers/overlay/ CREATE native-diff()-true
/run/user/1000/containers/overlay/ OPEN native-diff()-true
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE native-diff()-true
/run/user/1000/containers/ CREATE,ISDIR overlay-containers
/run/user/1000/containers/ OPEN,ISDIR overlay-containers
/run/user/1000/containers/ ACCESS,ISDIR overlay-containers
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-containers
/run/user/1000/containers/ CREATE,ISDIR overlay-locks
/run/user/1000/containers/ OPEN,ISDIR overlay-locks
/run/user/1000/containers/ ACCESS,ISDIR overlay-locks
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-locks
/run/user/1000/containers/ CREATE,ISDIR networks
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/containers/ OPEN,ISDIR networks
/run/user/1000/containers/ ACCESS,ISDIR networks
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR networks
/run/user/1000/libpod/tmp/ CREATE alive
/run/user/1000/libpod/tmp/ OPEN alive
/run/user/1000/libpod/tmp/ CLOSE_NOWRITE,CLOSE alive
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/systemd/units/ CREATE .#invocation:dbus.serviced739c18053185984
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:dbus.serviced739c18053185984
/run/user/1000/systemd/units/ MOVED_TO invocation:dbus.service
/run/user/1000/ CREATE,ISDIR dbus-1
/run/user/1000/ OPEN,ISDIR dbus-1
/run/user/1000/ ACCESS,ISDIR dbus-1
/run/user/1000/ CLOSE_NOWRITE,CLOSE,ISDIR dbus-1
/run/user/1000/dbus-1/ OPEN,ISDIR services
/run/user/1000/dbus-1/services/ OPEN,ISDIR
/run/user/1000/dbus-1/ ACCESS,ISDIR services
/run/user/1000/dbus-1/services/ ACCESS,ISDIR
/run/user/1000/dbus-1/ ACCESS,ISDIR services
/run/user/1000/dbus-1/services/ ACCESS,ISDIR
/run/user/1000/dbus-1/ CLOSE_NOWRITE,CLOSE,ISDIR services
/run/user/1000/dbus-1/services/ CLOSE_NOWRITE,CLOSE,ISDIR
/run/user/1000/systemd/ CREATE,ISDIR transient
/run/user/1000/systemd/ OPEN,ISDIR transient
/run/user/1000/systemd/ ACCESS,ISDIR transient
/run/user/1000/systemd/ CLOSE_NOWRITE,CLOSE,ISDIR transient
/run/user/1000/systemd/transient/ CREATE podman-2894.scope
/run/user/1000/systemd/transient/ OPEN podman-2894.scope
/run/user/1000/systemd/transient/ MODIFY podman-2894.scope
/run/user/1000/systemd/transient/ CLOSE_WRITE,CLOSE podman-2894.scope
/run/user/1000/systemd/units/ CREATE
.#invocation:podman-2894.scopeb6be723b1ec13b95
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:podman-2894.scopeb6be723b1ec13b95
/run/user/1000/systemd/units/ MOVED_TO invocation:podman-2894.scope
/run/user/1000/containers/ CREATE,ISDIR overlay-layers
/run/user/1000/containers/ OPEN,ISDIR overlay-layers
/run/user/1000/containers/ ACCESS,ISDIR overlay-layers
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-layers
/run/user/1000/containers/overlay-layers/ CREATE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/systemd/units/ DELETE invocation:podman-2894.scope
/run/user/1000/systemd/transient/ DELETE podman-2894.scope
/run/user/1000/libpod/tmp/ OPEN pause.pid
/run/user/1000/libpod/tmp/ ACCESS pause.pid
/run/user/1000/libpod/tmp/ CLOSE_NOWRITE,CLOSE pause.pid
/run/user/1000/systemd/transient/ CREATE podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ OPEN podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ MODIFY podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ CLOSE_WRITE,CLOSE podman-pause-f50834a6.scope
/run/user/1000/systemd/units/ CREATE
.#invocation:podman-pause-f50834a6.scope03db5d0ea8888975
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:podman-pause-f50834a6.scope03db5d0ea8888975
/run/user/1000/systemd/units/ MOVED_TO invocation:podman-pause-f50834a6.scope
```
Following is the output of `podman info` on my computer:
```
[pratham@sentinel] $ podman info
host:
arch: arm64
buildahVersion: 1.30.0
cgroupControllers:
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: Unknown
path: /run/current-system/sw/bin/conmon
version: 'conmon version 2.1.7, commit: '
cpuUtilization:
idlePercent: 81.03
systemPercent: 3.02
userPercent: 15.94
cpus: 4
databaseBackend: boltdb
distribution:
codename: stoat
distribution: nixos
version: "23.05"
eventLogger: journald
hostname: sentinel
idMappings:
gidmap:
- container_id: 0
host_id: 994
size: 1
- container_id: 1
host_id: 10000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 10000
size: 65536
kernel: 6.1.38
linkmode: dynamic
logDriver: journald
memFree: 3040059392
memTotal: 3944181760
networkBackend: netavark
ociRuntime:
name: crun
package: Unknown
path: /run/current-system/sw/bin/crun
version: |-
crun version 1.8.4
commit: 1.8.4
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities:
CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_
CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: ""
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable:
/nix/store/n8lbxja2hd766pnz89qki90na2b3g815-slirp4netns-1.2.0/bin/slirp4netns
package: Unknown
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 2957766656
swapTotal: 2957766656
uptime: 0h 5m 34.00s
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
- quay.io
store:
configFile: /home/pratham/.config/containers/storage.conf
containerStore:
number: 2
paused: 0
running: 0
stopped: 2
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/pratham/.local/share/containers/storage
graphRootAllocated: 13539516416
graphRootUsed: 7770832896
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 9
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /home/pratham/.local/share/containers/storage/volumes
version:
APIVersion: 4.5.0
Built: 315532800
BuiltTime: Tue Jan 1 05:30:00 1980
GitCommit: ""
GoVersion: go1.20.5
Os: linux
OsArch: linux/arm64
Version: 4.5.0
```
So my current question is how do I do this initial setup manually? I
don't want to log into `pratham`'s login shell every time I have to
reboot my machine for the Podman containers to start.
[0]: https://nixos.wiki/wiki/Podman#Run_Podman_containers_as_systemd_services
[1]: https://www.redhat.com/sysadmin/sudo-rootless-podman
- Pratham Patel
12:56 a.m.
**Any** discussions--including me being an idot and not understanding
things--are welcome. Please comment/reply so I can understand what is
happening! :)
- Pratham Patel
3:43 a.m.
I also noticed that podman-machine does not start after reboot, I have
to start it manually.
There is also a bug fixed in 4.6 version.
https://github.com/containers/podman/pull/18328
https://github.com/containers/podman/issues/16945
Check status of podman machine after reboot using this command
podman machine list
чт, 3 авг. 2023 г. в 08:57, Pratham Patel <thefirst1322(a)gmail.com>:
**Any** discussions--including me being an idot and not understanding
things--are welcome. Please comment/reply so I can understand what is
happening! :)
- Pratham Patel
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
--
С уважением,
А.И.
6:50 a.m.
On 8/3/23 04:43, Александр Илюшкин wrote:
I also noticed that podman-machine does not start after reboot, I
have
to start it manually.
There is also a bug fixed in 4.6 version.
https://github.com/containers/podman/pull/18328
https://github.com/containers/podman/issues/16945
Check status of podman machine after reboot using this command
podman machine list
чт, 3 авг. 2023 г. в 08:57, Pratham Patel <thefirst1322(a)gmail.com>:
> **Any** discussions--including me being an idot and not understanding
> things--are welcome. Please comment/reply so I can understand what is
> happening! :)
>
>
> - Pratham Patel
> _______________________________________________
> Podman mailing list -- podman(a)lists.podman.io
> To unsubscribe send an email to podman-leave(a)lists.podman.io
Podman runs as a service under systemd, so systemd starts podman. If
you question is how does Podman start containers that are autostarted at
boot, then podman has a a podman-restart.service which systemd
automatcially starts, and this service looks at the podman database and
starts any autostart services.
/usr/lib/systemd/system/podman-restart.service
/usr/lib/systemd/user/podman-restart.service
If you are asking about podman-machine, then someone else will need to
answer that question.
8:32 a.m.
If you would like podman-machine to start at boot, then using
podman-desktop would be the route to go.
On Thu, Aug 3, 2023 at 7:51 AM Daniel Walsh <dwalsh(a)redhat.com> wrote:
On 8/3/23 04:43, Александр Илюшкин wrote:
> I also noticed that podman-machine does not start after reboot, I have
> to start it manually.
> There is also a bug fixed in 4.6 version.
>
> https://github.com/containers/podman/pull/18328
> https://github.com/containers/podman/issues/16945
>
> Check status of podman machine after reboot using this command
>
> podman machine list
>
> чт, 3 авг. 2023 г. в 08:57, Pratham Patel <thefirst1322(a)gmail.com>:
>> **Any** discussions--including me being an idot and not understanding
>> things--are welcome. Please comment/reply so I can understand what is
>> happening! :)
>>
>>
>> - Pratham Patel
>> _______________________________________________
>> Podman mailing list -- podman(a)lists.podman.io
>> To unsubscribe send an email to podman-leave(a)lists.podman.io
>
>
Podman runs as a service under systemd, so systemd starts podman. If
you question is how does Podman start containers that are autostarted at
boot, then podman has a a podman-restart.service which systemd
automatcially starts, and this service looks at the podman database and
starts any autostart services.
/usr/lib/systemd/system/podman-restart.service
/usr/lib/systemd/user/podman-restart.service
If you are asking about podman-machine, then someone else will need to
answer that question.
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
11:22 a.m.
If you question is how does Podman start containers that are
autostarted at boot
Yes, that is exactly what I want to know more about.
then podman has a a podman-restart.service which systemd
automatcially starts, and this service looks at the podman database and
starts any autostart services
/usr/lib/systemd/system/podman-restart.service
/usr/lib/systemd/user/podman-restart.service
Thanks a lot! I will take a closer look when I am on my machine :)
9:22 p.m.
I made my Podman services depend on the `podman-restart.service` and it
worked. Thank you!
The issue has now been resolved :)
- Pratham Patel
475
days inactive
499
days old
6 comments
4 participants
participants (4)
-
Ashley Cui -
Daniel Walsh -
Pratham Patel -
Александр Илюшкин