[Podman] How does podman "initialize" after a reboot?

Tuesday, 11 July 2023

Hello everyone,

**Disclaimer: This is a long e-mail.**

I am on NixOS (23.05), using the podman binary provided by the
distribution package. There are several issues that I am facing but
the issue that I want resolved is that _I want rootless Podman
containers started at boot_.

I won't get much into NixOS other than what is needed (i.e. no
advocacy for NixOS). NixOS, being a distribution with reproducible
builds, has a different method of storing binaries. Instead of
binaries living in `/usr/bin`, binaries actually live in
`/nix/store/<hash>-pkg-ver/bin`. Thereafter, the binaries are linked
into `/run/current-system/sw/bin`. My `PATH` (from a login shell)
looks like the following:

```
[pratham@sentinel] $ echo $PATH
/home/pratham/.local/bin:/home/pratham/bin:/run/wrappers/bin:/home/pratham/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin:/home/pratham/.nix-profile/bin:/etc/profiles/per-user/pratham/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
```

NixOS, being an OS that you can build with configuration files (i.e.
almost zero bash code to install; except for formatting and mounting),
there exists a way to declare your Podman containers like you do in a
compose.yaml and those containers will automatically be created as a
systemd service [0]. This is great! But those service files are placed
in `/etc/systemd/user`. This has an issue: the Podman container now
runs as root. I checked this by **logging in as root** and checking
the output of `podman ps` (not just `sudo podman ps`). If I wanted
rootful containers, I wouldn't be using Podman...

So, for the time being, I have resorted to writing a systemd unit file
by hand (which is stored in `$HOME/.config/systemd/user`). But the
path `/run/current-system/sw/bin` is missing from the unit's PATH. No
biggie, I can just add it using the following line under the
`[Service]` section:

```
Environment="PATH=/run/current-system/sw/bin:$PATH"
```
(This is a temporary hack and is strongly advised against, but I did
this as a troubleshooting measure, not as a solution.)
But the service fails with the following log entries in journalctl:

```
Jul 11 10:46:47 sentinel podman[36673]:
time="2023-07-11T10:46:47+05:30" level=error msg="running
`/run/current-system/sw/bin/newuidmap 36686 0 1000 1 1 10000 65536`:
newuidmap: write to uid_map failed: Operation not permitted\n"
Jul 11 10:46:47 sentinel podman[36673]: Error: cannot set up namespace
using "/run/current-system/sw/bin/newuidmap": should have setuid or
have filecaps setuid: exit status 1
Jul 11 10:46:47 sentinel systemd[1317]: testing-env.service: Main
process exited, code=exited, status=125/n/a
```

I never encountered this error on Fedora or RHEL. While experimenting,
I noticed one thing: **If I run _any_ Podman command (even `podman
ps`) from my _login shell_ and then restart the Podman container's
systemd service, the service runs cleanly.**

From the _Why can't I use sudo with rootless Podman_ article [1]:

...
 One of the core reasons Podman requires a temporary files directory
is for detecting if the system has rebooted. After a reboot, all containers are no longer
running, all container filesystems are unmounted, and all network interfaces need to be
recreated (among many other things). Podman needs to update its database to reflect this
and perform some per-boot setup to ensure it is ready to launch containers. This is called
"refreshing the state."

 This is necessary because Podman is not a daemon. Each Podman command is run as a new
process and doesn't initially know what state containers are in. You can look in the
database for an accurate picture of all your current containers and their states.
Refreshing the state after a reboot is essential to making sure this picture continues to
be accurate.

 To perform the refresh, you need a reliable way of detecting a system reboot, and early
in development, the Podman team settled on using a sentinel file on a tmpfs filesystem. A
tmpfs is an in-memory filesystem that is not saved after a reboot—every time the system
starts, a tmpfs mount will be empty. By checking for the existence of a file on such a
filesystem and creating it if it does not exist, Podman can know if it's the first
time it has run since the system rebooted.

 The problem becomes in determining where you should put your temporary files directory.
The obvious answer is /tmp, but this is not guaranteed to be a tmpfs filesystem (though it
often is). Instead, by default, Podman will use /run, which is guaranteed to be a tmpfs.
Unfortunately, /run is only writable by root, so rootless Podman must look elsewhere. Our
team settled on the /run/user/$UID directories, a per-user temporary files directory.

This means that Podman needs some sort of "initialization" when the
system has rebooted. Apparently, due to NixOS' nature, this
"initialization" doesn't occur when Podman is invoked from a systemd
service (something is missing but I can't figure out _what_). So I
rebooted and setup an `inotifywait` job (logged in as `root`--not with
the `sudo` prefix--with the command `inotifywait /run/user/1000/
--recursive --monitor`; `XDG_RUNTIME_DIR` for user `pratham` is
`/run/user/1000`) and ran `podman ps` when I was logged in as user
`pratham`. It generated the following output:

```
/run/user/1000/ ATTRIB,ISDIR libpod
/run/user/1000/libpod/ ATTRIB,ISDIR
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ CREATE pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ OPEN pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MODIFY pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MOVED_FROM pause.pid.NjPiqQ
/run/user/1000/libpod/tmp/ MOVED_TO pause.pid
/run/user/1000/ ATTRIB,ISDIR libpod
/run/user/1000/libpod/ ATTRIB,ISDIR
/run/user/1000/containers/ CREATE,ISDIR overlay
/run/user/1000/containers/ OPEN,ISDIR overlay
/run/user/1000/containers/ ACCESS,ISDIR overlay
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay
/run/user/1000/containers/overlay/ CREATE overlay-true
/run/user/1000/containers/overlay/ OPEN overlay-true
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE overlay-true
/run/user/1000/containers/overlay/ OPEN overlay-true
/run/user/1000/containers/overlay/ CLOSE_NOWRITE,CLOSE overlay-true
/run/user/1000/containers/overlay/ CREATE metacopy()-false
/run/user/1000/containers/overlay/ OPEN metacopy()-false
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE metacopy()-false
/run/user/1000/containers/overlay/ CREATE native-diff()-true
/run/user/1000/containers/overlay/ OPEN native-diff()-true
/run/user/1000/containers/overlay/ CLOSE_WRITE,CLOSE native-diff()-true
/run/user/1000/containers/ CREATE,ISDIR overlay-containers
/run/user/1000/containers/ OPEN,ISDIR overlay-containers
/run/user/1000/containers/ ACCESS,ISDIR overlay-containers
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-containers
/run/user/1000/containers/ CREATE,ISDIR overlay-locks
/run/user/1000/containers/ OPEN,ISDIR overlay-locks
/run/user/1000/containers/ ACCESS,ISDIR overlay-locks
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-locks
/run/user/1000/containers/ CREATE,ISDIR networks
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/libpod/tmp/ OPEN alive.lck
/run/user/1000/containers/ OPEN,ISDIR networks
/run/user/1000/containers/ ACCESS,ISDIR networks
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR networks
/run/user/1000/libpod/tmp/ CREATE alive
/run/user/1000/libpod/tmp/ OPEN alive
/run/user/1000/libpod/tmp/ CLOSE_NOWRITE,CLOSE alive
/run/user/1000/libpod/tmp/ CLOSE_WRITE,CLOSE alive.lck
/run/user/1000/systemd/units/ CREATE .#invocation:dbus.serviced739c18053185984
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:dbus.serviced739c18053185984
/run/user/1000/systemd/units/ MOVED_TO invocation:dbus.service
/run/user/1000/ CREATE,ISDIR dbus-1
/run/user/1000/ OPEN,ISDIR dbus-1
/run/user/1000/ ACCESS,ISDIR dbus-1
/run/user/1000/ CLOSE_NOWRITE,CLOSE,ISDIR dbus-1
/run/user/1000/dbus-1/ OPEN,ISDIR services
/run/user/1000/dbus-1/services/ OPEN,ISDIR
/run/user/1000/dbus-1/ ACCESS,ISDIR services
/run/user/1000/dbus-1/services/ ACCESS,ISDIR
/run/user/1000/dbus-1/ ACCESS,ISDIR services
/run/user/1000/dbus-1/services/ ACCESS,ISDIR
/run/user/1000/dbus-1/ CLOSE_NOWRITE,CLOSE,ISDIR services
/run/user/1000/dbus-1/services/ CLOSE_NOWRITE,CLOSE,ISDIR
/run/user/1000/systemd/ CREATE,ISDIR transient
/run/user/1000/systemd/ OPEN,ISDIR transient
/run/user/1000/systemd/ ACCESS,ISDIR transient
/run/user/1000/systemd/ CLOSE_NOWRITE,CLOSE,ISDIR transient
/run/user/1000/systemd/transient/ CREATE podman-2894.scope
/run/user/1000/systemd/transient/ OPEN podman-2894.scope
/run/user/1000/systemd/transient/ MODIFY podman-2894.scope
/run/user/1000/systemd/transient/ CLOSE_WRITE,CLOSE podman-2894.scope
/run/user/1000/systemd/units/ CREATE
.#invocation:podman-2894.scopeb6be723b1ec13b95
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:podman-2894.scopeb6be723b1ec13b95
/run/user/1000/systemd/units/ MOVED_TO invocation:podman-2894.scope
/run/user/1000/containers/ CREATE,ISDIR overlay-layers
/run/user/1000/containers/ OPEN,ISDIR overlay-layers
/run/user/1000/containers/ ACCESS,ISDIR overlay-layers
/run/user/1000/containers/ CLOSE_NOWRITE,CLOSE,ISDIR overlay-layers
/run/user/1000/containers/overlay-layers/ CREATE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/containers/overlay-layers/ OPEN mountpoints.lock
/run/user/1000/containers/overlay-layers/ CLOSE_WRITE,CLOSE mountpoints.lock
/run/user/1000/systemd/units/ DELETE invocation:podman-2894.scope
/run/user/1000/systemd/transient/ DELETE podman-2894.scope
/run/user/1000/libpod/tmp/ OPEN pause.pid
/run/user/1000/libpod/tmp/ ACCESS pause.pid
/run/user/1000/libpod/tmp/ CLOSE_NOWRITE,CLOSE pause.pid
/run/user/1000/systemd/transient/ CREATE podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ OPEN podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ MODIFY podman-pause-f50834a6.scope
/run/user/1000/systemd/transient/ CLOSE_WRITE,CLOSE podman-pause-f50834a6.scope
/run/user/1000/systemd/units/ CREATE
.#invocation:podman-pause-f50834a6.scope03db5d0ea8888975
/run/user/1000/systemd/units/ MOVED_FROM
.#invocation:podman-pause-f50834a6.scope03db5d0ea8888975
/run/user/1000/systemd/units/ MOVED_TO invocation:podman-pause-f50834a6.scope
```

Following is the output of `podman info` on my computer:

```
[pratham@sentinel] $ podman info
host:
  arch: arm64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /run/current-system/sw/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 81.03
    systemPercent: 3.02
    userPercent: 15.94
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: stoat
    distribution: nixos
    version: "23.05"
  eventLogger: journald
  hostname: sentinel
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 994
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 6.1.38
  linkmode: dynamic
  logDriver: journald
  memFree: 3040059392
  memTotal: 3944181760
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: Unknown
    path: /run/current-system/sw/bin/crun
    version: |-
      crun version 1.8.4
      commit: 1.8.4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities:
CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_
CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable:
/nix/store/n8lbxja2hd766pnz89qki90na2b3g815-slirp4netns-1.2.0/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 2957766656
  swapTotal: 2957766656
  uptime: 0h 5m 34.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/pratham/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/pratham/.local/share/containers/storage
  graphRootAllocated: 13539516416
  graphRootUsed: 7770832896
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/pratham/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 315532800
  BuiltTime: Tue Jan  1 05:30:00 1980
  GitCommit: ""
  GoVersion: go1.20.5
  Os: linux
  OsArch: linux/arm64
  Version: 4.5.0
```

So my current question is how do I do this initial setup manually? I
don't want to log into `pratham`'s login shell every time I have to
reboot my machine for the Podman containers to start.

[0]: https://nixos.wiki/wiki/Podman#Run_Podman_containers_as_systemd_services
[1]: https://www.redhat.com/sysadmin/sudo-rootless-podman

 - Pratham Patel

2024

2023

2022

2021

2020

2019

[Podman] How does podman "initialize" after a reboot?