3:51 p.m.
Hello!
I'm using podman on Fedora CoreOS. The standard setup for a
podman-based service tends to look like this (according to the
documentation):
---
[Unit]
Description=looseleaf
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
TimeoutStartSec=60
User=_looseleaf
Group=_looseleaf
Restart=on-failure
RestartSec=10s
Environment="_JAVA_OPTIONS=-XX:+UseSerialGC -Xmx64m -Xms64m"
ExecStartPre=-/bin/podman kill looseleaf
ExecStartPre=-/bin/podman rm looseleaf
ExecStartPre=/bin/podman pull docker.io/io7m/looseleaf:0.0.4
ExecStart=/bin/podman run \
--name looseleaf \
--volume /var/storage/looseleaf/etc:/looseleaf/etc:Z,ro \
--volume /var/storage/looseleaf/var:/looseleaf/var:Z,rw \
--publish 20000:20000/tcp \
--memory=128m \
--memory-reservation=80m \
docker.io/io7m/looseleaf:{{looseleaf_version}} \
/looseleaf/bin/looseleaf server --file /looseleaf/etc/config.json
ExecStop=/bin/podman stop looseleaf
[Install]
WantedBy=multi-user.target
---
The important line is this one:
/bin/podman pull docker.io/io7m/looseleaf:0.0.4
Unfortunately, this line can fail. That in itself isn't a problem, the
service will be restarted and it'll run again. The real problem is that
it can fail in ways that will break all subsequent executions.
On new Fedora CoreOS deployments, there's often a lot of network
traffic happening on first boot as the rest of the system updates
itself, and it's not unusual for `podman pull` to fail and leave the
services permanently broken (unless someone goes in and fixes them).
This is what will typically happen:
Feb 02 20:31:05 control1.io7m.com podman[1934]: Trying to pull
docker.io/io7m/looseleaf:0.0.4...
Feb 02 20:31:48 control1.io7m.com podman[1934]: time="2023-02-02T20:31:48Z"
level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source
docker://io7m/looseleaf:0.0.4: pinging container registry registry-1.docker.io: Get
\"https://regist>
Feb 02 20:31:50 control1.io7m.com podman[1934]: Getting image source signatures
Feb 02 20:31:50 control1.io7m.com podman[1934]: Copying blob
sha256:9794579c486abc6811cea048073584c869db02a4d9b615eeaa1d29e9c75738b9
Feb 02 20:31:50 control1.io7m.com podman[1934]: Copying blob
sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9
Feb 02 20:31:50 control1.io7m.com podman[1934]: Copying blob
sha256:846e3b32ee5a149e3ccb99051cdb52e96e11488293cdf72ee88168c88dd335c7
Feb 02 20:31:50 control1.io7m.com podman[1934]: Copying blob
sha256:7f516ed68e97f9655d26ae3312c2aeede3dfda2dd3d19d2f9c9c118027543e87
Feb 02 20:31:50 control1.io7m.com podman[1934]: Copying blob
sha256:e88daf71a034bed777eda8657762faad07639a9e27c7afb719b9a117946d1b8a
Feb 02 20:32:03 control1.io7m.com systemd[1]: looseleaf.service: start-pre operation timed
out. Terminating.
It'll usually happen again on the next service restart. Then, this will
tend to happen:
Feb 02 20:34:13 control1.io7m.com podman[2745]: time="2023-02-02T20:34:13Z"
level=error msg="Image docker.io/io7m/looseleaf:0.0.4 exists in local storage but may
be corrupted (remove the image to resolve the issue): size for layer
\"13cfed814d5b083572142bc>
Feb 02 20:34:13 control1.io7m.com podman[2745]: Trying to pull
docker.io/io7m/looseleaf:0.0.4...
Feb 02 20:34:14 control1.io7m.com podman[2745]: Getting image source signatures
Feb 02 20:34:14 control1.io7m.com podman[2745]: Copying blob
sha256:9794579c486abc6811cea048073584c869db02a4d9b615eeaa1d29e9c75738b9
Feb 02 20:34:14 control1.io7m.com podman[2745]: Copying blob
sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9
Feb 02 20:34:14 control1.io7m.com podman[2745]: Copying blob
sha256:846e3b32ee5a149e3ccb99051cdb52e96e11488293cdf72ee88168c88dd335c7
Feb 02 20:34:14 control1.io7m.com podman[2745]: Copying blob
sha256:7f516ed68e97f9655d26ae3312c2aeede3dfda2dd3d19d2f9c9c118027543e87
Feb 02 20:34:14 control1.io7m.com podman[2745]: Copying blob
sha256:e88daf71a034bed777eda8657762faad07639a9e27c7afb719b9a117946d1b8a
Feb 02 20:34:18 control1.io7m.com podman[2745]: Copying config
sha256:cce9701f3b6e34e3fc26332da58edcba85bbf4f625bdb5f508805d2fa5e62e3e
Feb 02 20:34:18 control1.io7m.com podman[2745]: Writing manifest to image destination
Feb 02 20:34:18 control1.io7m.com podman[2745]: Storing signatures
Feb 02 20:34:18 control1.io7m.com podman[2745]: Error: checking platform of image
cce9701f3b6e34e3fc26332da58edcba85bbf4f625bdb5f508805d2fa5e62e3e: inspecting image: size
for layer "13cfed814d5b083572142bc068ae7f890f323258135f0cffe87b04cb62c3742e" is
unkno>
Feb 02 20:34:18 control1.io7m.com systemd[1]: looseleaf.service: Control process exited,
code=exited, status=125/n/a
At this point, there's really nothing that can be done aside from
having a human log in and running something like "podman system reset".
These systems are supposed to be as immutable as possible, and
deployments are supposed to be automated. As it stands currently, I
can't actually a deploy a machine and not have it immediately break and
require a manual intervention.
Is there some better way to handle this?
--
Mark Raynsford | https://www.io7m.com