Improving our CI-driven, rootless, systemd user service-based container deployment with Quadlet?
Hi all. Late last year, we managed to deploy one of our web applications as a rootless Podman container and a systemd user service on a Ubuntu 22.04 VPS, using Podman 4.2.0, also building a full GitLab CI pipeline around it. Now Podman 4.4.0 is out, featuring Quadlet. There's no mention of the release on podman.io, let alone any docs for the Podman-integrated version of Quadlet that I can find. (Any ETA on these? I only realized it was out thanks to this mailing list.) Below I'll describe our deployment workflow and show an annotated version of the systemd service we use. What I'd like to know is: is there a way of using Quadlet for this use case? Would it be an improvement over the current setup in terms of using systemd/Podman "optimally"? It would especially be nice if Quadlet could give us a working Type=notify unit file, to eliminate the need for PID files. I was previously unable to get it to work (I no longer remember why), and had to use Type=forking instead. - - - - - Container deployment is done with an Ansible playbook, run by GitLab CI. It connects to the VPS, updates the image from our private registry, builds an entirely new container out of it with a unique suffix (appname-<imagetag>_<githash>) and templates out a unit file with a matching name (the Jinja2 template is based on 'podman generate systemd' output). The old systemd service is stopped and disabled, and the new one started and enabled. We do this because we want deploys to be "atomic", minimizing downtime. Building a new container instead of updating the old one lets us quickly revert to the previous version if the new container is faulty somehow. (Old containers and their unit files are eventually removed with a cron job.) Here's the unit file, with some annotations about our changes: [Unit] Description=AppName Wants=network-online.target After=network-online.target RequiresMountsFor=%t/containers [Service] # No Environment=PODMAN_SYSTEMD_UNIT=%n clause, because we don't use podman-auto-update Restart=on-failure TimeoutStopSec=10 # Don't try to start unless the container exists that this service is named after. ExecCondition=/bin/bash -c "/usr/bin/podman container inspect %N -f '{{.State.Status}}' 2> /dev/null" # Symlink the PID file to a more convenient location (not strictly necessary, # looks nicer here in PIDFile and ExecStopPost). ExecStartPre=/bin/bash -c "ln -sf \ $XDG_RUNTIME_DIR/containers/vfs-containers/$(podman container inspect %N -f '{{.Id}}')/userdata/conmon.pid \ $XDG_RUNTIME_DIR/.%N_conmon.pid" # Type=notify would be nicer Type=forking PIDFile=%t/.%N_conmon.pid ExecStart=/usr/bin/podman start %N # This pattern of running 'podman stop' in both ExecStop and ExecStopPost # is from podman-generate-systemd, but I never understood the reasoning for it. ExecStop=/usr/bin/podman stop --ignore -t 10 %N ExecStopPost=/usr/bin/podman stop --ignore -t 10 %N # Clean up the PID file symlink ExecStopPost=/bin/rm -f $XDG_RUNTIME_DIR/.%N_conmon.pid [Install] # Saves us from having to deal with the full unit name with the image tag and the # git hash; the symlink to this name is replaced to point to the new unit file # during the Ansible deployment. Alias=appname.service WantedBy=default.target - - - - - This is pretty robust for our purposes, but my systemd and overall Podman knowledge is limited, so I don't know what I could be doing better. Quadlet has a rather different philosophy overall than what we're used to, but can it be leveraged in this workflow, for CI-driven replacements of rootless containers running as systemd user services?
Hey, Since nobody else replied, I'll take a stab at your primary questions. Though caveat: I am not a quadlet expert! I believe what you're asking (socket-activation) is possible to configure with quadlet: See 'SocketActivated` at: https://github.com/containers/quadlet/blob/main/docs/Fileformat.md However, after reading what you're doing with Ansible and seeing your template, I wonder if quadlet is a good fit. It's designed to make systemd-managed containers easier to declare. So I'm questioning if all your customization + quadlet will just end up being MORE complex than what you have now. In any case, it's probably worth trying. Maybe you can get the template simplified enough it could just be a copy-task (maybe wishful thinking). Let us know how it goes. --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If it ain't broke, your hammer isn't wide 'nough. On 2/14/23 10:53, jklaiho@iki.fi wrote:
Hi all. Late last year, we managed to deploy one of our web applications as a rootless Podman container and a systemd user service on a Ubuntu 22.04 VPS, using Podman 4.2.0, also building a full GitLab CI pipeline around it.
Now Podman 4.4.0 is out, featuring Quadlet. There's no mention of the release on podman.io, let alone any docs for the Podman-integrated version of Quadlet that I can find. (Any ETA on these? I only realized it was out thanks to this mailing list.)
Below I'll describe our deployment workflow and show an annotated version of the systemd service we use. What I'd like to know is: is there a way of using Quadlet for this use case? Would it be an improvement over the current setup in terms of using systemd/Podman "optimally"?
It would especially be nice if Quadlet could give us a working Type=notify unit file, to eliminate the need for PID files. I was previously unable to get it to work (I no longer remember why), and had to use Type=forking instead.
- - - - -
Container deployment is done with an Ansible playbook, run by GitLab CI. It connects to the VPS, updates the image from our private registry, builds an entirely new container out of it with a unique suffix (appname-<imagetag>_<githash>) and templates out a unit file with a matching name (the Jinja2 template is based on 'podman generate systemd' output). The old systemd service is stopped and disabled, and the new one started and enabled.
We do this because we want deploys to be "atomic", minimizing downtime. Building a new container instead of updating the old one lets us quickly revert to the previous version if the new container is faulty somehow. (Old containers and their unit files are eventually removed with a cron job.)
Here's the unit file, with some annotations about our changes:
[Unit] Description=AppName Wants=network-online.target After=network-online.target RequiresMountsFor=%t/containers
[Service] # No Environment=PODMAN_SYSTEMD_UNIT=%n clause, because we don't use podman-auto-update
Restart=on-failure TimeoutStopSec=10
# Don't try to start unless the container exists that this service is named after. ExecCondition=/bin/bash -c "/usr/bin/podman container inspect %N -f '{{.State.Status}}' 2> /dev/null"
# Symlink the PID file to a more convenient location (not strictly necessary, # looks nicer here in PIDFile and ExecStopPost). ExecStartPre=/bin/bash -c "ln -sf \ $XDG_RUNTIME_DIR/containers/vfs-containers/$(podman container inspect %N -f '{{.Id}}')/userdata/conmon.pid \ $XDG_RUNTIME_DIR/.%N_conmon.pid"
# Type=notify would be nicer Type=forking PIDFile=%t/.%N_conmon.pid
ExecStart=/usr/bin/podman start %N
# This pattern of running 'podman stop' in both ExecStop and ExecStopPost # is from podman-generate-systemd, but I never understood the reasoning for it. ExecStop=/usr/bin/podman stop --ignore -t 10 %N ExecStopPost=/usr/bin/podman stop --ignore -t 10 %N
# Clean up the PID file symlink ExecStopPost=/bin/rm -f $XDG_RUNTIME_DIR/.%N_conmon.pid
[Install] # Saves us from having to deal with the full unit name with the image tag and the # git hash; the symlink to this name is replaced to point to the new unit file # during the Ansible deployment. Alias=appname.service WantedBy=default.target
- - - - -
This is pretty robust for our purposes, but my systemd and overall Podman knowledge is limited, so I don't know what I could be doing better. Quadlet has a rather different philosophy overall than what we're used to, but can it be leveraged in this workflow, for CI-driven replacements of rootless containers running as systemd user services? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On 2/14/23 10:53, jklaiho@iki.fi wrote:
Hi all. Late last year, we managed to deploy one of our web applications as a rootless Podman container and a systemd user service on a Ubuntu 22.04 VPS, using Podman 4.2.0, also building a full GitLab CI pipeline around it.
Now Podman 4.4.0 is out, featuring Quadlet. There's no mention of the release on podman.io, let alone any docs for the Podman-integrated version of Quadlet that I can find. (Any ETA on these? I only realized it was out thanks to this mailing list.)
Below I'll describe our deployment workflow and show an annotated version of the systemd service we use. What I'd like to know is: is there a way of using Quadlet for this use case? Would it be an improvement over the current setup in terms of using systemd/Podman "optimally"?
It would especially be nice if Quadlet could give us a working Type=notify unit file, to eliminate the need for PID files. I was previously unable to get it to work (I no longer remember why), and had to use Type=forking instead.
- - - - -
Container deployment is done with an Ansible playbook, run by GitLab CI. It connects to the VPS, updates the image from our private registry, builds an entirely new container out of it with a unique suffix (appname-<imagetag>_<githash>) and templates out a unit file with a matching name (the Jinja2 template is based on 'podman generate systemd' output). The old systemd service is stopped and disabled, and the new one started and enabled.
We do this because we want deploys to be "atomic", minimizing downtime. Building a new container instead of updating the old one lets us quickly revert to the previous version if the new container is faulty somehow. (Old containers and their unit files are eventually removed with a cron job.)
Here's the unit file, with some annotations about our changes:
[Unit] Description=AppName Wants=network-online.target After=network-online.target RequiresMountsFor=%t/containers
[Service] # No Environment=PODMAN_SYSTEMD_UNIT=%n clause, because we don't use podman-auto-update
Restart=on-failure TimeoutStopSec=10
# Don't try to start unless the container exists that this service is named after. ExecCondition=/bin/bash -c "/usr/bin/podman container inspect %N -f '{{.State.Status}}' 2> /dev/null"
# Symlink the PID file to a more convenient location (not strictly necessary, # looks nicer here in PIDFile and ExecStopPost). ExecStartPre=/bin/bash -c "ln -sf \ $XDG_RUNTIME_DIR/containers/vfs-containers/$(podman container inspect %N -f '{{.Id}}')/userdata/conmon.pid \ $XDG_RUNTIME_DIR/.%N_conmon.pid"
# Type=notify would be nicer Type=forking PIDFile=%t/.%N_conmon.pid
ExecStart=/usr/bin/podman start %N
# This pattern of running 'podman stop' in both ExecStop and ExecStopPost # is from podman-generate-systemd, but I never understood the reasoning for it. ExecStop=/usr/bin/podman stop --ignore -t 10 %N ExecStopPost=/usr/bin/podman stop --ignore -t 10 %N
# Clean up the PID file symlink ExecStopPost=/bin/rm -f $XDG_RUNTIME_DIR/.%N_conmon.pid
[Install] # Saves us from having to deal with the full unit name with the image tag and the # git hash; the symlink to this name is replaced to point to the new unit file # during the Ansible deployment. Alias=appname.service WantedBy=default.target
- - - - -
This is pretty robust for our purposes, but my systemd and overall Podman knowledge is limited, so I don't know what I could be doing better. Quadlet has a rather different philosophy overall than what we're used to, but can it be leveraged in this workflow, for CI-driven replacements of rootless containers running as systemd user services? _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
I just finished the first blog on quadlet, which will hopefully come out next week. We then plan at least two more blogs touting its features. Valentin should be able to help you with your sdnotify question.
Thanks for reaching out! On Tue, Feb 14, 2023 at 4:54 PM <jklaiho@iki.fi> wrote:
Hi all. Late last year, we managed to deploy one of our web applications as a rootless Podman container and a systemd user service on a Ubuntu 22.04 VPS, using Podman 4.2.0, also building a full GitLab CI pipeline around it.
Now Podman 4.4.0 is out, featuring Quadlet. There's no mention of the release on podman.io, let alone any docs for the Podman-integrated version of Quadlet that I can find. (Any ETA on these? I only realized it was out thanks to this mailing list.)
Below I'll describe our deployment workflow and show an annotated version of the systemd service we use. What I'd like to know is: is there a way of using Quadlet for this use case? Would it be an improvement over the current setup in terms of using systemd/Podman "optimally"?
You can find the docs of Quadlet here: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html Note that we only support systemd units generated by `podman-generate-systemd` and Quadlet. Any other customized units are not supported. There are too many things that can go wrong.
It would especially be nice if Quadlet could give us a working Type=notify unit file, to eliminate the need for PID files. I was previously unable to get it to work (I no longer remember why), and had to use Type=forking instead.
Yes, Quadlet supports that. `podman-generate-systemd --new` supports that, too.
- - - - -
Container deployment is done with an Ansible playbook, run by GitLab CI. It connects to the VPS, updates the image from our private registry, builds an entirely new container out of it with a unique suffix (appname-<imagetag>_<githash>) and templates out a unit file with a matching name (the Jinja2 template is based on 'podman generate systemd' output). The old systemd service is stopped and disabled, and the new one started and enabled.
We do this because we want deploys to be "atomic", minimizing downtime. Building a new container instead of updating the old one lets us quickly revert to the previous version if the new container is faulty somehow. (Old containers and their unit files are eventually removed with a cron job.)
Here's the unit file, with some annotations about our changes:
[Unit] Description=AppName Wants=network-online.target After=network-online.target RequiresMountsFor=%t/containers
[Service] # No Environment=PODMAN_SYSTEMD_UNIT=%n clause, because we don't use podman-auto-update
Restart=on-failure TimeoutStopSec=10
# Don't try to start unless the container exists that this service is named after. ExecCondition=/bin/bash -c "/usr/bin/podman container inspect %N -f '{{.State.Status}}' 2> /dev/null"
# Symlink the PID file to a more convenient location (not strictly necessary, # looks nicer here in PIDFile and ExecStopPost). ExecStartPre=/bin/bash -c "ln -sf \ $XDG_RUNTIME_DIR/containers/vfs-containers/$(podman container inspect %N -f '{{.Id}}')/userdata/conmon.pid \ $XDG_RUNTIME_DIR/.%N_conmon.pid"
# Type=notify would be nicer Type=forking PIDFile=%t/.%N_conmon.pid
ExecStart=/usr/bin/podman start %N
# This pattern of running 'podman stop' in both ExecStop and ExecStopPost # is from podman-generate-systemd, but I never understood the reasoning for it. ExecStop=/usr/bin/podman stop --ignore -t 10 %N ExecStopPost=/usr/bin/podman stop --ignore -t 10 %N
# Clean up the PID file symlink ExecStopPost=/bin/rm -f $XDG_RUNTIME_DIR/.%N_conmon.pid
[Install] # Saves us from having to deal with the full unit name with the image tag and the # git hash; the symlink to this name is replaced to point to the new unit file # during the Ansible deployment. Alias=appname.service WantedBy=default.target
- - - - -
This is pretty robust for our purposes, but my systemd and overall Podman knowledge is limited, so I don't know what I could be doing better. Quadlet has a rather different philosophy overall than what we're used to, but can it be leveraged in this workflow, for CI-driven replacements of rootless containers running as systemd user services?
Yes, Quadlet should integrate well into this workflow. Kind regards, Valentin
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
participants (4)
-
Chris Evich -
Daniel Walsh -
jklaiho＠iki.fi -
Valentin Rothberg