Ansible `template` tasks and rootless podman volume content management
Hey podman community, While exploring Ansible management of rootless podman on a remote host, I ran into a stinky volume-contents idempotency issue. I have an idea[0] on how to solve this, but thought I'd reach out and see if/how others have dealt with this situation. --- Here's the setup: 1. I'm running an Ansible playbook against a host for which I ONLY have access to a non-root (user) account. 2. The playbook configures `quadlet` for `systemd` management of a configuration (podman) volume and a pod with several containers in it running services. 3. The contents of the podman volume are 10-30 configuration files, owned by several different UIDs/GIDs within the allocated user-namespace. For example, some files are owned by $UID:$GID, others may be 100123:100123, and others could be 100321:100321 (depending on the exact user-namespace allocation details). 4. Ansible uses the 'template' module to manage 10-30 configuration files and directories destined for the rootless podman volume. Ref: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template... 5. When configuration files "change", Ansible uses a handler to restart the pod. Ref: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.ht... --- The problem: The 'template' module knows nothing about user-namespaces. Because it's running as a regular user, it can't `chown` the files into the user-namespace range (permission denied). So the template module is CONSTANTLY (and needlessly) triggering the handler to restart the pod (due to file ownership differences). Also as you'd expect, when `template` sets the file's UID/GID wrong, the containerized services fail on restart. --- Idea[0]: (untested) For the `template` task, set `ansible_python_interpreter` to a wrapper script that execs `podman unshare /usr/bin/python3 "$@"`. -- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If it ain't broke, your hammer isn't wide 'nough.
Hi Chris, On 05.06.23 at 16:51 Chris Evich wrote:
2. The playbook configures `quadlet` for `systemd` management of a configuration (podman) volume and a pod with several containers in it running services.
Any reason for using quadlet via template instead of the containers.podman collection? Not sure if that solves the issue, but as quadlet is only a podman->systemd translator, you can get the same using ansible and containers.podman. And: My first idea would have been to use a handler to restart the systemd user services. Or is this what you mean by "restarting the pod"? Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 6/15/23 02:47, Johannes Kastl wrote:
Hi Chris,
On 05.06.23 at 16:51 Chris Evich wrote:
2. The playbook configures `quadlet` for `systemd` management of a configuration (podman) volume and a pod with several containers in it running services.
Any reason for using quadlet via template instead of the containers.podman collection? Not sure if that solves the issue, but as quadlet is only a podman->systemd translator, you can get the same using ansible and containers.podman.
And: My first idea would have been to use a handler to restart the systemd user services. Or is this what you mean by "restarting the pod"?
Re: `containers.podman` vs quadlet: Simply because if the container host restarts or a container crashes unexpectedly, quadlet+systemd can ensure pods come back up. I considered using cron + Ansible pull, but that seemed slightly more complex/fragile. Re: Use handler: Yeah, the handler uses systemd to restart services (generated by quadlet). The main problem is handlers run on change. But because user-namespaces are in play, ansible's template module (rendering config files into podman volumes) always thinks there's a "change" (i.e. in file or directory ownership) even if there isn't. I still haven't tried (since it seems really really hacky) setting `ansible_python_interpreter` to a wrapper script that execs `podman unshare /usr/bin/python3 "$@"`. In theory this would work for Ansible `template` tasks, as the config files would be rendered INSIDE the user-namespace rather than outside. The only other idea I could come up with was to have a kind of secondary, conditional handler. Whereby an initial set of listening handlers updates the templated files (in the podman volume) with the correct UID/GID. Another one gathers a hash of the volume contents, then a final conditional handler only does the pod restart if the hashed contents have changed WRT a hash gathered prior to the original template task. This seems way to complex :S --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If it ain't broke, your hammer isn't wide 'nough.
Hi Chris, On 15.06.23 at 15:54 Chris Evich wrote:
On 6/15/23 02:47, Johannes Kastl wrote:
On 05.06.23 at 16:51 Chris Evich wrote:
2. The playbook configures `quadlet` for `systemd` management of a configuration (podman) volume and a pod with several containers in it running services.
Any reason for using quadlet via template instead of the containers.podman collection? Not sure if that solves the issue, but as quadlet is only a podman->systemd translator, you can get the same using ansible and containers.podman.
And: My first idea would have been to use a handler to restart the systemd user services. Or is this what you mean by "restarting the pod"?
Re: `containers.podman` vs quadlet: Simply because if the container host restarts or a container crashes unexpectedly, quadlet+systemd can ensure pods come back up. I considered using cron + Ansible pull, but that seemed slightly more complex/fragile.
You got me wrong. You can create systemd services for your pods etc. directly via Ansible, without having to write the unit files yourself, even without having quadlet (which is a nice piece of software, do not get me wrong) as another layer in between.
Re: Use handler: Yeah, the handler uses systemd to restart services (generated by quadlet). The main problem is handlers run on change. But because user-namespaces are in play, ansible's template module (rendering config files into podman volumes) always thinks there's a "change" (i.e. in file or directory ownership) even if there isn't.
I never needed to fumble around with user namespaces until now, so my naive question would be: Do you need to chown the files to the respective user namespaces in the first place? I thought all of that was handled by podman in the background? Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 6/16/23 02:00, Johannes Kastl wrote:
You got me wrong. You can create systemd services for your pods etc. directly via Ansible, without having to write the unit files yourself,
Oh okay, yeah that would be simpler. I've never done that before. I'll take your word for it and give it a spin. Thanks!
I never needed to fumble around with user namespaces until now, so my naive question would be: Do you need to chown the files to the respective user namespaces in the first place? I thought all of that was handled by podman in the background?
It can be handled by podman (at container run time), and this is the simplest thing to do. This is what the `:U` volume mount option is for. Also the `-o o=uid=<ID>,gid=<ID>` option to `podman volume create` However, podman will manage the ownership by changing the UIDs/GIDs of the volume content. This is the crux of the issue. On the next playbook run, the template module will register the managed content as "changed" - because the UIDs/GIDs don't match. Worse, if the template module tries to move updated files into place, it may receive a "permission-denied" - because the user may no-longer "own" the file's directory (podman chown'ed it too!). If you try to overcome this by (calculating the ID offsets and) specifying UID/GID values to the template module, that can also fail because the playbook is running as a user. Users aren't permitted to (directly) chown things to user-namespaced IDs (see `podman unshare` below). ---quick user-namespace primer--- On the host, `/etc/sub{uid,gid}` contain per user ID range allocations, which don't (typically) overlap each-other, and aren't (typically) used by anything else on the host (though nothing actually verifies/blocks this). It's like each user gets their own little (alternate) ID fiefdom on the system. The net effect is, a file or directory in a volume used by a rootless container will be owned (from the host POV) by either the user's UID/GID (corresponding to `root:root` in the container). Or, the UID/GID in the container + an offset calculated from the second field of `/etc/sub{uid,gid}`, bounded by summation of the third field (effectively, the total number of IDs allocated to the range). The wrapper `podman unshare <command>` lets a user "enter" their allocated user-namespace on the host (for <command>), and use or chown things to their heart's content (within their ID allocation). The actual ownership (outside <command>) will be the values as described above.
On 6/15/23 09:54, Chris Evich wrote:
I still haven't tried (since it seems really really hacky) setting `ansible_python_interpreter` to a wrapper script that execs `podman unshare /usr/bin/python3 "$@"`. In theory this would work for Ansible `template` tasks, as the config files would be rendered INSIDE the user-namespace rather than outside.
I realize this is a really old thread, but is a tricky thing I finally took time to experiment with. So I thought I'd share what I discovered. As a reminder of the problem: How can you template files into a rootless-podman volume on a host where you don't have root, and w/o flapping[0] or needing to fuss with the user-namespaced ownership? Assuming `/tmp/template.j2` exists with some test content (left up to your imagination), here's a playbook that demonstrates a possible solution: ---cut playbook--- - hosts: localhost connection: local # no ssh required become: false # Don't sudo to root tasks: - name: User-namespaced tasks vars: # N/B: assumed to already exist on the inventory host ansible_python_interpreter: '/tmp/unshare_python3.sh' block: - name: A testdir exists file: path: /tmp/testdir state: directory owner: 12345 group: 54321 - name: Render testfile into testdir template: src: '/tmp/template.j2' dest: /tmp/testdir/testfile owner: 54321 group: 12345 ---cut playbook--- And the wrapper script that makes the magic happen. ---cut /tmp/unshare_python3.sh wrapper--- #!/bin/sh exec /usr/bin/podman unshare $(type -p python3) "$@" ---cut /tmp/unshare_python3.sh wrapper--- After running the playbook, with a `/etc/sub{uid,gid}` entry of: `cevich:100000:65536` I end up with: ``` $ ls -la /tmp/testdir total 4 drwxr-xr-x. 2 112344 154320 60 Aug 11 12:46 . drwxrwxrwt. 56 root root 1160 Aug 11 13:02 .. -rw-r--r--. 1 154320 112344 49 Aug 11 12:46 testfile ``` But a key thing to note is what happens when you run the playbook more than once. Ansible will NOT needlessly "fix" the user-namespaced owner/group due to it not matching the `ansible_user` `UID:GID`. Hmmm...maybe I should write a blog on this? [0]: Flapping - Given the same inputs & conditions, an operation that's executed multiple times but doesn't always yield the same result. --- Chris Evich (he/him), RHCA III Senior Quality Assurance Engineer If there's a "hard-way", I'm the first one to implement it.
participants (2)
-
Chris Evich -
Johannes Kastl