9:51 a.m.
Hey podman community,
While exploring Ansible management of rootless podman on a remote host,
I ran into a stinky volume-contents idempotency issue. I have an
idea[0] on how to solve this, but thought I'd reach out and see if/how
others have dealt with this situation.
---
Here's the setup:
1. I'm running an Ansible playbook against a host for which I ONLY have
access to a non-root (user) account.
2. The playbook configures `quadlet` for `systemd` management of a
configuration (podman) volume and a pod with several containers in it
running services.
3. The contents of the podman volume are 10-30 configuration files,
owned by several different UIDs/GIDs within the allocated
user-namespace. For example, some files are owned by $UID:$GID, others
may be 100123:100123, and others could be 100321:100321 (depending on
the exact user-namespace allocation details).
4. Ansible uses the 'template' module to manage 10-30 configuration
files and directories destined for the rootless podman volume. Ref:
https://docs.ansible.com/ansible/latest/collections/ansible/builtin/templ...
5. When configuration files "change", Ansible uses a handler to restart
the pod. Ref:
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers...
---
The problem:
The 'template' module knows nothing about user-namespaces. Because it's
running as a regular user, it can't `chown` the files into the
user-namespace range (permission denied). So the template module is
CONSTANTLY (and needlessly) triggering the handler to restart the pod
(due to file ownership differences). Also as you'd expect, when
`template` sets the file's UID/GID wrong, the containerized services
fail on restart.
---
Idea[0]: (untested) For the `template` task, set
`ansible_python_interpreter` to a wrapper script that execs `podman
unshare /usr/bin/python3 "$@"`.
--
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If it ain't broke, your hammer isn't wide 'nough.
1:47 a.m.
New subject: Ansible `template` tasks and rootless podman volume content management
Hi Chris,
On 05.06.23 at 16:51 Chris Evich wrote:
2. The playbook configures `quadlet` for `systemd` management of a
configuration
(podman) volume and a pod with several containers in it running services.
Any reason for using quadlet via template instead of the containers.podman
collection? Not sure if that solves the issue, but as quadlet is only a
podman->systemd translator, you can get the same using ansible and
containers.podman.
And: My first idea would have been to use a handler to restart the systemd user
services. Or is this what you mean by "restarting the pod"?
Kind Regards,
Johannes
--
Johannes Kastl
Linux Consultant & Trainer
Tel.: +49 (0) 151 2372 5802
Mail: kastl(a)b1-systems.de
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg
http://www.b1-systems.de
GF: Ralph Dehner
Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
8:54 a.m.
New subject: Ansible `template` tasks and rootless podman volume content management
On 6/15/23 02:47, Johannes Kastl wrote:
Hi Chris,
On 05.06.23 at 16:51 Chris Evich wrote:
> 2. The playbook configures `quadlet` for `systemd` management of a
> configuration (podman) volume and a pod with several containers in it
> running services.
Any reason for using quadlet via template instead of the
containers.podman collection? Not sure if that solves the issue, but as
quadlet is only a podman->systemd translator, you can get the same using
ansible and containers.podman.
And: My first idea would have been to use a handler to restart the
systemd user services. Or is this what you mean by "restarting the pod"?
Re: `containers.podman` vs quadlet: Simply because if the container host
restarts or a container crashes unexpectedly, quadlet+systemd can ensure
pods come back up. I considered using cron + Ansible pull, but that
seemed slightly more complex/fragile.
Re: Use handler: Yeah, the handler uses systemd to restart services
(generated by quadlet). The main problem is handlers run on change.
But because user-namespaces are in play, ansible's template module
(rendering config files into podman volumes) always thinks there's a
"change" (i.e. in file or directory ownership) even if there isn't.
I still haven't tried (since it seems really really hacky) setting
`ansible_python_interpreter` to a wrapper script that execs `podman
unshare /usr/bin/python3 "$@"`. In theory this would work for Ansible
`template` tasks, as the config files would be rendered INSIDE the
user-namespace rather than outside.
The only other idea I could come up with was to have a kind of
secondary, conditional handler. Whereby an initial set of listening
handlers updates the templated files (in the podman volume) with the
correct UID/GID. Another one gathers a hash of the volume contents,
then a final conditional handler only does the pod restart if the hashed
contents have changed WRT a hash gathered prior to the original template
task. This seems way to complex :S
---
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If it ain't broke, your hammer isn't wide 'nough.
1 a.m.
New subject: Ansible `template` tasks and rootless podman volume content management
Hi Chris,
On 15.06.23 at 15:54 Chris Evich wrote:
On 6/15/23 02:47, Johannes Kastl wrote:
> On 05.06.23 at 16:51 Chris Evich wrote:
>
>> 2. The playbook configures `quadlet` for `systemd` management of a
>> configuration (podman) volume and a pod with several containers in it running
>> services.
>
> Any reason for using quadlet via template instead of the containers.podman
> collection? Not sure if that solves the issue, but as quadlet is only a
> podman->systemd translator, you can get the same using ansible and
> containers.podman.
>
> And: My first idea would have been to use a handler to restart the systemd
> user services. Or is this what you mean by "restarting the pod"?
Re: `containers.podman` vs quadlet: Simply because if the container host
restarts or a container crashes unexpectedly, quadlet+systemd can ensure pods
come back up. I considered using cron + Ansible pull, but that seemed slightly
more complex/fragile.
You got me wrong. You can create systemd services for your pods etc. directly
via Ansible, without having to write the unit files yourself, even without
having quadlet (which is a nice piece of software, do not get me wrong) as
another layer in between.
Re: Use handler: Yeah, the handler uses systemd to restart services
(generated
by quadlet). The main problem is handlers run on change. But because
user-namespaces are in play, ansible's template module (rendering config files
into podman volumes) always thinks there's a "change" (i.e. in file or
directory
ownership) even if there isn't.
I never needed to fumble around with user namespaces until now, so my naive
question would be: Do you need to chown the files to the respective user
namespaces in the first place? I thought all of that was handled by podman in
the background?
Kind Regards,
Johannes
--
Johannes Kastl
Linux Consultant & Trainer
Tel.: +49 (0) 151 2372 5802
Mail: kastl(a)b1-systems.de
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg
http://www.b1-systems.de
GF: Ralph Dehner
Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
2:33 p.m.
New subject: Ansible `template` tasks and rootless podman volume content management
On 6/16/23 02:00, Johannes Kastl wrote:
You got me wrong. You can create systemd services for your pods etc.
directly via Ansible, without having to write the unit files yourself,
Oh okay, yeah that would be simpler. I've never done that before. I'll
take your word for it and give it a spin. Thanks!
I never needed to fumble around with user namespaces until now, so my
naive question would be: Do you need to chown the files to the
respective user namespaces in the first place? I thought all of that was
handled by podman in the background?
It can be handled by podman (at container run time), and this is the
simplest thing to do. This is what the `:U` volume mount option is for.
Also the `-o o=uid=<ID>,gid=<ID>` option to `podman volume create`
However, podman will manage the ownership by changing the UIDs/GIDs of
the volume content. This is the crux of the issue. On the next
playbook run, the template module will register the managed content as
"changed" - because the UIDs/GIDs don't match. Worse, if the template
module tries to move updated files into place, it may receive a
"permission-denied" - because the user may no-longer "own" the
file's
directory (podman chown'ed it too!).
If you try to overcome this by (calculating the ID offsets and)
specifying UID/GID values to the template module, that can also fail
because the playbook is running as a user. Users aren't permitted to
(directly) chown things to user-namespaced IDs (see `podman unshare` below).
---quick user-namespace primer---
On the host, `/etc/sub{uid,gid}` contain per user ID range allocations,
which don't (typically) overlap each-other, and aren't (typically) used
by anything else on the host (though nothing actually verifies/blocks
this). It's like each user gets their own little (alternate) ID fiefdom
on the system.
The net effect is, a file or directory in a volume used by a rootless
container will be owned (from the host POV) by either the user's UID/GID
(corresponding to `root:root` in the container). Or, the UID/GID in the
container + an offset calculated from the second field of
`/etc/sub{uid,gid}`, bounded by summation of the third field
(effectively, the total number of IDs allocated to the range).
The wrapper `podman unshare <command>` lets a user "enter" their
allocated user-namespace on the host (for <command>), and use or chown
things to their heart's content (within their ID allocation). The
actual ownership (outside <command>) will be the values as described above.
12:28 p.m.
On 6/15/23 09:54, Chris Evich wrote:
I still haven't tried (since it seems really really hacky)
setting
`ansible_python_interpreter` to a wrapper script that execs `podman
unshare /usr/bin/python3 "$(a)"`. In theory this would work for Ansible
`template` tasks, as the config files would be rendered INSIDE the
user-namespace rather than outside.
I realize this is a really old thread, but is a tricky thing I finally
took time to experiment with. So I thought I'd share what I discovered.
As a reminder of the problem: How can you template files into a
rootless-podman volume on a host where you don't have root, and w/o
flapping[0] or needing to fuss with the user-namespaced ownership?
Assuming `/tmp/template.j2` exists with some test content (left up to
your imagination), here's a playbook that demonstrates a possible solution:
---cut playbook---
- hosts: localhost
connection: local # no ssh required
become: false # Don't sudo to root
tasks:
- name: User-namespaced tasks
vars:
# N/B: assumed to already exist on the inventory host
ansible_python_interpreter: '/tmp/unshare_python3.sh'
block:
- name: A testdir exists
file:
path: /tmp/testdir
state: directory
owner: 12345
group: 54321
- name: Render testfile into testdir
template:
src: '/tmp/template.j2'
dest: /tmp/testdir/testfile
owner: 54321
group: 12345
---cut playbook---
And the wrapper script that makes the magic happen.
---cut /tmp/unshare_python3.sh wrapper---
#!/bin/sh
exec /usr/bin/podman unshare $(type -p python3) "$@"
---cut /tmp/unshare_python3.sh wrapper---
After running the playbook, with a `/etc/sub{uid,gid}` entry of:
`cevich:100000:65536`
I end up with:
```
$ ls -la /tmp/testdir
total 4
drwxr-xr-x. 2 112344 154320 60 Aug 11 12:46 .
drwxrwxrwt. 56 root root 1160 Aug 11 13:02 ..
-rw-r--r--. 1 154320 112344 49 Aug 11 12:46 testfile
```
But a key thing to note is what happens when you run the playbook more
than once. Ansible will NOT needlessly "fix" the user-namespaced
owner/group due to it not matching the `ansible_user` `UID:GID`.
Hmmm...maybe I should write a blog on this?
[0]: Flapping - Given the same inputs & conditions, an operation that's
executed multiple times but doesn't always yield the same result.
---
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If there's a "hard-way", I'm the first one to implement it.
501
days inactive
568
days old
5 comments
2 participants
participants (2)
-
Chris Evich -
Johannes Kastl