9:40 a.m.
Dear Podman users and developers
I wonder if you have encountered a similar problem to mine and whether
you may have a solution.
Description of the problem:
I cannot run containers where non-root users of the container (e.g. uid
101 of nginx for a nginx container) have possibility to write to a
mounted folder from virtualbox. Even when mounting the shared folder
with full writing permissions (dmode/fmode 0777), the new files created
appear as belonging to nobody (uid 65534). This prevents any application
which creates new files or directories (e.g. wordpress).
The setup:
* I have a linux virtual machine (Alpine 3.11.6, tested on both 5.4.27
and 5.4.34 kernels) running over a linux host (Ubuntu 18.04.4 LTS) on
which I create and run rootless pods and containers.
* virtualbox version: 5.2.34
* podman version: tried both 1.8.2 and 1.9.1
* shared folder and its contents are owned by 1001:1001 on the host
* the folder is mounted in the vm with the command /sbin/mount.vboxsf -o
rw,uid=100100,gid=1001,dmode=0775,fmode=0664 wpcontent
/media/sf_wpcontent
* the user running the container in the vm has id/gid 1001:1001 and
subuid 100000:65536
Investigation:
I'll paste the result of commands run at three levels: the host, the vm
and the container.
* This is on the underlying host:
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
$ touch uploads/sites/3/2020/host-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
-rw-rw-r-- 1 1001 1001 0 Apr 22 15:10 host-touched
$ sudo chown 100100:100100 uploads/sites/3/2020/host-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
-rw-rw-r-- 1 100100 100100 0 Apr 22 15:10 host-touched
* This is in the VM:
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
$ touch /media/sf_wpcontent/uploads/sites/3/2020/vm-touched
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:28 vm-touched
$ chown 100100:100100
/media/sf_wpcontent/uploads/sites/3/2020/vm-touched
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:28 vm-touched
This is in the container:
# ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
# touch uploads/sites/3/2020/cont-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:31 cont-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
# chown 100100:100100 uploads/sites/3/2020/cont-
touched
chown: uploads/sites/3/2020/cont-touched: Invalid argument
# chown nginx:nginx uploads/sites/3/2020/cont-to
uched
# ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:31 cont-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
rm -f uploads/sites/3/2020/cont-touched
rm: can't remove 'uploads/sites/3/2020/cont-touched': Value too large
for data type
The behaviour is inconsistent, for example:
in the vm:
$ ls -ln uploads/sites/
total 16
drwxrwxr-x 1 100100 1001 4096 Apr 22 15:52 2
drwxrwxr-x 1 100100 1001 4096 Apr 22 08:52 3
drwxrwxr-x 1 100100 1001 4096 Jan 21 14:30 4
drwxrwxr-x 1 100100 1001 4096 Jun 14 2017 5
inside the container:
# ls -ln uploads/sites/
total 16
drwxrwxr-x 1 65534 1001 4096 Apr 22 15:52 2
drwxrwxr-x 1 101 0 4096 Apr 22 08:52 3
drwxrwxr-x 1 65534 1001 4096 Jan 21 14:30 4
drwxrwxr-x 1 65534 1001 4096 Jun 14 2017 5
To clear the inconsistencies I can use the command (as vm's root):
# echo 3 > /proc/sys/vm/drop_caches
This appears to "correct" the problem by realigning the id of the
directories. But everytime a new file or directory is created, they are
still created as user nobody (65534).
Even for applications where no new files/directories are created (e.g.
appending to existing file), the drop_caches command would need to be
applied everytime the machine is rebooted, thus rendering the whole
system not really stable/usable.
The only two "solutions" that "work" are:
1) to have containers where the services run as root. Thus not using
subuid and files are created as the user running podman. Mounting the
shared folder with uid equal to the podman user is ok.
Obviously this is not always possible/desirable, as containers often
have a different user running the command.
2) mounting the shared folder with full permissions and without using a
subuid, i.e. like this:
/sbin/mount.vboxsf -o rw,uid=1001,gid=101,dmode=0777,fmode=0555
wpcontent /media/sf_wpcontent
this is obviously far from desirable, as it allows any user of the vm
permission to write into the folder or delete files
If you have used rootless containers writing on shared virtual folders
and have found a working solution, please let me know.
Thank you very much!
Best regards
Giuseppe
PS: I've also tried adding a user with uid/gid 100100:100100 in the vm.
This does not make any difference to the problems reported above
PPS: I've also tried changing id/gid of host folder or of the user in
the vm running podman. No changes.