[Podman] Re: Follow-up: Rootless storage usage

Daniel Walsh<dwalsh(a)redhat.com> writes: > On 1/9/23 14:29, Михаил Иванов wrote: > > I repeated everything again: podman system reset, then deleted everything in > ~/.local/share/containers to be on the safe side and ran buildah to create an image. > No overlay directories appeared under ~/.local/share/containers, only vfs* instead. > > When I run same buildah command as root, overlay* directories do appear under > /var/lib/containers and no vfs ones. > > My debian is the latest one (sid) and it has the following fuse-related packages: > > ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers > ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) > ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) > ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) > > podman is 4.3.1 > > For both root and rootless configs podman system info reports runc runtime > (runc_1.1.4+ds1-1+b1_amd64 package). > > Rgrds, > > Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested. > On 03.01.2023 18:25, Daniel Walsh wrote: > > On 12/30/22 08:35, Михаил Иванов wrote: > > > You could do a podman system reset and then remove all content >> from the storage with >> rm -rf ~/.local/share/containers >> To make sure there is nothing hidden there, > But that's almost exactly what I did: >> I just purged the whole storage using podman system reset. >> I verified that ~/.local/share/containers became empty >> (only bolt database was still remaining using about 200Mb space) > I'm using whatever storage was provided by default podman install > (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to > different type? I assumed this has to be done in storage.conf but > this file is not present anywhere at all. > > I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could > try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. > > Best regards, > On 29.12.2022 15:04, Daniel Walsh wrote: > > On 12/27/22 13:19, Михаил Иванов wrote: > > Hallo again, > I just purged the whole storage using podman system reset. > I verified that ~/.local/share/containers became empty > (only bolt database was still remaining using about 200Mb space) > I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. > podman system df reports space usage by images 4Gb and by containers 6Mb. > podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) > system df command shows that 32Gb is used on ~/.local/share/containers filesystem > du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir > This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, > rest subdirs take anywhere from 94Mb to 646Mb > When I try to diff -rw for directories of same size, I see only reports for missing > symlink files, but never real file differences. > No real activity was performed with podman apart from running these two containers. > I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) > > What is wrong? > > I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay? > > Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah? > > ``` > $ podman ps --all > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether > $ podman ps --all --external > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether > 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container > ``` > > You could do a podman system reset and then remove all content from the storage with > > rm -rf ~/.local/share/containers > > To make sure there is nothing hidden there, > > Other then that I am not sure what could be showing the difference in storage. > > Best regards, > -- > On 23.12.2022 19:43, Михаил Иванов wrote: > > Hallo, > I notice some disk space discrepancy when running rootless podman containers. > I use dedicated fs for podman storage mountrd to ~/.local/share/containers. > df and du show consistent used disk space: > island:named [master]> df -h ~/.local/share/containers > /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers > > island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} > 74G /home/ivans/.local/share/containers/storage/vfs > 11G /home/ivans/.local/share/containers/storage/volumes > island:named [master]> > But space usage shown by podman system df is about 44% less than reported above: > > island:named [master]> podman system df > TYPE TOTAL ACTIVE SIZE RECLAIMABLE > Images 32 5 39.49GB 25.96GB (66%) > Containers 7 7 1.85GB 0B (0%) > Local Volumes 2 2 10.83GB 0B (0%) > > Volume space is practically same, it's vfs space (where as I understand images > and containers are located) that differs. > I also run buildah as same user, but buildah ls shows nothing. > I have ran podman system prune, but it reclaimed 0 bytes. > So is this extra space usage expected? Or is it sthing wrong with my storage? > Thanks and regards,