Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) What is wrong?

Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote: > Hallo, > I notice some disk space discrepancy when running rootless podman containers. > I use dedicated fs for podman storage mountrd to ~/.local/share/containers. > df and du show consistent used disk space: >   island:named [master]> df -h ~/.local/share/containers >   /dev/mapper/sys-containers   117G          84G   32G           73% ~/.local/share/containers > >   island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} >   74G    /home/ivans/.local/share/containers/storage/vfs >   11G    /home/ivans/.local/share/containers/storage/volumes >   island:named [master]> > But space usage shown by podman system df is about 44% less than reported above: > > island:named [master]> podman system df > TYPE TOTAL ACTIVE SIZE RECLAIMABLE > Images 32 5 39.49GB 25.96GB (66%) > Containers 7 7 1.85GB 0B (0%) > Local Volumes 2 2 10.83GB 0B (0%) > > Volume space is practically same, it's vfs space (where as I understand images > and containers are located) that differs. > I also run buildah as same user, but buildah ls shows nothing. > I have ran podman system prune, but it reclaimed 0 bytes. > So is this extra space usage expected? Or is it sthing wrong with my storage? > Thanks and regards, > -- > Michael Ivanov > > _______________________________________________ > Podman mailing list --podman(a)lists.podman.io > To unsubscribe send an email topodman-leave(a)lists.podman.io _______________________________________________ Podman mailing list --podman(a)lists.podman.io To unsubscribe send an email topodman-leave(a)lists.podman.io

> You could do a podman system reset and then remove all content > from the storage with > rm -rf  ~/.local/share/containers > To make sure there is nothing hidden there, But that's almost exactly what I did: > I just purged the whole storage using podman system reset. > I verified that ~/.local/share/containers became empty > (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.

Best regards, On 29.12.2022 15:04, Daniel Walsh wrote: > On 12/27/22 13:19, Михаил Иванов wrote: >> Hallo again, >> I just purged the whole storage using podman system reset. >> I verified that ~/.local/share/containers became empty >> (only bolt database was still remaining using about 200Mb space) >> I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. >> podman system df reports space usage by images 4Gb and by containers 6Mb. >> podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) >> system df command shows that 32Gb is used on ~/.local/share/containers filesystem >> du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir >> This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, >> rest subdirs take anywhere from 94Mb to 646Mb >> When I try to diff -rw for directories of same size, I see only reports for missing >> symlink files, but never real file differences. >> No real activity was performed with podman apart from running these two containers. >> I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) >> >> What is wrong? > > I guess the first question I would have for you is why are you using > VFS storage? And not overlay or fuse-overlay? > > Could there be other containers or storage that is un-accounted for?  > Did you do any podman builds?  Or Buildah? > > > ``` > $ podman ps --all > CONTAINER ID  IMAGE COMMAND           CREATED     STATUS PORTS       > NAMES > 2e7070eacb0f  docker.io/library/alpine:latest  touch /dan/walsh  6 > days ago  Exited (0) 6 days ago nervous_noether > $ podman ps --all --external > CONTAINER ID  IMAGE COMMAND           CREATED         STATUS > PORTS       NAMES > 2e7070eacb0f  docker.io/library/alpine:latest  touch /dan/walsh  6 > days ago      Exited (0) 6 days ago nervous_noether > 9478fed6d8db  docker.io/library/alpine:latest buildah           14 > seconds ago Storage                            alpine-working-container > ``` > > You could do a podman system reset and then remove all content from > the storage with > > rm -rf  ~/.local/share/containers > >  To make sure there is nothing hidden there, > > > Other then that I am not sure what could be showing the difference in > storage. > >> Best regards, >> -- >> On 23.12.2022 19:43, Михаил Иванов wrote: >>> Hallo, >>> I notice some disk space discrepancy when running rootless podman containers. >>> I use dedicated fs for podman storage mountrd to ~/.local/share/containers. >>> df and du show consistent used disk space: >>>   island:named [master]> df -h ~/.local/share/containers >>>   /dev/mapper/sys-containers   117G          84G   32G           73% ~/.local/share/containers >>> >>>   island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} >>>   74G    /home/ivans/.local/share/containers/storage/vfs >>>   11G    /home/ivans/.local/share/containers/storage/volumes >>>   island:named [master]> >>> But space usage shown by podman system df is about 44% less than reported above: >>> >>> island:named [master]> podman system df >>> TYPE TOTAL ACTIVE SIZE RECLAIMABLE >>> Images 32 5 39.49GB 25.96GB (66%) >>> Containers 7 7 1.85GB 0B (0%) >>> Local Volumes 2 2 10.83GB 0B (0%) >>> >>> Volume space is practically same, it's vfs space (where as I understand images >>> and containers are located) that differs. >>> I also run buildah as same user, but buildah ls shows nothing. >>> I have ran podman system prune, but it reclaimed 0 bytes. >>> So is this extra space usage expected? Or is it sthing wrong with my storage? >>> Thanks and regards, >>> -- >>> Michael Ivanov >>> >>> _______________________________________________ >>> Podman mailing list --podman(a)lists.podman.io >>> To unsubscribe send an email topodman-leave(a)lists.podman.io >> >> _______________________________________________ >> Podman mailing list --podman(a)lists.podman.io >> To unsubscribe send an email topodman-leave(a)lists.podman.io > > > > _______________________________________________ > Podman mailing list --podman(a)lists.podman.io > To unsubscribe send an email topodman-leave(a)lists.podman.io _______________________________________________ Podman mailing list --podman(a)lists.podman.io To unsubscribe send an email topodman-leave(a)lists.podman.io

I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead. When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones. My debian is the latest one (sid) and it has the following fuse-related packages: ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) podman is 4.3.1 For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package). Rgrds,

On 03.01.2023 18:25, Daniel Walsh wrote: > On 12/30/22 08:35, Михаил Иванов wrote: >> > You could do a podman system reset and then remove all content >> > from the storage with >> > rm -rf  ~/.local/share/containers >> > To make sure there is nothing hidden there, >> But that's almost exactly what I did: >> > I just purged the whole storage using podman system reset. >> > I verified that ~/.local/share/containers became empty >> > (only bolt database was still remaining using about 200Mb space) >> >> I'm using whatever storage was provided by default podman install >> (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to >> different type? I assumed this has to be done in storage.conf but >> this file is not present anywhere at all. > I have no idea why debian would be choosing VFS, unless this is an > older version of debian and did not support rootless overaly.  You > could try installling fuse-overlayfs and doing another reset, then > Podman info should show you using overlay with fuse-overlayfs. >> Best regards, >> On 29.12.2022 15:04, Daniel Walsh wrote: >>> On 12/27/22 13:19, Михаил Иванов wrote: >>>> Hallo again, >>>> I just purged the whole storage using podman system reset. >>>> I verified that ~/.local/share/containers became empty >>>> (only bolt database was still remaining using about 200Mb space) >>>> I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. >>>> podman system df reports space usage by images 4Gb and by containers 6Mb. >>>> podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) >>>> system df command shows that 32Gb is used on ~/.local/share/containers filesystem >>>> du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir >>>> This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, >>>> rest subdirs take anywhere from 94Mb to 646Mb >>>> When I try to diff -rw for directories of same size, I see only reports for missing >>>> symlink files, but never real file differences. >>>> No real activity was performed with podman apart from running these two containers. >>>> I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) >>>> >>>> What is wrong? >>> >>> I guess the first question I would have for you is why are you >>> using VFS storage? And not overlay or fuse-overlay? >>> >>> Could there be other containers or storage that is un-accounted >>> for?  Did you do any podman builds?  Or Buildah? >>> >>> >>> ``` >>> $ podman ps --all >>> CONTAINER ID  IMAGE COMMAND           CREATED     STATUS >>> PORTS       NAMES >>> 2e7070eacb0f  docker.io/library/alpine:latest  touch /dan/walsh  6 >>> days ago  Exited (0) 6 days ago nervous_noether >>> $ podman ps --all --external >>> CONTAINER ID  IMAGE COMMAND           CREATED         STATUS >>> PORTS       NAMES >>> 2e7070eacb0f  docker.io/library/alpine:latest  touch /dan/walsh  6 >>> days ago      Exited (0) 6 days ago              nervous_noether >>> 9478fed6d8db  docker.io/library/alpine:latest buildah           14 >>> seconds ago Storage alpine-working-container >>> ``` >>> >>> You could do a podman system reset and then remove all content from >>> the storage with >>> >>> rm -rf  ~/.local/share/containers >>> >>>  To make sure there is nothing hidden there, >>> >>> >>> Other then that I am not sure what could be showing the difference >>> in storage. >>> >>>> Best regards, >>>> -- >>>> On 23.12.2022 19:43, Михаил Иванов wrote: >>>>> Hallo, >>>>> I notice some disk space discrepancy when running rootless podman containers. >>>>> I use dedicated fs for podman storage mountrd to ~/.local/share/containers. >>>>> df and du show consistent used disk space: >>>>>   island:named [master]> df -h ~/.local/share/containers >>>>>   /dev/mapper/sys-containers   117G          84G   32G           73% ~/.local/share/containers >>>>> >>>>>   island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} >>>>>   74G    /home/ivans/.local/share/containers/storage/vfs >>>>>   11G    /home/ivans/.local/share/containers/storage/volumes >>>>>   island:named [master]> >>>>> But space usage shown by podman system df is about 44% less than reported above: >>>>> >>>>> island:named [master]> podman system df >>>>> TYPE TOTAL ACTIVE SIZE RECLAIMABLE >>>>> Images 32 5 39.49GB 25.96GB (66%) >>>>> Containers 7 7 1.85GB 0B (0%) >>>>> Local Volumes 2 2 10.83GB 0B (0%) >>>>> >>>>> Volume space is practically same, it's vfs space (where as I understand images >>>>> and containers are located) that differs. >>>>> I also run buildah as same user, but buildah ls shows nothing. >>>>> I have ran podman system prune, but it reclaimed 0 bytes. >>>>> So is this extra space usage expected? Or is it sthing wrong with my storage? >>>>> Thanks and regards, >>>>> -- >>>>> Michael Ivanov >>>>> >>>>> _______________________________________________ >>>>> Podman mailing list --podman(a)lists.podman.io >>>>> To unsubscribe send an email topodman-leave(a)lists.podman.io >>>> >>>> _______________________________________________ >>>> Podman mailing list --podman(a)lists.podman.io >>>> To unsubscribe send an email topodman-leave(a)lists.podman.io >>> >>> >>> >>> _______________________________________________ >>> Podman mailing list --podman(a)lists.podman.io >>> To unsubscribe send an email topodman-leave(a)lists.podman.io >> >> _______________________________________________ >> Podman mailing list --podman(a)lists.podman.io >> To unsubscribe send an email topodman-leave(a)lists.podman.io > > > > _______________________________________________ > Podman mailing list --podman(a)lists.podman.io > To unsubscribe send an email topodman-leave(a)lists.podman.io

Daniel Walsh<dwalsh(a)redhat.com> writes: > On 1/9/23 14:29, Михаил Иванов wrote: > > I repeated everything again: podman system reset, then deleted everything in > ~/.local/share/containers to be on the safe side and ran buildah to create an image. > No overlay directories appeared under ~/.local/share/containers, only vfs* instead. > > When I run same buildah command as root, overlay* directories do appear under > /var/lib/containers and no vfs ones. > > My debian is the latest one (sid) and it has the following fuse-related packages: > > ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers > ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) > ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) > ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) > > podman is 4.3.1 > > For both root and rootless configs podman system info reports runc runtime > (runc_1.1.4+ds1-1+b1_amd64 package). > > Rgrds, > > Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested. > On 03.01.2023 18:25, Daniel Walsh wrote: > > On 12/30/22 08:35, Михаил Иванов wrote: > > > You could do a podman system reset and then remove all content >> from the storage with >> rm -rf ~/.local/share/containers >> To make sure there is nothing hidden there, > But that's almost exactly what I did: >> I just purged the whole storage using podman system reset. >> I verified that ~/.local/share/containers became empty >> (only bolt database was still remaining using about 200Mb space) > I'm using whatever storage was provided by default podman install > (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to > different type? I assumed this has to be done in storage.conf but > this file is not present anywhere at all. > > I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could > try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. > > Best regards, > On 29.12.2022 15:04, Daniel Walsh wrote: > > On 12/27/22 13:19, Михаил Иванов wrote: > > Hallo again, > I just purged the whole storage using podman system reset. > I verified that ~/.local/share/containers became empty > (only bolt database was still remaining using about 200Mb space) > I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. > podman system df reports space usage by images 4Gb and by containers 6Mb. > podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) > system df command shows that 32Gb is used on ~/.local/share/containers filesystem > du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir > This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, > rest subdirs take anywhere from 94Mb to 646Mb > When I try to diff -rw for directories of same size, I see only reports for missing > symlink files, but never real file differences. > No real activity was performed with podman apart from running these two containers. > I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) > > What is wrong? > > I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay? > > Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah? > > ``` > $ podman ps --all > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether > $ podman ps --all --external > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether > 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container > ``` > > You could do a podman system reset and then remove all content from the storage with > > rm -rf ~/.local/share/containers > > To make sure there is nothing hidden there, > > Other then that I am not sure what could be showing the difference in storage. > > Best regards, > -- > On 23.12.2022 19:43, Михаил Иванов wrote: > > Hallo, > I notice some disk space discrepancy when running rootless podman containers. > I use dedicated fs for podman storage mountrd to ~/.local/share/containers. > df and du show consistent used disk space: > island:named [master]> df -h ~/.local/share/containers > /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers > > island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} > 74G /home/ivans/.local/share/containers/storage/vfs > 11G /home/ivans/.local/share/containers/storage/volumes > island:named [master]> > But space usage shown by podman system df is about 44% less than reported above: > > island:named [master]> podman system df > TYPE TOTAL ACTIVE SIZE RECLAIMABLE > Images 32 5 39.49GB 25.96GB (66%) > Containers 7 7 1.85GB 0B (0%) > Local Volumes 2 2 10.83GB 0B (0%) > > Volume space is practically same, it's vfs space (where as I understand images > and containers are located) that differs. > I also run buildah as same user, but buildah ls shows nothing. > I have ran podman system prune, but it reclaimed 0 bytes. > So is this extra space usage expected? Or is it sthing wrong with my storage? > Thanks and regards,

Please run this command AFTER running podman system reset On 1/10/23 13:32, Михаил Иванов wrote: > I have run podman both as root and rootless with --log-level debug. Rootless reports the following > (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): > island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 > INFO[0000] podman filtering at log level debug > DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) > DEBU[0000] Merged system config "/usr/share/containers/containers.conf" > DEBU[0000] Using conmon: "/usr/bin/conmon" > DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db > DEBU[0000] Using graph driver vfs > DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage > DEBU[0000] Using run root /run/user/1007/containers > DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod > DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp > DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes > DEBU[0000] Set libpod namespace to "" > DEBU[0000] [graphdriver] trying provided driver "vfs" > DEBU[0000] Initializing event backend journald > DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument > DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument > DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument > DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument > DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument > DEBU[0000] Using OCI runtime "/usr/bin/runc" > INFO[0000] Setting parallel job count to 25 > DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) > DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage > DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... > DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" > DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } > DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 > DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" > Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... > Same for root podman attempt: > root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 > INFO[0000] podman filtering at log level debug > DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) > DEBU[0000] Merged system config "/usr/share/containers/containers.conf" > DEBU[0000] Using conmon: "/usr/bin/conmon" > DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db > DEBU[0000] Using graph driver > DEBU[0000] Using graph root /var/lib/containers/storage > DEBU[0000] Using run root /run/containers/storage > DEBU[0000] Using static dir /var/lib/containers/storage/libpod > DEBU[0000] Using tmp dir /run/libpod > DEBU[0000] Using volume path /var/lib/containers/storage/volumes > DEBU[0000] Set libpod namespace to "" > DEBU[0000] overlay: test mount with multiple lowers succeeded > DEBU[0000] Cached value indicated that overlay is supported > DEBU[0000] overlay: test mount indicated that metacopy is not being used > DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false > DEBU[0000] Initializing event backend journald > DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument > DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument > DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument > DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument > DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument > DEBU[0000] Using OCI runtime "/usr/bin/runc" > DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 > 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} > DEBU[0000] Successfully loaded 1 networks > DEBU[0000] Podman detected system restart - performing state refresh > INFO[0000] Setting parallel job count to 25 > DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) > DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage > DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... > DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... > DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" > DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } > DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 > DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" > Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... > Best regards, > On 10.01.2023 15:40, Giuseppe Scrivano wrote: >> Daniel Walsh&lt;dwalsh(a)redhat.com&gt; writes: >> >>> On 1/9/23 14:29, Михаил Иванов wrote: >>> >>> I repeated everything again: podman system reset, then deleted everything in >>> ~/.local/share/containers to be on the safe side and ran buildah to create an image. >>> No overlay directories appeared under ~/.local/share/containers, only vfs* instead. >>> >>> When I run same buildah command as root, overlay* directories do appear under >>> /var/lib/containers and no vfs ones. >>> >>> My debian is the latest one (sid) and it has the following fuse-related packages: >>> >>> ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers >>> ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) >>> ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) >>> ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) >>> >>> podman is 4.3.1 >>> >>> For both root and rootless configs podman system info reports runc runtime >>> (runc_1.1.4+ds1-1+b1_amd64 package). >>> >>> Rgrds, >>> >>> Giuseppe any ideas? >> not sure why overlay or fuse-overlayfs are not picked. It would be >> helpful to open an upstream issue and attach the output of `podman >> --log-level debug ...` in addition to all the other information that is >> requested. >> >> >>> On 03.01.2023 18:25, Daniel Walsh wrote: >>> >>> On 12/30/22 08:35, Михаил Иванов wrote: >>> >>> > You could do a podman system reset and then remove all content >>>> from the storage with >>>> rm -rf ~/.local/share/containers >>>> To make sure there is nothing hidden there, >>> But that's almost exactly what I did: >>>> I just purged the whole storage using podman system reset. >>>> I verified that ~/.local/share/containers became empty >>>> (only bolt database was still remaining using about 200Mb space) >>> I'm using whatever storage was provided by default podman install >>> (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to >>> different type? I assumed this has to be done in storage.conf but >>> this file is not present anywhere at all. >>> >>> I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could >>> try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. >>> >>> Best regards, >>> On 29.12.2022 15:04, Daniel Walsh wrote: >>> >>> On 12/27/22 13:19, Михаил Иванов wrote: >>> >>> Hallo again, >>> I just purged the whole storage using podman system reset. >>> I verified that ~/.local/share/containers became empty >>> (only bolt database was still remaining using about 200Mb space) >>> I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. >>> podman system df reports space usage by images 4Gb and by containers 6Mb. >>> podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) >>> system df command shows that 32Gb is used on ~/.local/share/containers filesystem >>> du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir >>> This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, >>> rest subdirs take anywhere from 94Mb to 646Mb >>> When I try to diff -rw for directories of same size, I see only reports for missing >>> symlink files, but never real file differences. >>> No real activity was performed with podman apart from running these two containers. >>> I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) >>> >>> What is wrong? >>> >>> I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay? >>> >>> Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah? >>> >>> ``` >>> $ podman ps --all >>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES >>> 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether >>> $ podman ps --all --external >>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES >>> 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether >>> 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container >>> ``` >>> >>> You could do a podman system reset and then remove all content from the storage with >>> >>> rm -rf ~/.local/share/containers >>> >>> To make sure there is nothing hidden there, >>> >>> Other then that I am not sure what could be showing the difference in storage. >>> >>> Best regards, >>> -- >>> On 23.12.2022 19:43, Михаил Иванов wrote: >>> >>> Hallo, >>> I notice some disk space discrepancy when running rootless podman containers. >>> I use dedicated fs for podman storage mountrd to ~/.local/share/containers. >>> df and du show consistent used disk space: >>> island:named [master]> df -h ~/.local/share/containers >>> /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers >>> >>> island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} >>> 74G /home/ivans/.local/share/containers/storage/vfs >>> 11G /home/ivans/.local/share/containers/storage/volumes >>> island:named [master]> >>> But space usage shown by podman system df is about 44% less than reported above: >>> >>> island:named [master]> podman system df >>> TYPE TOTAL ACTIVE SIZE RECLAIMABLE >>> Images 32 5 39.49GB 25.96GB (66%) >>> Containers 7 7 1.85GB 0B (0%) >>> Local Volumes 2 2 10.83GB 0B (0%) >>> >>> Volume space is practically same, it's vfs space (where as I understand images >>> and containers are located) that differs. >>> I also run buildah as same user, but buildah ls shows nothing. >>> I have ran podman system prune, but it reclaimed 0 bytes. >>> So is this extra space usage expected? Or is it sthing wrong with my storage? >>> Thanks and regards, _______________________________________________ Podman mailing list --podman(a)lists.podman.io To unsubscribe send an email topodman-leave(a)lists.podman.io

okay, it finally started to use overlays when I explicitly requested it in ~/.config/containers/storage.conf as follows: [storage] driver = "overlay" [storage.options] mount_program = "/usr/bin/fuse-overlayfs" Normally I do not have storage.conf file, neither as root nor as rootless. Best regards, On 11.01.2023 10:12, Михаил Иванов wrote: > Of course I did it after system reset in both cases. > On 10.01.2023 23:28, Daniel Walsh wrote: >> Please run this command AFTER running podman system reset >> >> On 1/10/23 13:32, Михаил Иванов wrote: >>> I have run podman both as root and rootless with --log-level debug. Rootless reports the following >>> (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): >>> island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 >>> INFO[0000] podman filtering at log level debug >>> DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) >>> DEBU[0000] Merged system config "/usr/share/containers/containers.conf" >>> DEBU[0000] Using conmon: "/usr/bin/conmon" >>> DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db >>> DEBU[0000] Using graph driver vfs >>> DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage >>> DEBU[0000] Using run root /run/user/1007/containers >>> DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod >>> DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp >>> DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes >>> DEBU[0000] Set libpod namespace to "" >>> DEBU[0000] [graphdriver] trying provided driver "vfs" >>> DEBU[0000] Initializing event backend journald >>> DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument >>> DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument >>> DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument >>> DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument >>> DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument >>> DEBU[0000] Using OCI runtime "/usr/bin/runc" >>> INFO[0000] Setting parallel job count to 25 >>> DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) >>> DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage >>> DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... >>> DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" >>> DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } >>> DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 >>> DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" >>> Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... >>> Same for root podman attempt: >>> root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 >>> INFO[0000] podman filtering at log level debug >>> DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) >>> DEBU[0000] Merged system config "/usr/share/containers/containers.conf" >>> DEBU[0000] Using conmon: "/usr/bin/conmon" >>> DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db >>> DEBU[0000] Using graph driver >>> DEBU[0000] Using graph root /var/lib/containers/storage >>> DEBU[0000] Using run root /run/containers/storage >>> DEBU[0000] Using static dir /var/lib/containers/storage/libpod >>> DEBU[0000] Using tmp dir /run/libpod >>> DEBU[0000] Using volume path /var/lib/containers/storage/volumes >>> DEBU[0000] Set libpod namespace to "" >>> DEBU[0000] overlay: test mount with multiple lowers succeeded >>> DEBU[0000] Cached value indicated that overlay is supported >>> DEBU[0000] overlay: test mount indicated that metacopy is not being used >>> DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false >>> DEBU[0000] Initializing event backend journald >>> DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument >>> DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument >>> DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument >>> DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument >>> DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument >>> DEBU[0000] Using OCI runtime "/usr/bin/runc" >>> DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 >>> 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} >>> DEBU[0000] Successfully loaded 1 networks >>> DEBU[0000] Podman detected system restart - performing state refresh >>> INFO[0000] Setting parallel job count to 25 >>> DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) >>> DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage >>> DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... >>> DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... >>> DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" >>> DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } >>> DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 >>> DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" >>> Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... >>> Best regards, >>> On 10.01.2023 15:40, Giuseppe Scrivano wrote: >>>> Daniel Walsh&lt;dwalsh(a)redhat.com&gt; writes: >>>> >>>>> On 1/9/23 14:29, Михаил Иванов wrote: >>>>> >>>>> I repeated everything again: podman system reset, then deleted everything in >>>>> ~/.local/share/containers to be on the safe side and ran buildah to create an image. >>>>> No overlay directories appeared under ~/.local/share/containers, only vfs* instead. >>>>> >>>>> When I run same buildah command as root, overlay* directories do appear under >>>>> /var/lib/containers and no vfs ones. >>>>> >>>>> My debian is the latest one (sid) and it has the following fuse-related packages: >>>>> >>>>> ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers >>>>> ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) >>>>> ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) >>>>> ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) >>>>> >>>>> podman is 4.3.1 >>>>> >>>>> For both root and rootless configs podman system info reports runc runtime >>>>> (runc_1.1.4+ds1-1+b1_amd64 package). >>>>> >>>>> Rgrds, >>>>> >>>>> Giuseppe any ideas? >>>> not sure why overlay or fuse-overlayfs are not picked. It would be >>>> helpful to open an upstream issue and attach the output of `podman >>>> --log-level debug ...` in addition to all the other information that is >>>> requested. >>>> >>>> >>>>> On 03.01.2023 18:25, Daniel Walsh wrote: >>>>> >>>>> On 12/30/22 08:35, Михаил Иванов wrote: >>>>> >>>>> > You could do a podman system reset and then remove all content >>>>>> from the storage with >>>>>> rm -rf ~/.local/share/containers >>>>>> To make sure there is nothing hidden there, >>>>> But that's almost exactly what I did: >>>>>> I just purged the whole storage using podman system reset. >>>>>> I verified that ~/.local/share/containers became empty >>>>>> (only bolt database was still remaining using about 200Mb space) >>>>> I'm using whatever storage was provided by default podman install >>>>> (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to >>>>> different type? I assumed this has to be done in storage.conf but >>>>> this file is not present anywhere at all. >>>>> >>>>> I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could >>>>> try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. >>>>> >>>>> Best regards, >>>>> On 29.12.2022 15:04, Daniel Walsh wrote: >>>>> >>>>> On 12/27/22 13:19, Михаил Иванов wrote: >>>>> >>>>> Hallo again, >>>>> I just purged the whole storage using podman system reset. >>>>> I verified that ~/.local/share/containers became empty >>>>> (only bolt database was still remaining using about 200Mb space) >>>>> I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. >>>>> podman system df reports space usage by images 4Gb and by containers 6Mb. >>>>> podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) >>>>> system df command shows that 32Gb is used on ~/.local/share/containers filesystem >>>>> du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir >>>>> This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, >>>>> rest subdirs take anywhere from 94Mb to 646Mb >>>>> When I try to diff -rw for directories of same size, I see only reports for missing >>>>> symlink files, but never real file differences. >>>>> No real activity was performed with podman apart from running these two containers. >>>>> I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) >>>>> >>>>> What is wrong? >>>>> >>>>> I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay? >>>>> >>>>> Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah? >>>>> >>>>> ``` >>>>> $ podman ps --all >>>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES >>>>> 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether >>>>> $ podman ps --all --external >>>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES >>>>> 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether >>>>> 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container >>>>> ``` >>>>> >>>>> You could do a podman system reset and then remove all content from the storage with >>>>> >>>>> rm -rf ~/.local/share/containers >>>>> >>>>> To make sure there is nothing hidden there, >>>>> >>>>> Other then that I am not sure what could be showing the difference in storage. >>>>> >>>>> Best regards, >>>>> -- >>>>> On 23.12.2022 19:43, Михаил Иванов wrote: >>>>> >>>>> Hallo, >>>>> I notice some disk space discrepancy when running rootless podman containers. >>>>> I use dedicated fs for podman storage mountrd to ~/.local/share/containers. >>>>> df and du show consistent used disk space: >>>>> island:named [master]> df -h ~/.local/share/containers >>>>> /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers >>>>> >>>>> island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} >>>>> 74G /home/ivans/.local/share/containers/storage/vfs >>>>> 11G /home/ivans/.local/share/containers/storage/volumes >>>>> island:named [master]> >>>>> But space usage shown by podman system df is about 44% less than reported above: >>>>> >>>>> island:named [master]> podman system df >>>>> TYPE TOTAL ACTIVE SIZE RECLAIMABLE >>>>> Images 32 5 39.49GB 25.96GB (66%) >>>>> Containers 7 7 1.85GB 0B (0%) >>>>> Local Volumes 2 2 10.83GB 0B (0%) >>>>> >>>>> Volume space is practically same, it's vfs space (where as I understand images >>>>> and containers are located) that differs. >>>>> I also run buildah as same user, but buildah ls shows nothing. >>>>> I have ran podman system prune, but it reclaimed 0 bytes. >>>>> So is this extra space usage expected? Or is it sthing wrong with my storage? >>>>> Thanks and regards, >> >> >> >> _______________________________________________ >> Podman mailing list --podman(a)lists.podman.io >> To unsubscribe send an email topodman-leave(a)lists.podman.io > > _______________________________________________ > Podman mailing list --podman(a)lists.podman.io > To unsubscribe send an email topodman-leave(a)lists.podman.io _______________________________________________ Podman mailing list --podman(a)lists.podman.io To unsubscribe send an email topodman-leave(a)lists.podman.io

On Tue, Jan 3, 2023 at 10:27 AM Daniel Walsh <dwalsh(a)redhat.com&gt; wrote: On 12/30/22 08:35, Михаил Иванов wrote: > > You could do a podman system reset and then remove all content > > from the storage with > > rm -rf  ~/.local/share/containers > > To make sure there is nothing hidden there, > But that's almost exactly what I did: > > I just purged the whole storage using podman system reset. > > I verified that ~/.local/share/containers became empty > > (only bolt database was still remaining using about 200Mb space) > > I'm using whatever storage was provided by default podman install > (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to > different type? I assumed this has to be done in storage.conf but > this file is not present anywhere at all. I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly.  You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. Dan, In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a... This file is installed verbatim to /usr/share/containers/storage.conf. Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else? Thanks, -rt

On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh <dwalsh(a)redhat.com&gt; wrote: On 1/24/23 03:47, Reinhard Tartler wrote: Dan, In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a... This file is installed verbatim to /usr/share/containers/storage.conf. Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else? Thanks, -rt That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default? If not then it would fail over to VFS if fuse-overlayfs is not installed. I'm a bit confused about what you mean with that. In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default. What else is required for rootless overlay? Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience?

Is native overlay available in rootless mode? When I run podman as root there's no problem, overlayfs is picked up as default in debian. VFS is selected as default only in rootless mode. Rgrds, On 25.01.2023 14:03, Giuseppe Scrivano wrote: > Reinhard Tartler&lt;siretart(a)gmail.com&gt; writes: > >> On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh&lt;dwalsh(a)redhat.com&gt; wrote: >> >> On 1/24/23 03:47, Reinhard Tartler wrote: >> >> Dan, >> >> In Debian, I've chosen to just go with the upstream defaults: >> https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a... >> >> This file is installed verbatim to /usr/share/containers/storage.conf. >> >> Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else? >> >> Thanks, >> -rt >> >> That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default? >> >> If not then it would fail over to VFS if fuse-overlayfs is not installed. >> >> I'm a bit confused about what you mean with that. >> >> In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not >> install it by configuring apt to not install "Recommends" by default. >> >> What else is required for rootless overlay? >> >> Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program >> option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct >> mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an >> appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience? > what issue do you see if you use native overlay? > > Podman prefers native overlay if it is available, since it is faster. > If not, it tries fuse-overlays and if it is not available, it falls back > to vfs. > > Could you try from a fresh storage though? If fuse-overlayfs was > already used, then Podman will continue using it even if native overlay > is available, since the storage metadata is slightly different. > > Thanks, > Giuseppe > _______________________________________________ > Podman mailing list --podman(a)lists.podman.io > To unsubscribe send an email topodman-leave(a)lists.podman.io