Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above: island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%) Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8) What is wrong? Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay? Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah? ``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ``` You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there,
But that's almost exactly what I did:
I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all. Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all. I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs. Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead. When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones. My debian is the latest one (sid) and it has the following fuse-related packages: ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version) podman is 4.3.1 For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package). Rgrds, On 03.01.2023 18:25, Daniel Walsh wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all. I have no idea why debian would be choosing VFS, unless this is an
On 12/30/22 08:35, Михаил Иванов wrote: older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds, Giuseppe any ideas? On 03.01.2023 18:25, Daniel Walsh wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all. I have no idea why debian would be choosing VFS, unless this is an
On 12/30/22 08:35, Михаил Иванов wrote: older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards, -- Michael Ivanov
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
Daniel Walsh <dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas?
not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
I have run podman both as root and rootless with --log-level debug. Rootless reports the following (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage DEBU[0000] Using run root /run/user/1007/containers DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Same for root podman attempt: root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] overlay: test mount with multiple lowers succeeded DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] overlay: test mount indicated that metacopy is not being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Podman detected system restart - performing state refresh INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Best regards, On 10.01.2023 15:40, Giuseppe Scrivano wrote:
Daniel Walsh<dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
Please run this command AFTER running podman system reset On 1/10/23 13:32, Михаил Иванов wrote:
I have run podman both as root and rootless with --log-level debug. Rootless reports the following (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage DEBU[0000] Using run root /run/user/1007/containers DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Same for root podman attempt: root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] overlay: test mount with multiple lowers succeeded DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] overlay: test mount indicated that metacopy is not being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Podman detected system restart - performing state refresh INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Best regards, On 10.01.2023 15:40, Giuseppe Scrivano wrote:
Daniel Walsh<dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
Of course I did it after system reset in both cases. On 10.01.2023 23:28, Daniel Walsh wrote:
Please run this command AFTER running podman system reset
On 1/10/23 13:32, Михаил Иванов wrote:
I have run podman both as root and rootless with --log-level debug. Rootless reports the following (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage DEBU[0000] Using run root /run/user/1007/containers DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Same for root podman attempt: root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] overlay: test mount with multiple lowers succeeded DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] overlay: test mount indicated that metacopy is not being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Podman detected system restart - performing state refresh INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Best regards, On 10.01.2023 15:40, Giuseppe Scrivano wrote:
Daniel Walsh<dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
okay, it finally started to use overlays when I explicitly requested it in ~/.config/containers/storage.conf as follows: [storage] driver = "overlay" [storage.options] mount_program = "/usr/bin/fuse-overlayfs" Normally I do not have storage.conf file, neither as root nor as rootless. Best regards, On 11.01.2023 10:12, Михаил Иванов wrote:
Of course I did it after system reset in both cases. On 10.01.2023 23:28, Daniel Walsh wrote:
Please run this command AFTER running podman system reset
On 1/10/23 13:32, Михаил Иванов wrote:
I have run podman both as root and rootless with --log-level debug. Rootless reports the following (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage DEBU[0000] Using run root /run/user/1007/containers DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Same for root podman attempt: root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] overlay: test mount with multiple lowers succeeded DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] overlay: test mount indicated that metacopy is not being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Podman detected system restart - performing state refresh INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Best regards, On 10.01.2023 15:40, Giuseppe Scrivano wrote:
Daniel Walsh<dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there, But that's almost exactly what I did: I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
Hallo, overlay in rootless mode still gives problems with certain operations, namely when I am bootstrapping rocky linux container using buildah unshare --mount. When rpm packages are unpacked in such configuration, shadow-util package aborts with following error: Error unpacking rpm package shadow-utils-2:4.6-17.el8.x86_64 error: unpacking of archive failed on file /usr/bin/newgidmap;63cecdbe: cpio: cap_set_file failed - No data available error: shadow-utils-2:4.6-17.el8.x86_64: install failed If I try buildah mount command it responds that it is not supported with overlay system in rootless mode and can be used only as --mount option with buildah unshare command. I tried to set cap_setpcap and cap_setfcap capabilities to buildah executable but it did not help. When I use vfs storage everything works fine. Best regards, On 18.01.2023 22:51, Михаил Иванов wrote:
okay, it finally started to use overlays when I explicitly requested it in ~/.config/containers/storage.conf as follows: [storage] driver = "overlay" [storage.options] mount_program = "/usr/bin/fuse-overlayfs" Normally I do not have storage.conf file, neither as root nor as rootless. Best regards, On 11.01.2023 10:12, Михаил Иванов wrote:
Of course I did it after system reset in both cases. On 10.01.2023 23:28, Daniel Walsh wrote:
Please run this command AFTER running podman system reset
On 1/10/23 13:32, Михаил Иванов wrote:
I have run podman both as root and rootless with --log-level debug. Rootless reports the following (I show only beginning of the logs where, as I understand, the decision about vfs or overlay usage is made): island:container [master]> podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /home/tstusr/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver vfs DEBU[0000] Using graph root /home/tstusr/.local/share/containers/storage DEBU[0000] Using run root /run/user/1007/containers DEBU[0000] Using static dir /home/tstusr/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1007/libpod/tmp DEBU[0000] Using volume path /home/tstusr/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "vfs" DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/home/tstusr/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[vfs@/home/tstusr/.local/share/containers/storage+/run/user/1007/containers]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Same for root podman attempt: root@island:~# podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9 INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --name tstsys --detach --init --log-level debug docker.example.com/test/tstsys-devel-rl9) DEBU[0000] Merged system config "/usr/share/containers/containers.conf" DEBU[0000] Using conmon: "/usr/bin/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] overlay: test mount with multiple lowers succeeded DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] overlay: test mount indicated that metacopy is not being used DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2022-05-13 11:05:41.68704218 4 +0300 MSK [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] false false false map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Podman detected system restart - performing state refresh INFO[0000] Setting parallel job count to 25 DEBU[0000] Pulling image docker.example.com/test/tstsys-devel-rl9 (policy: missing) DEBU[0000] Looking up image "docker.example.com/test/tstsys-devel-rl9" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9:latest" ... DEBU[0000] Trying "docker.example.com/test/tstsys-devel-rl9" ... DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate docker.example.com/test/tstsys-devel-rl9:latest for docker.example.com/test/tstsys-devel-rl9 DEBU[0000] parsed reference into"[overlay@/var/lib/containers/storage+/run/containers/storage]docker.example.com/test/tstsys-devel-rl9:latest" Trying to pull docker.example.com/test/tstsys-devel-rl9:latest... Best regards, On 10.01.2023 15:40, Giuseppe Scrivano wrote:
Daniel Walsh<dwalsh@redhat.com> writes:
On 1/9/23 14:29, Михаил Иванов wrote:
I repeated everything again: podman system reset, then deleted everything in ~/.local/share/containers to be on the safe side and ran buildah to create an image. No overlay directories appeared under ~/.local/share/containers, only vfs* instead.
When I run same buildah command as root, overlay* directories do appear under /var/lib/containers and no vfs ones.
My debian is the latest one (sid) and it has the following fuse-related packages:
ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.12.0-1 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.12.0-1 amd64 Filesystem in Userspace (library) (3.x version)
podman is 4.3.1
For both root and rootless configs podman system info reports runc runtime (runc_1.1.4+ds1-1+b1_amd64 package).
Rgrds,
Giuseppe any ideas? not sure why overlay or fuse-overlayfs are not picked. It would be helpful to open an upstream issue and attach the output of `podman --log-level debug ...` in addition to all the other information that is requested.
On 03.01.2023 18:25, Daniel Walsh wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
> You could do a podman system reset and then remove all content > from the storage with > rm -rf ~/.local/share/containers > To make sure there is nothing hidden there, But that's almost exactly what I did: > I just purged the whole storage using podman system reset. > I verified that ~/.local/share/containers became empty > (only bolt database was still remaining using about 200Mb space) I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Best regards, On 29.12.2022 15:04, Daniel Walsh wrote:
On 12/27/22 13:19, Михаил Иванов wrote:
Hallo again, I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space) I have run 2 containers from docker.io: ibmcom/db2 and ibmcom/db2console. podman system df reports space usage by images 4Gb and by containers 6Mb. podman images command shows consistent values (two images, 2.83Gb + 1.21Gb) system df command shows that 32Gb is used on ~/.local/share/containers filesystem du shows that all this space is located under ~/.local/share/containers/storage/vfs/dir This directory contains 32 subdirs, 11 subdirs of 1.1Gb each, 6 subdirs of 2.6Gb each, rest subdirs take anywhere from 94Mb to 646Mb When I try to diff -rw for directories of same size, I see only reports for missing symlink files, but never real file differences. No real activity was performed with podman apart from running these two containers. I am running podman 4.3.1 on debian bookworm (kernel 6.0.8)
What is wrong?
I guess the first question I would have for you is why are you using VFS storage? And not overlay or fuse-overlay?
Could there be other containers or storage that is un-accounted for? Did you do any podman builds? Or Buildah?
``` $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether $ podman ps --all --external CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2e7070eacb0f docker.io/library/alpine:latest touch /dan/walsh 6 days ago Exited (0) 6 days ago nervous_noether 9478fed6d8db docker.io/library/alpine:latest buildah 14 seconds ago Storage alpine-working-container ```
You could do a podman system reset and then remove all content from the storage with
rm -rf ~/.local/share/containers
To make sure there is nothing hidden there,
Other then that I am not sure what could be showing the difference in storage.
Best regards, -- On 23.12.2022 19:43, Михаил Иванов wrote:
Hallo, I notice some disk space discrepancy when running rootless podman containers. I use dedicated fs for podman storage mountrd to ~/.local/share/containers. df and du show consistent used disk space: island:named [master]> df -h ~/.local/share/containers /dev/mapper/sys-containers 117G 84G 32G 73% ~/.local/share/containers
island:named [master]> sudo du -sh ~/.local/share/containers/storage/{vfs,volumes} 74G /home/ivans/.local/share/containers/storage/vfs 11G /home/ivans/.local/share/containers/storage/volumes island:named [master]> But space usage shown by podman system df is about 44% less than reported above:
island:named [master]> podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 32 5 39.49GB 25.96GB (66%) Containers 7 7 1.85GB 0B (0%) Local Volumes 2 2 10.83GB 0B (0%)
Volume space is practically same, it's vfs space (where as I understand images and containers are located) that differs. I also run buildah as same user, but buildah ls shows nothing. I have ran podman system prune, but it reclaimed 0 bytes. So is this extra space usage expected? Or is it sthing wrong with my storage? Thanks and regards,
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
_______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
On Tue, Jan 3, 2023 at 10:27 AM Daniel Walsh <dwalsh@redhat.com> wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
You could do a podman system reset and then remove all content from the storage with rm -rf ~/.local/share/containers To make sure there is nothing hidden there,
But that's almost exactly what I did:
I just purged the whole storage using podman system reset. I verified that ~/.local/share/containers became empty (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Dan, In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f... This file is installed verbatim to /usr/share/containers/storage.conf. Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else? Thanks, -rt
On 1/24/23 03:47, Reinhard Tartler wrote:
On Tue, Jan 3, 2023 at 10:27 AM Daniel Walsh <dwalsh@redhat.com> wrote:
On 12/30/22 08:35, Михаил Иванов wrote:
> You could do a podman system reset and then remove all content > from the storage with > rm -rf ~/.local/share/containers > To make sure there is nothing hidden there, But that's almost exactly what I did: > I just purged the whole storage using podman system reset. > I verified that ~/.local/share/containers became empty > (only bolt database was still remaining using about 200Mb space)
I'm using whatever storage was provided by default podman install (debian sid/bookworm) podman is 4.3.1 How can I reconfigure it to different type? I assumed this has to be done in storage.conf but this file is not present anywhere at all.
I have no idea why debian would be choosing VFS, unless this is an older version of debian and did not support rootless overaly. You could try installling fuse-overlayfs and doing another reset, then Podman info should show you using overlay with fuse-overlayfs.
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f...
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default? If not then it would fail over to VFS if fuse-overlayfs is not installed.
On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh <dwalsh@redhat.com> wrote:
On 1/24/23 03:47, Reinhard Tartler wrote:
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3 c4cded8fd7702af36a09f02b858f/storage.conf#L116-L118
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default?
If not then it would fail over to VFS if fuse-overlayfs is not installed.
I'm a bit confused about what you mean with that. In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default. What else is required for rootless overlay? Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience? -rt -- regards, Reinhard
Reinhard Tartler <siretart@gmail.com> writes:
On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh <dwalsh@redhat.com> wrote:
On 1/24/23 03:47, Reinhard Tartler wrote:
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f...
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default?
If not then it would fail over to VFS if fuse-overlayfs is not installed.
I'm a bit confused about what you mean with that.
In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default.
What else is required for rootless overlay?
Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience?
what issue do you see if you use native overlay? Podman prefers native overlay if it is available, since it is faster. If not, it tries fuse-overlays and if it is not available, it falls back to vfs. Could you try from a fresh storage though? If fuse-overlayfs was already used, then Podman will continue using it even if native overlay is available, since the storage metadata is slightly different. Thanks, Giuseppe
Is native overlay available in rootless mode? When I run podman as root there's no problem, overlayfs is picked up as default in debian. VFS is selected as default only in rootless mode. Rgrds, On 25.01.2023 14:03, Giuseppe Scrivano wrote:
Reinhard Tartler<siretart@gmail.com> writes:
On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh<dwalsh@redhat.com> wrote:
On 1/24/23 03:47, Reinhard Tartler wrote:
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f...
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default?
If not then it would fail over to VFS if fuse-overlayfs is not installed.
I'm a bit confused about what you mean with that.
In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default.
What else is required for rootless overlay?
Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience? what issue do you see if you use native overlay?
Podman prefers native overlay if it is available, since it is faster. If not, it tries fuse-overlays and if it is not available, it falls back to vfs.
Could you try from a fresh storage though? If fuse-overlayfs was already used, then Podman will continue using it even if native overlay is available, since the storage metadata is slightly different.
Thanks, Giuseppe _______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
Is there any config files in ~/.config/containers? podman system reset Should remove everything, and from then on Podman should use rootless overlay. On 1/25/23 09:52, Михаил Иванов wrote:
Is native overlay available in rootless mode? When I run podman as root there's no problem, overlayfs is picked up as default in debian. VFS is selected as default only in rootless mode. Rgrds, On 25.01.2023 14:03, Giuseppe Scrivano wrote:
Reinhard Tartler<siretart@gmail.com> writes:
On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh<dwalsh@redhat.com> wrote:
On 1/24/23 03:47, Reinhard Tartler wrote:
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f...
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default?
If not then it would fail over to VFS if fuse-overlayfs is not installed.
I'm a bit confused about what you mean with that.
In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default.
What else is required for rootless overlay?
Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience? what issue do you see if you use native overlay?
Podman prefers native overlay if it is available, since it is faster. If not, it tries fuse-overlays and if it is not available, it falls back to vfs.
Could you try from a fresh storage though? If fuse-overlayfs was already used, then Podman will continue using it even if native overlay is available, since the storage metadata is slightly different.
Thanks, Giuseppe _______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
I did this already several times. It does not. I also checked ~/.config/containers, there were only two items there: empty directory podman and registries.conf with following contents: ## [[registry]] ## prefix = "test" ## location = "docker.test.local" unqualified-search-registries = [ 'docker.test.local', 'docker.io' ] To be on the very safe side I repeated the test, I ran podman system reset and then cleaned up whatever was remaining in ~/.config/containers and ~/.local/share/containers manually. Then I just podman run alpine. vfs was used again: island:kit [master]> ls -la ~/.local/share/containers/storage/ итого 48 drwx------ 10 ivans users 4096 янв 25 20:41 ./ drwxr-xr-x 5 ivans users 4096 янв 25 20:41 ../ -rw-r--r-- 1 ivans users 8 янв 25 20:41 defaultNetworkBackend drwx------ 2 ivans users 4096 янв 25 20:41 libpod/ drwx------ 2 ivans users 4096 янв 25 20:41 mounts/ drwx------ 2 ivans users 4096 янв 25 20:41 networks/ -rw-r--r-- 1 ivans users 64 янв 25 20:41 storage.lock drwx------ 2 ivans users 4096 янв 25 20:41 tmp/ -rw-r--r-- 1 ivans users 0 янв 25 20:41 userns.lock drwx------ 3 ivans users 4096 янв 25 20:41 vfs/ drwx------ 3 ivans users 4096 янв 25 20:41 vfs-containers/ drwx------ 3 ivans users 4096 янв 25 20:41 vfs-images/ drwx------ 2 ivans users 4096 янв 25 20:41 vfs-layers/ My system has following fuse-related packages: island:kit [master]> dpkg -l | grep fuse ii fuse-overlayfs 1.9-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers ii fuse3 3.13.0-2 amd64 Filesystem in Userspace (3.x version) ii libfuse2:amd64 2.9.9-6 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.13.0-2 amd64 Filesystem in Userspace (library) (3.x version) Rgrds, On 25.01.2023 17:55, Daniel Walsh wrote:
Is there any config files in ~/.config/containers?
podman system reset
Should remove everything, and from then on Podman should use rootless overlay.
On 1/25/23 09:52, Михаил Иванов wrote:
Is native overlay available in rootless mode? When I run podman as root there's no problem, overlayfs is picked up as default in debian. VFS is selected as default only in rootless mode. Rgrds, On 25.01.2023 14:03, Giuseppe Scrivano wrote:
Reinhard Tartler<siretart@gmail.com> writes:
On Tue, Jan 24, 2023 at 2:08 PM Daniel Walsh<dwalsh@redhat.com> wrote:
On 1/24/23 03:47, Reinhard Tartler wrote:
Dan,
In Debian, I've chosen to just go with the upstream defaults: https://github.com/containers/storage/blob/8428fad6d0d3c4cded8fd7702af36a09f...
This file is installed verbatim to /usr/share/containers/storage.conf.
Is there a better choice? Does Fedora/Redhat provide a default storage.conf from somewhere else?
Thanks, -rt
That should be fine. Fedora goes with that default as well. Does debian support rootless overlay by default?
If not then it would fail over to VFS if fuse-overlayfs is not installed.
I'm a bit confused about what you mean with that.
In Debian releases that ship podman 4.x we have at least Linux kernel 6.0. The fuse-overlayfs package is installed by default, but users may opt to not install it by configuring apt to not install "Recommends" by default.
What else is required for rootless overlay?
Also, if I follow this conversation, then it appears that the default storage.conf requires modification in line 118 (to uncomment the mount_program option) in order to actually use fuse-overlayfs. I would have expected podman to use fuse-overlayfs if it happens to be installed, and fallback to direct mount if not. I read Michail's email thread that this appears to be not the case and he had to spend a lot of effort figuring out how to install an appropriate configuration file. Maybe I'm missing something, but I wonder what we can do to improve the user experience? what issue do you see if you use native overlay?
Podman prefers native overlay if it is available, since it is faster. If not, it tries fuse-overlays and if it is not available, it falls back to vfs.
Could you try from a fresh storage though? If fuse-overlayfs was already used, then Podman will continue using it even if native overlay is available, since the storage metadata is slightly different.
Thanks, Giuseppe _______________________________________________ Podman mailing list --podman@lists.podman.io To unsubscribe send an email topodman-leave@lists.podman.io
participants (4)
-
Daniel Walsh -
Giuseppe Scrivano -
Reinhard Tartler -
Михаил Иванов