I have never touched WSL, Haven't touched a windows machine in 15 years. :^)You lucky dog. ;) I'm running rootless, but I just did some tests as root. Interestingly, root seems to perform better than rootless when not in /mnt. But, I'm running into other issues. Strangely, running "buildah run $container npm install [anything]" hangs until the network times out. I can install packages with apk (this is an alpine container) via buildah run no problem. Also, the same npm install running rootless works (albeit slowly). For example, if I run: # temp=$(buildah from node:12-alpine) # buildah run $temp ping registry.npmjs.org ...works, no packet loss # buildah run $temp npm install ---global gulp-cli ...hangs until network times out The exact same commands run without issue when rootless. However if I run: # buildah run $temp apk --update --no-cache add python That runs quickly as root and hangs forever at: fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz when running rootless until it eventually completes. # time buildah copy $temp . /srv This directory has about 24,000 files (almost all in node_modules) and weighs about 194MB. When running rootless: real 0m18.702s user 0m7.263s sys 0m15.037s as root: real 0m5.782s user 0m4.632s sys 0m5.409s The root performance isn't too bad actually. For reference, when I run the same command in /mnt as root I get: real 10m21.947s <--- ouch user 0m26.626s sys 1m40.322s To sum up, I have no idea what's going on. I think my original red flag with copy taking forever, was the /mnt performance issue I linked in a previous email. If I keep everything in /home that issue is resolved. Working that way will be a major pain, but doable. I'm left with the network issues (apk add takes forever when rootless and npm install fails as root). Any ideas or debugging info I can provide at this point?
Try to run with --net=host which should run the commands without a network namespace.
See if the issue improves.
Also try podman build with a Dockerfile, to see if CNI
networking is better.
Added a few more egineers to get their ideas.
Daniel Walsh <dwalsh@redhat.com> wrote:Are you running as root or rootless? I have never touched WSL, Haven't touched a windows machine in 15 years. :^) On 6/5/20 00:16, Dominic wrote:Thanks for the input, Dan. As I understand the output of the buildah info command in my last email I am using overlay, or am I missing something? After more searching, I thought I was running into this WSL2 issue: https://github.com/microsoft/WSL/issues/4197 But, when I tried the build entirely from my home dir in Linux (not touching /mnt at all) the performance was the same. Any other ideas for things I can try? Daniel Walsh <dwalsh@redhat.com> wrote:Sounds like you are not using overlay driver, but maybe vfs? On 6/4/20 00:25, Dominic wrote:Thanks for the suggestions Scott and James. I really appreciate it. I'm testing both with Ubuntu 18.04 downloaded from Microsoft and with Fedora 32 setup by using wsl --import on a Fedora container rootfs. Both distributions are exhibiting the same behavior. I have followed the instructions on the sysadmin blog on both as well. It may also be wroth noting that running containers with podman is also "slow", but that's harder to quantify. Minutes to do a copy with buildah tells me something is seriously wrong with my setup. With podman, starting a container and running commands inside are just really sluggish. The problems might be related. Below are the buildah info outputs from both distributions. The Fedora 32 one was running overlay out of the box. For the Ubuntu one I had to build fuse-overlayfs using the instructions on the readme and configure buildah/podman to use it. Performance wasn't noticeably different from the default vfs driver. Ubuntu 18.04 { "host": { "CgroupVersion": "v1", "Distribution": { "distribution": "ubuntu", "version": "18.04" }, "MemTotal": 53851373568, "MenFree": 52583653376, "OCIRuntime": "runc", "SwapFree": 13958643712, "SwapTotal": 13958643712, "arch": "amd64", "cpus": 12, "hostname": "User-1", "kernel": "4.19.104-microsoft-standard", "os": "linux", "rootless": true, "uptime": "1h 18m 58.71s (Approximately 0.04 days)" }, "store": { "ContainerStore": { "number": 1 }, "GraphDriverName": "overlay", "GraphOptions": [ "overlay.mount_program=/usr/bin/fuse-overlayfs" ], "GraphRoot": "/home/user/.local/share/containers/storage", "GraphStatus": { "Backing Filesystem": "extfs", "Native Overlay Diff": "false", "Supports d_type": "true", "Using metacopy": "false" }, "ImageStore": { "number": 1 }, "RunRoot": "/var/tmp/1000/containers" } } Fedora 32 { "host": { "CgroupVersion": "v1", "Distribution": { "distribution": "fedora", "version": "32" }, "MemTotal": 53851373568, "MenFree": 52536049664, "OCIRuntime": "runc", "SwapFree": 13958643712, "SwapTotal": 13958643712, "arch": "amd64", "cpus": 12, "hostname": "User-1", "kernel": "4.19.104-microsoft-standard", "os": "linux", "rootless": true, "uptime": "1h 19m 33.42s (Approximately 0.04 days)" }, "store": { "ContainerStore": { "number": 2 }, "GraphDriverName": "overlay", "GraphOptions": [ "overlay.mount_program=/usr/bin/fuse-overlayfs" ], "GraphRoot": "/home/user/.local/share/containers/storage", "GraphStatus": { "Backing Filesystem": "extfs", "Native Overlay Diff": "false", "Supports d_type": "true", "Using metacopy": "false" }, "ImageStore": { "number": 7 }, "RunRoot": "/var/tmp/1000/containers" } } Let me know if there's anything else I can try to debug this. Scott McCarty <smccarty@redhat.com> wrote:James has a pretty darn good guess. Could you share the output of the "buildah info" command? Also, are you using an Ubuntu userspace on WSL? Note there can be things you need to do to tune the WSL kernel, like this: https://www.redhat.com/sysadmin/podman-windows-wsl2 Though, I wouldn't expect cgroups to affect buildah, as it's using chroots so I don't "think" the cgroups should matter. On Wed, Jun 3, 2020 at 12:31 PM James Cassell <fedoraproject@cyberpear.com> wrote:On Wed, Jun 3, 2020, at 2:24 AM, Dominic wrote:Hello, I've been using podman and buildah for a while on native Ubuntu and it's been great. With the release of WSL2 I'm trying to start building containers on Windows, but I'm running into some performance issues. Running something like: buildah copy $container . /srv in my project root takes maybe a second or two on native linux and minutes on my WSL2 system. Windows task manager doesn't show high resource utilization from any particular process during this time. Any ideas on how I can debug this?My guess is it's not using overlay capable fs, but is instead using VFS. V/r, James Cassell _______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io
_______________________________________________ Buildah mailing list -- buildah@lists.buildah.io To unsubscribe send an email to buildah-leave@lists.buildah.io