12:09 a.m.
I have never touched WSL, Haven't touched a windows machine in
15 years. :^)
You lucky dog. ;)
I'm running rootless, but I just did some tests as root.
Interestingly, root seems to perform better than rootless when
not in /mnt. But, I'm running into other issues. Strangely,
running "buildah run $container npm install [anything]" hangs
until the network times out. I can install packages with apk
(this is an alpine container) via buildah run no problem. Also,
the same npm install running rootless works (albeit slowly).
For example, if I run:
# temp=$(buildah from node:12-alpine)
# buildah run $temp ping registry.npmjs.org
...works, no packet loss
# buildah run $temp npm install ---global gulp-cli
...hangs until network times out
The exact same commands run without issue when rootless. However
if I run:
# buildah run $temp apk --update --no-cache add python
That runs quickly as root and hangs forever at: fetch
http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
when running rootless until it eventually completes.
# time buildah copy $temp . /srv
This directory has about 24,000 files (almost all in
node_modules) and weighs about 194MB. When running rootless:
real 0m18.702s
user 0m7.263s
sys 0m15.037s
as root:
real 0m5.782s
user 0m4.632s
sys 0m5.409s
The root performance isn't too bad actually. For reference, when
I run the same command in /mnt as root I get:
real 10m21.947s <--- ouch
user 0m26.626s
sys 1m40.322s
To sum up, I have no idea what's going on. I think my original
red flag with copy taking forever, was the /mnt performance issue
I linked in a previous email. If I keep everything in /home that
issue is resolved. Working that way will be a major pain, but
doable. I'm left with the network issues (apk add takes forever
when rootless and npm install fails as root).
Any ideas or debugging info I can provide at this point?
Daniel Walsh <dwalsh(a)redhat.com> wrote:
> Are you running as root or rootless?
>
I have never touched WSL, Haven't touched a windows machine in
15 years. :^)
>
> On 6/5/20 00:16, Dominic wrote:
> > Thanks for the input, Dan. As I understand the output of the
> > buildah info command in my last email I am using overlay, or am I
> > missing something?
> >
> > After more searching, I thought I was running into this WSL2
> > issue: https://github.com/microsoft/WSL/issues/4197
> >
> > But, when I tried the build entirely from my home dir in Linux
> > (not touching /mnt at all) the performance was the same.
> >
> > Any other ideas for things I can try?
> >
> >
> > Daniel Walsh <dwalsh(a)redhat.com> wrote:
> >> Sounds like you are not using overlay driver, but maybe vfs?
> >>
> >> On 6/4/20 00:25, Dominic wrote:
> >>> Thanks for the suggestions Scott and James. I really appreciate
> >>> it.
> >>>
> >>> I'm testing both with Ubuntu 18.04 downloaded from Microsoft and
> >>> with Fedora 32 setup by using wsl --import on a Fedora container
> >>> rootfs. Both distributions are exhibiting the same behavior. I
> >>> have followed the instructions on the sysadmin blog on both as
> >>> well.
> >>>
> >>> It may also be wroth noting that running containers with podman
> >>> is also "slow", but that's harder to quantify. Minutes to
do a
> >>> copy with buildah tells me something is seriously wrong with my
> >>> setup. With podman, starting a container and running commands
> >>> inside are just really sluggish. The problems might be related.
> >>>
> >>> Below are the buildah info outputs from both distributions. The
> >>> Fedora 32 one was running overlay out of the box. For the Ubuntu
> >>> one I had to build fuse-overlayfs using the instructions on the
> >>> readme and configure buildah/podman to use it. Performance wasn't
> >>> noticeably different from the default vfs driver.
> >>>
> >>> Ubuntu 18.04
> >>> {
> >>> "host": {
> >>> "CgroupVersion": "v1",
> >>> "Distribution": {
> >>> "distribution": "ubuntu",
> >>> "version": "18.04"
> >>> },
> >>> "MemTotal": 53851373568,
> >>> "MenFree": 52583653376,
> >>> "OCIRuntime": "runc",
> >>> "SwapFree": 13958643712,
> >>> "SwapTotal": 13958643712,
> >>> "arch": "amd64",
> >>> "cpus": 12,
> >>> "hostname": "User-1",
> >>> "kernel": "4.19.104-microsoft-standard",
> >>> "os": "linux",
> >>> "rootless": true,
> >>> "uptime": "1h 18m 58.71s (Approximately 0.04
days)"
> >>> },
> >>> "store": {
> >>> "ContainerStore": {
> >>> "number": 1
> >>> },
> >>> "GraphDriverName": "overlay",
> >>> "GraphOptions": [
> >>> "overlay.mount_program=/usr/bin/fuse-overlayfs"
> >>> ],
> >>> "GraphRoot":
"/home/user/.local/share/containers/storage",
> >>> "GraphStatus": {
> >>> "Backing Filesystem": "extfs",
> >>> "Native Overlay Diff": "false",
> >>> "Supports d_type": "true",
> >>> "Using metacopy": "false"
> >>> },
> >>> "ImageStore": {
> >>> "number": 1
> >>> },
> >>> "RunRoot": "/var/tmp/1000/containers"
> >>> }
> >>> }
> >>>
> >>> Fedora 32
> >>> {
> >>> "host": {
> >>> "CgroupVersion": "v1",
> >>> "Distribution": {
> >>> "distribution": "fedora",
> >>> "version": "32"
> >>> },
> >>> "MemTotal": 53851373568,
> >>> "MenFree": 52536049664,
> >>> "OCIRuntime": "runc",
> >>> "SwapFree": 13958643712,
> >>> "SwapTotal": 13958643712,
> >>> "arch": "amd64",
> >>> "cpus": 12,
> >>> "hostname": "User-1",
> >>> "kernel": "4.19.104-microsoft-standard",
> >>> "os": "linux",
> >>> "rootless": true,
> >>> "uptime": "1h 19m 33.42s (Approximately 0.04
days)"
> >>> },
> >>> "store": {
> >>> "ContainerStore": {
> >>> "number": 2
> >>> },
> >>> "GraphDriverName": "overlay",
> >>> "GraphOptions": [
> >>> "overlay.mount_program=/usr/bin/fuse-overlayfs"
> >>> ],
> >>> "GraphRoot":
"/home/user/.local/share/containers/storage",
> >>> "GraphStatus": {
> >>> "Backing Filesystem": "extfs",
> >>> "Native Overlay Diff": "false",
> >>> "Supports d_type": "true",
> >>> "Using metacopy": "false"
> >>> },
> >>> "ImageStore": {
> >>> "number": 7
> >>> },
> >>> "RunRoot": "/var/tmp/1000/containers"
> >>> }
> >>> }
> >>>
> >>> Let me know if there's anything else I can try to debug this.
> >>>
> >>>
> >>> Scott McCarty <smccarty(a)redhat.com> wrote:
> >>>> James has a pretty darn good guess. Could you share the output
> >>>> of the "buildah info" command? Also, are you using an
Ubuntu
> >>>> userspace on WSL? Note there can be things you need to do to
> >>>> tune the WSL kernel, like this:
> >>>>
> >>>> https://www.redhat.com/sysadmin/podman-windows-wsl2
> >>>>
> >>>> Though, I wouldn't expect cgroups to affect buildah, as
it's
> >>>> using chroots so I don't "think" the cgroups should
matter.
> >>>>
> >>>> On Wed, Jun 3, 2020 at 12:31 PM James Cassell
> >>>> <fedoraproject(a)cyberpear.com> wrote:
> >>>>
> >>>>> On Wed, Jun 3, 2020, at 2:24 AM, Dominic wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I've been using podman and buildah for a while on native
Ubuntu
> >>>>>> and it's been great. With the release of WSL2 I'm
trying to start
> >>>>>> building containers on Windows, but I'm running into
some
> >>>>>> performance issues. Running something like: buildah copy
> >>>>>> $container . /srv in my project root takes maybe a second or
two
> >>>>>> on native linux and minutes on my WSL2 system.
> >>>>>>
> >>>>>> Windows task manager doesn't show high resource
utilization from
> >>>>>> any particular process during this time. Any ideas on how I
can
> >>>>>> debug this?
> >>>>>>
> >>>>> My guess is it's not using overlay capable fs, but is
instead using VFS.
> >>>>>
> >>>>> V/r,
> >>>>> James Cassell
> >>>>> _______________________________________________
> >>>>> Buildah mailing list -- buildah(a)lists.buildah.io
> >>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
> >>>>>
> >>>> _______________________________________________
> >>>> Buildah mailing list -- buildah(a)lists.buildah.io
> >>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
> >>> _______________________________________________
> >>> Buildah mailing list -- buildah(a)lists.buildah.io
> >>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
> >>
> >> _______________________________________________
> >> Buildah mailing list -- buildah(a)lists.buildah.io
> >> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
> >
> > _______________________________________________
> > Buildah mailing list -- buildah(a)lists.buildah.io
> > To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>
>
> _______________________________________________
> Buildah mailing list -- buildah(a)lists.buildah.io
> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
--
Thank you,
Dominic