11:23 a.m.
On 6/6/20 01:09, Dominic wrote:
> I have never touched WSL, Haven't touched a windows machine
in
> 15 years. :^)
You lucky dog. ;)
I'm running rootless, but I just did some tests as root.
Interestingly, root seems to perform better than rootless when
not in /mnt. But, I'm running into other issues. Strangely,
running "buildah run $container npm install [anything]" hangs
until the network times out. I can install packages with apk
(this is an alpine container) via buildah run no problem. Also,
the same npm install running rootless works (albeit slowly).
For example, if I run:
# temp=$(buildah from node:12-alpine)
# buildah run $temp ping registry.npmjs.org
...works, no packet loss
# buildah run $temp npm install ---global gulp-cli
...hangs until network times out
The exact same commands run without issue when rootless. However
if I run:
# buildah run $temp apk --update --no-cache add python
That runs quickly as root and hangs forever at: fetch
http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
when running rootless until it eventually completes.
# time buildah copy $temp . /srv
This directory has about 24,000 files (almost all in
node_modules) and weighs about 194MB. When running rootless:
real 0m18.702s
user 0m7.263s
sys 0m15.037s
as root:
real 0m5.782s
user 0m4.632s
sys 0m5.409s
The root performance isn't too bad actually. For reference, when
I run the same command in /mnt as root I get:
real 10m21.947s <--- ouch
user 0m26.626s
sys 1m40.322s
To sum up, I have no idea what's going on. I think my original
red flag with copy taking forever, was the /mnt performance issue
I linked in a previous email. If I keep everything in /home that
issue is resolved. Working that way will be a major pain, but
doable. I'm left with the network issues (apk add takes forever
when rootless and npm install fails as root).
Any ideas or debugging info I can provide at this point?
Try to run with --net=host which should run the commands without a
network namespace.
See if the issue improves.
Also try podman build with a Dockerfile, to see if CNI networking is
better.
Added a few more egineers to get their ideas.
Daniel Walsh <dwalsh(a)redhat.com> wrote:
> Are you running as root or rootless?
>
> I have never touched WSL, Haven't touched a windows machine in
> 15 years. :^)
>
> On 6/5/20 00:16, Dominic wrote:
>> Thanks for the input, Dan. As I understand the output of the
>> buildah info command in my last email I am using overlay, or am I
>> missing something?
>>
>> After more searching, I thought I was running into this WSL2
>> issue: https://github.com/microsoft/WSL/issues/4197
>>
>> But, when I tried the build entirely from my home dir in Linux
>> (not touching /mnt at all) the performance was the same.
>>
>> Any other ideas for things I can try?
>>
>>
>> Daniel Walsh <dwalsh(a)redhat.com> wrote:
>>> Sounds like you are not using overlay driver, but maybe vfs?
>>>
>>> On 6/4/20 00:25, Dominic wrote:
>>>> Thanks for the suggestions Scott and James. I really appreciate
>>>> it.
>>>>
>>>> I'm testing both with Ubuntu 18.04 downloaded from Microsoft and
>>>> with Fedora 32 setup by using wsl --import on a Fedora container
>>>> rootfs. Both distributions are exhibiting the same behavior. I
>>>> have followed the instructions on the sysadmin blog on both as
>>>> well.
>>>>
>>>> It may also be wroth noting that running containers with podman
>>>> is also "slow", but that's harder to quantify. Minutes to
do a
>>>> copy with buildah tells me something is seriously wrong with my
>>>> setup. With podman, starting a container and running commands
>>>> inside are just really sluggish. The problems might be related.
>>>>
>>>> Below are the buildah info outputs from both distributions. The
>>>> Fedora 32 one was running overlay out of the box. For the Ubuntu
>>>> one I had to build fuse-overlayfs using the instructions on the
>>>> readme and configure buildah/podman to use it. Performance wasn't
>>>> noticeably different from the default vfs driver.
>>>>
>>>> Ubuntu 18.04
>>>> {
>>>> "host": {
>>>> "CgroupVersion": "v1",
>>>> "Distribution": {
>>>> "distribution": "ubuntu",
>>>> "version": "18.04"
>>>> },
>>>> "MemTotal": 53851373568,
>>>> "MenFree": 52583653376,
>>>> "OCIRuntime": "runc",
>>>> "SwapFree": 13958643712,
>>>> "SwapTotal": 13958643712,
>>>> "arch": "amd64",
>>>> "cpus": 12,
>>>> "hostname": "User-1",
>>>> "kernel": "4.19.104-microsoft-standard",
>>>> "os": "linux",
>>>> "rootless": true,
>>>> "uptime": "1h 18m 58.71s (Approximately 0.04
days)"
>>>> },
>>>> "store": {
>>>> "ContainerStore": {
>>>> "number": 1
>>>> },
>>>> "GraphDriverName": "overlay",
>>>> "GraphOptions": [
>>>> "overlay.mount_program=/usr/bin/fuse-overlayfs"
>>>> ],
>>>> "GraphRoot":
"/home/user/.local/share/containers/storage",
>>>> "GraphStatus": {
>>>> "Backing Filesystem": "extfs",
>>>> "Native Overlay Diff": "false",
>>>> "Supports d_type": "true",
>>>> "Using metacopy": "false"
>>>> },
>>>> "ImageStore": {
>>>> "number": 1
>>>> },
>>>> "RunRoot": "/var/tmp/1000/containers"
>>>> }
>>>> }
>>>>
>>>> Fedora 32
>>>> {
>>>> "host": {
>>>> "CgroupVersion": "v1",
>>>> "Distribution": {
>>>> "distribution": "fedora",
>>>> "version": "32"
>>>> },
>>>> "MemTotal": 53851373568,
>>>> "MenFree": 52536049664,
>>>> "OCIRuntime": "runc",
>>>> "SwapFree": 13958643712,
>>>> "SwapTotal": 13958643712,
>>>> "arch": "amd64",
>>>> "cpus": 12,
>>>> "hostname": "User-1",
>>>> "kernel": "4.19.104-microsoft-standard",
>>>> "os": "linux",
>>>> "rootless": true,
>>>> "uptime": "1h 19m 33.42s (Approximately 0.04
days)"
>>>> },
>>>> "store": {
>>>> "ContainerStore": {
>>>> "number": 2
>>>> },
>>>> "GraphDriverName": "overlay",
>>>> "GraphOptions": [
>>>> "overlay.mount_program=/usr/bin/fuse-overlayfs"
>>>> ],
>>>> "GraphRoot":
"/home/user/.local/share/containers/storage",
>>>> "GraphStatus": {
>>>> "Backing Filesystem": "extfs",
>>>> "Native Overlay Diff": "false",
>>>> "Supports d_type": "true",
>>>> "Using metacopy": "false"
>>>> },
>>>> "ImageStore": {
>>>> "number": 7
>>>> },
>>>> "RunRoot": "/var/tmp/1000/containers"
>>>> }
>>>> }
>>>>
>>>> Let me know if there's anything else I can try to debug this.
>>>>
>>>>
>>>> Scott McCarty <smccarty(a)redhat.com> wrote:
>>>>> James has a pretty darn good guess. Could you share the output
>>>>> of the "buildah info" command? Also, are you using an
Ubuntu
>>>>> userspace on WSL? Note there can be things you need to do to
>>>>> tune the WSL kernel, like this:
>>>>>
>>>>> https://www.redhat.com/sysadmin/podman-windows-wsl2
>>>>>
>>>>> Though, I wouldn't expect cgroups to affect buildah, as it's
>>>>> using chroots so I don't "think" the cgroups should
matter.
>>>>>
>>>>> On Wed, Jun 3, 2020 at 12:31 PM James Cassell
>>>>> <fedoraproject(a)cyberpear.com> wrote:
>>>>>
>>>>>> On Wed, Jun 3, 2020, at 2:24 AM, Dominic wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I've been using podman and buildah for a while on native
Ubuntu
>>>>>>> and it's been great. With the release of WSL2 I'm
trying to start
>>>>>>> building containers on Windows, but I'm running into
some
>>>>>>> performance issues. Running something like: buildah copy
>>>>>>> $container . /srv in my project root takes maybe a second or
two
>>>>>>> on native linux and minutes on my WSL2 system.
>>>>>>>
>>>>>>> Windows task manager doesn't show high resource
utilization from
>>>>>>> any particular process during this time. Any ideas on how I
can
>>>>>>> debug this?
>>>>>>>
>>>>>> My guess is it's not using overlay capable fs, but is instead
using VFS.
>>>>>>
>>>>>> V/r,
>>>>>> James Cassell
>>>>>> _______________________________________________
>>>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>>>>>
>>>>> _______________________________________________
>>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>>> _______________________________________________
>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>> _______________________________________________
>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>> _______________________________________________
>> Buildah mailing list -- buildah(a)lists.buildah.io
>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>
> _______________________________________________
> Buildah mailing list -- buildah(a)lists.buildah.io
> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
_______________________________________________
Buildah mailing list -- buildah(a)lists.buildah.io
To unsubscribe send an email to buildah-leave(a)lists.buildah.io