11:08 p.m.
Thanks for the suggestions. I really appreciate it.
I tried running with --net=host and got some interesting results.
When running as root, there was no effect, the npm install
command still hung until the network timed out and the apk add
still ran smoothly. However, when running rootless, the apk add
command ran normally (no long delay at fetch
http://dl-cdn.alpinelinux.org...), but then the npm install
command hung like it does when running as root.
I also tried using podman build with this Dockerfile:
FROM node:12-alpine
RUN apk --update --no-cache add python
RUN npm install --global gulp-cli
When running as root the behavior was the same as described
above: apk add ran fine, npm install hung until timeout both with
and without --net=host.
When running rootless, the behavior was a bit different than
using buildah commands. When using the Dockerfile the rootless
behavior was the same as with root: apk add ran ok and npm
install hung both with and without --net=host. Very strange.
So, the only scenario where I can get npm install to run inside
the container is rootless via buildah commands. That's also the
only scenario where apk add hangs at fetch.
As I side note, I also tested the non-alpine version of the
node:12 image and I saw the same failure to run npm install as
root via buildah.
Ok, I think that's all I have for now. Let me know if there is
any other debugging I can do.
Daniel Walsh <dwalsh(a)redhat.com> wrote:
On 6/6/20 01:09, Dominic wrote:
>> I have never touched WSL, Haven't touched a windows machine in
>> 15 years. :^)
> You lucky dog. ;)
>
> I'm running rootless, but I just did some tests as root.
> Interestingly, root seems to perform better than rootless when
> not in /mnt. But, I'm running into other issues. Strangely,
> running "buildah run $container npm install [anything]" hangs
> until the network times out. I can install packages with apk
> (this is an alpine container) via buildah run no problem. Also,
> the same npm install running rootless works (albeit slowly).
>
> For example, if I run:
>
> # temp=$(buildah from node:12-alpine)
> # buildah run $temp ping registry.npmjs.org
>
> ...works, no packet loss
>
> # buildah run $temp npm install ---global gulp-cli
>
> ...hangs until network times out
>
> The exact same commands run without issue when rootless. However
> if I run:
>
> # buildah run $temp apk --update --no-cache add python
>
> That runs quickly as root and hangs forever at: fetch
> http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
> when running rootless until it eventually completes.
>
> # time buildah copy $temp . /srv
>
> This directory has about 24,000 files (almost all in
> node_modules) and weighs about 194MB. When running rootless:
>
> real 0m18.702s
> user 0m7.263s
> sys 0m15.037s
>
> as root:
>
> real 0m5.782s
> user 0m4.632s
> sys 0m5.409s
>
> The root performance isn't too bad actually. For reference, when
> I run the same command in /mnt as root I get:
>
> real 10m21.947s <--- ouch
> user 0m26.626s
> sys 1m40.322s
>
> To sum up, I have no idea what's going on. I think my original
> red flag with copy taking forever, was the /mnt performance issue
> I linked in a previous email. If I keep everything in /home that
> issue is resolved. Working that way will be a major pain, but
> doable. I'm left with the network issues (apk add takes forever
> when rootless and npm install fails as root).
>
> Any ideas or debugging info I can provide at this point?
Try to run with --net=host which should run the commands
without a network namespace.
See if the issue improves.
Also try podman build with a Dockerfile, to see if CNI
networking is better.
Added a few more egineers to get their ideas.
>
> Daniel Walsh <dwalsh(a)redhat.com> wrote:
>> Are you running as root or rootless?
>>
>> I have never touched WSL, Haven't touched a windows machine in
>> 15 years. :^)
>>
>> On 6/5/20 00:16, Dominic wrote:
>>> Thanks for the input, Dan. As I understand the output of the
>>> buildah info command in my last email I am using overlay, or am I
>>> missing something?
>>>
>>> After more searching, I thought I was running into this WSL2
>>> issue: https://github.com/microsoft/WSL/issues/4197
>>>
>>> But, when I tried the build entirely from my home dir in Linux
>>> (not touching /mnt at all) the performance was the same.
>>>
>>> Any other ideas for things I can try?
>>>
>>>
>>> Daniel Walsh <dwalsh(a)redhat.com> wrote:
>>>> Sounds like you are not using overlay driver, but maybe vfs?
>>>>
>>>> On 6/4/20 00:25, Dominic wrote:
>>>>> Thanks for the suggestions Scott and James. I really appreciate
>>>>> it.
>>>>>
>>>>> I'm testing both with Ubuntu 18.04 downloaded from Microsoft
and
>>>>> with Fedora 32 setup by using wsl --import on a Fedora container
>>>>> rootfs. Both distributions are exhibiting the same behavior. I
>>>>> have followed the instructions on the sysadmin blog on both as
>>>>> well.
>>>>>
>>>>> It may also be wroth noting that running containers with podman
>>>>> is also "slow", but that's harder to quantify. Minutes
to do a
>>>>> copy with buildah tells me something is seriously wrong with my
>>>>> setup. With podman, starting a container and running commands
>>>>> inside are just really sluggish. The problems might be related.
>>>>>
>>>>> Below are the buildah info outputs from both distributions. The
>>>>> Fedora 32 one was running overlay out of the box. For the Ubuntu
>>>>> one I had to build fuse-overlayfs using the instructions on the
>>>>> readme and configure buildah/podman to use it. Performance
wasn't
>>>>> noticeably different from the default vfs driver.
>>>>>
>>>>> Ubuntu 18.04
>>>>> {
>>>>> "host": {
>>>>> "CgroupVersion": "v1",
>>>>> "Distribution": {
>>>>> "distribution": "ubuntu",
>>>>> "version": "18.04"
>>>>> },
>>>>> "MemTotal": 53851373568,
>>>>> "MenFree": 52583653376,
>>>>> "OCIRuntime": "runc",
>>>>> "SwapFree": 13958643712,
>>>>> "SwapTotal": 13958643712,
>>>>> "arch": "amd64",
>>>>> "cpus": 12,
>>>>> "hostname": "User-1",
>>>>> "kernel":
"4.19.104-microsoft-standard",
>>>>> "os": "linux",
>>>>> "rootless": true,
>>>>> "uptime": "1h 18m 58.71s (Approximately 0.04
days)"
>>>>> },
>>>>> "store": {
>>>>> "ContainerStore": {
>>>>> "number": 1
>>>>> },
>>>>> "GraphDriverName": "overlay",
>>>>> "GraphOptions": [
>>>>>
"overlay.mount_program=/usr/bin/fuse-overlayfs"
>>>>> ],
>>>>> "GraphRoot":
"/home/user/.local/share/containers/storage",
>>>>> "GraphStatus": {
>>>>> "Backing Filesystem": "extfs",
>>>>> "Native Overlay Diff": "false",
>>>>> "Supports d_type": "true",
>>>>> "Using metacopy": "false"
>>>>> },
>>>>> "ImageStore": {
>>>>> "number": 1
>>>>> },
>>>>> "RunRoot": "/var/tmp/1000/containers"
>>>>> }
>>>>> }
>>>>>
>>>>> Fedora 32
>>>>> {
>>>>> "host": {
>>>>> "CgroupVersion": "v1",
>>>>> "Distribution": {
>>>>> "distribution": "fedora",
>>>>> "version": "32"
>>>>> },
>>>>> "MemTotal": 53851373568,
>>>>> "MenFree": 52536049664,
>>>>> "OCIRuntime": "runc",
>>>>> "SwapFree": 13958643712,
>>>>> "SwapTotal": 13958643712,
>>>>> "arch": "amd64",
>>>>> "cpus": 12,
>>>>> "hostname": "User-1",
>>>>> "kernel":
"4.19.104-microsoft-standard",
>>>>> "os": "linux",
>>>>> "rootless": true,
>>>>> "uptime": "1h 19m 33.42s (Approximately 0.04
days)"
>>>>> },
>>>>> "store": {
>>>>> "ContainerStore": {
>>>>> "number": 2
>>>>> },
>>>>> "GraphDriverName": "overlay",
>>>>> "GraphOptions": [
>>>>>
"overlay.mount_program=/usr/bin/fuse-overlayfs"
>>>>> ],
>>>>> "GraphRoot":
"/home/user/.local/share/containers/storage",
>>>>> "GraphStatus": {
>>>>> "Backing Filesystem": "extfs",
>>>>> "Native Overlay Diff": "false",
>>>>> "Supports d_type": "true",
>>>>> "Using metacopy": "false"
>>>>> },
>>>>> "ImageStore": {
>>>>> "number": 7
>>>>> },
>>>>> "RunRoot": "/var/tmp/1000/containers"
>>>>> }
>>>>> }
>>>>>
>>>>> Let me know if there's anything else I can try to debug this.
>>>>>
>>>>>
>>>>> Scott McCarty <smccarty(a)redhat.com> wrote:
>>>>>> James has a pretty darn good guess. Could you share the output
>>>>>> of the "buildah info" command? Also, are you using an
Ubuntu
>>>>>> userspace on WSL? Note there can be things you need to do to
>>>>>> tune the WSL kernel, like this:
>>>>>>
>>>>>> https://www.redhat.com/sysadmin/podman-windows-wsl2
>>>>>>
>>>>>> Though, I wouldn't expect cgroups to affect buildah, as
it's
>>>>>> using chroots so I don't "think" the cgroups
should matter.
>>>>>>
>>>>>> On Wed, Jun 3, 2020 at 12:31 PM James Cassell
>>>>>> <fedoraproject(a)cyberpear.com> wrote:
>>>>>>
>>>>>>> On Wed, Jun 3, 2020, at 2:24 AM, Dominic wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I've been using podman and buildah for a while on
native Ubuntu
>>>>>>>> and it's been great. With the release of WSL2
I'm trying to start
>>>>>>>> building containers on Windows, but I'm running into
some
>>>>>>>> performance issues. Running something like: buildah
copy
>>>>>>>> $container . /srv in my project root takes maybe a
second or two
>>>>>>>> on native linux and minutes on my WSL2 system.
>>>>>>>>
>>>>>>>> Windows task manager doesn't show high resource
utilization from
>>>>>>>> any particular process during this time. Any ideas on
how I can
>>>>>>>> debug this?
>>>>>>>>
>>>>>>> My guess is it's not using overlay capable fs, but is
instead using VFS.
>>>>>>>
>>>>>>> V/r,
>>>>>>> James Cassell
>>>>>>> _______________________________________________
>>>>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>>>>> To unsubscribe send an email to
buildah-leave(a)lists.buildah.io
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>>>> _______________________________________________
>>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>>> _______________________________________________
>>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>> _______________________________________________
>>> Buildah mailing list -- buildah(a)lists.buildah.io
>>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>>
>> _______________________________________________
>> Buildah mailing list -- buildah(a)lists.buildah.io
>> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
>
> _______________________________________________
> Buildah mailing list -- buildah(a)lists.buildah.io
> To unsubscribe send an email to buildah-leave(a)lists.buildah.io
_______________________________________________
Buildah mailing list -- buildah(a)lists.buildah.io
To unsubscribe send an email to buildah-leave(a)lists.buildah.io
--
Thank you,
Dominic