On Fri, Jan 28, 2022 at 11:30 AM Nalin Dahyabhai <nalin@redhat.com> wrote:
On Thu, Jan 27, 2022 at 12:17:52PM -0500, Peter Portante wrote:
> We are struggling to understand why we can run rootless containers on RHEL
> 8.5.
>
> Why can't I do the following (as described at [1]) as a non-root user:
>
> [pportant@intlab-006 ~]$ podman run --rm --name=myubi -it
> registry.access.redhat.com/ubi8/ubi /bin/bash
> [pportant@intlab-006 ~]$ echo $?
> 0
>
> Shouldn't that start an interactive shell in the container?
>
> When I run as root I see:
>
> [root@intlab-006 ~]# podman run --rm --name=myubi -it
> registry.access.redhat.com/ubi8/ubi /bin/bash
> [root@intlab-006 ~]# echo $?
> 127
>
> While on another RHEL 8.5 host it works just fine:
>
> [pportant@intlabproxy-002 ~]$ podman run --rm --name=myubi -it
> registry.access.redhat.com/ubi8/ubi /bin/bash
> [root@a9ef24a2578b /]#
>
> Any help would be appreciated.

What differences do you see between the two situations when you pass a
--log-level=info, or --log-level=debug, to podman?

[root@intlab-006 ~]# podman --log-level=info run --rm --name=myubi -it registry.access.redhat.com/ubi8/ubi /bin/bash
INFO[0000] podman filtering at log level info
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
INFO[0000] Setting parallel job count to 97
INFO[0000] Got pod network &{Name:myubi Namespace:myubi ID:c198a57f8fb8eebb2c8f391341fbb8bf0c02b84be2ee5b8b648e675adf07fb72 NetNS:/run/netns/cni-e0647a42-5e73-d803-a59b-b6d7102a61d3 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Adding pod myubi_myubi to CNI network "podman" (type=bridge)
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-c198a57f8fb8eebb2c8f391341fbb8bf0c02b84be2ee5b8b648e675adf07fb72.scope
INFO[0000] Got Conmon PID as 527872
/bin/bash: error while loading shared libraries: libtinfo.so.6: cannot change memory protections
INFO[0000] Got pod network &{Name:myubi Namespace:myubi ID:c198a57f8fb8eebb2c8f391341fbb8bf0c02b84be2ee5b8b648e675adf07fb72 NetNS:/run/netns/cni-e0647a42-5e73-d803-a59b-b6d7102a61d3 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Deleting pod myubi_myubi from CNI network "podman" (type=bridge)
 

[root@intlab-006 ~]# podman --log-level=debug run --rm --name=myubi -it registry.access.redhat.com/ubi8/ubi /bin/bash
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --rm --name=myubi -it registry.access.redhat.com/ubi8/ubi /bin/bash)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 97
DEBU[0000] Pulling image registry.access.redhat.com/ubi8/ubi (policy: missing)
DEBU[0000] Looking up image "registry.access.redhat.com/ubi8/ubi" in local containers storage
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi" ...
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36)
DEBU[0000] Looking up image "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi:latest" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi:latest" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36)
DEBU[0000] Looking up image "registry.access.redhat.com/ubi8/ubi" in local containers storage
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi" ...
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36)
DEBU[0000] Inspecting image fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Looking up image "registry.access.redhat.com/ubi8/ubi" in local containers storage
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi" ...
DEBU[0000] Trying "registry.access.redhat.com/ubi8/ubi:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage
DEBU[0000] Found image "registry.access.redhat.com/ubi8/ubi" as "registry.access.redhat.com/ubi8/ubi:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36)
DEBU[0000] Inspecting image fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] Inspecting image fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36
DEBU[0000] using systemd mode: false
DEBU[0000] setting container name myubi
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 0 for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] exporting opaque data as blob "sha256:fca12da1dc30ed8e7d03afb84b287fc695673fff9c04bfcb2ff404b558670a36"
DEBU[0000] created container "278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b"
DEBU[0000] container "278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b" has work directory "/var/lib/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata"
DEBU[0000] container "278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b" has run directory "/run/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata"
DEBU[0000] Handling terminal attach
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/O3PSOUOEW3QSVICI3R4JKCLUZM:/var/lib/containers/storage/overlay/l/D3BH2BNFY3ROCUAPMECXAE73X6,upperdir=/var/lib/containers/storage/overlay/b98d9c77348a5c73f50158fddb496fec6e73dd4b5258df1137bddf62095606c1/diff,workdir=/var/lib/containers/storage/overlay/b98d9c77348a5c73f50158fddb496fec6e73dd4b5258df1137bddf62095606c1/work,volatile,context="system_u:object_r:container_file_t:s0:c767,c1001"
DEBU[0000] mounted container "278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b" at "/var/lib/containers/storage/overlay/b98d9c77348a5c73f50158fddb496fec6e73dd4b5258df1137bddf62095606c1/merged"
DEBU[0000] Created root filesystem for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b at /var/lib/containers/storage/overlay/b98d9c77348a5c73f50158fddb496fec6e73dd4b5258df1137bddf62095606c1/merged
DEBU[0000] Made network namespace at /run/netns/cni-61dfd753-babd-927f-78f8-6bafafbb78a5 for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
INFO[0000] Got pod network &{Name:myubi Namespace:myubi ID:278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b NetNS:/run/netns/cni-61dfd753-babd-927f-78f8-6bafafbb78a5 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Adding pod myubi_myubi to CNI network "podman" (type=bridge)
DEBU[0000] [0] CNI result: &{0.4.0 [{Name:cni-podman0 Mac:2a:5c:33:fc:7e:44 Sandbox:} {Name:veth9766c440 Mac:d2:62:4b:e5:c2:29 Sandbox:} {Name:eth0 Mac:2a:6e:79:27:dc:9c Sandbox:/run/netns/cni-61dfd753-babd-927f-78f8-6bafafbb78a5}] [{Version:4 Interface:0xc0001ff3a8 Address:{IP:10.88.0.9 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}] {[]  [] []}}
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/b98d9c77348a5c73f50158fddb496fec6e73dd4b5258df1137bddf62095606c1/merged"
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting CGroups for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b to machine.slice:libpod:278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b at /var/lib/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b -u 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata -p /run/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata/pidfile -n myubi --exit-dir /run/libpod/exits --full-attach -s -l k8s-file:/var/lib/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/containers/storage/overlay-containers/278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b.scope
DEBU[0000] Received: 528374
INFO[0000] Got Conmon PID as 528361
DEBU[0000] Created container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b in OCI runtime
DEBU[0000] Attaching to container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] Received a resize event: {Width:194 Height:151}
DEBU[0000] Starting container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b with command [/bin/bash]
DEBU[0000] Started container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
/bin/bash: error while loading shared libraries: libtinfo.so.6: cannot change memory protections
DEBU[0000] Enabling signal proxying
DEBU[0000] Checking if container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b should restart
DEBU[0000] Removing container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] Removing all exec sessions for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] Cleaning up container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] Tearing down network namespace at /run/netns/cni-61dfd753-babd-927f-78f8-6bafafbb78a5 for container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
INFO[0000] Got pod network &{Name:myubi Namespace:myubi ID:278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b NetNS:/run/netns/cni-61dfd753-babd-927f-78f8-6bafafbb78a5 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Deleting pod myubi_myubi from CNI network "podman" (type=bridge)
DEBU[0000] Successfully cleaned up container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b
DEBU[0000] unmounted container "278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b"
DEBU[0000] Container 278fce491f48c3ed0496881a5adf544719e9f713f57189074c9214eb494fc46b storage is already unmounted, skipping...
DEBU[0000] Called run.PersistentPostRunE(podman --log-level=debug run --rm --name=myubi -it registry.access.redhat.com/ubi8/ubi /bin/bash)
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true



Nalin