10:13 p.m.
Dear Podman list,
I am just starting to dip my toes into running containers in Podman 2.2.1 on RHEL 8.3
(Linux kernel 4.18) and would appreciate guidance on how to diagnose and solve an error I
get when running as non-root.
My goal is to do R language development in containers, and I used `podman pull` to get
these R images from the popular Rocker project (https://www.rocker-project.org/):
https://hub.docker.com/r/rocker/r-base
https://hub.docker.com/r/rocker/tidyverse
https://hub.docker.com/r/rocker/rstudio
The `r-base` image is just the R interpreter and it starts fine where I can just attach it
to a terminal to run R commands.
However, the other images (`tidyverse` and `rstudio`) are built on top of `r-base` and
when I run them as non-root (e.g. `podman run -ti rocker/tidyverse`) they give this error:
```
Error: OCI runtime error: container_linux.go:370: starting container process caused:
process_linux.go:459: container init caused: Running hook #0:: error running hook: exit
status 1, stdout: , stderr:
```
I have no idea how to interpret and understand this error message. The only clue I have is
that these two images run without errors if I append `sudo` to my `podman run` command. I
also checked the Dockerfiles (such as this one:
https://github.com/rocker-org/rocker-versioned/blob/master/tidyverse/3.6....) and
it's not clear what might be giving the error.
Oh, and the `rocker/rstudio` image allows you to specify ports i.e. `-p 8787:8787` but
even if I leave out this argument, I still get the error above when running `podman run`
on this image. So I don't think it's an issue with opening or mapping ports (and
Podman allows this as non-root, right?).
How should I begin to diagnose the root of this problem and start fixing it?
To be clear, my goal is to run the images listed above ideally as a non-root user. Thank
you!
9:06 a.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
Hi there,
Thanks for reaching out!
On Fri, Feb 19, 2021 at 10:14 PM boardbill_unpretended--- via Podman <
podman(a)lists.podman.io> wrote:
Dear Podman list,
I am just starting to dip my toes into running containers in Podman 2.2.1
on RHEL 8.3 (Linux kernel 4.18) and would appreciate guidance on how to
diagnose and solve an error I get when running as non-root.
My goal is to do R language development in containers, and I used `podman
pull` to get these R images from the popular Rocker project (
https://www.rocker-project.org/):
https://hub.docker.com/r/rocker/r-base
https://hub.docker.com/r/rocker/tidyverse
https://hub.docker.com/r/rocker/rstudio
The `r-base` image is just the R interpreter and it starts fine where I
can just attach it to a terminal to run R commands.
However, the other images (`tidyverse` and `rstudio`) are built on top of
`r-base` and when I run them as non-root (e.g. `podman run -ti
rocker/tidyverse`) they give this error:
```
Error: OCI runtime error: container_linux.go:370: starting container
process caused: process_linux.go:459: container init caused: Running hook
#0:: error running hook: exit status 1, stdout: , stderr:
```
This indicates that an OCI hook failed to run. Can you do an `$ ls
/usr/libexec/oci/hooks.d/` and see if there are any hooks? There might
also be config files in `/etc/containers/oci/hooks.d` or
`/usr/share/containers/oci/hooks.d` to control possibly installed hooks.
Do other images run? For instance, `podman run --rm fedora ls`?
I *suspect* that there's an OCI hook installed that requires root and may
run unconditionally.
Kind regards,
Valentin
I have no idea how to interpret and understand this error message. The only
clue I have is that these two images run without errors if I append
`sudo`
to my `podman run` command. I also checked the Dockerfiles (such as this
one:
https://github.com/rocker-org/rocker-versioned/blob/master/tidyverse/3.6....)
and it's not clear what might be giving the error.
Oh, and the `rocker/rstudio` image allows you to specify ports i.e. `-p
8787:8787` but even if I leave out this argument, I still get the error
above when running `podman run` on this image. So I don't think it's an
issue with opening or mapping ports (and Podman allows this as non-root,
right?).
How should I begin to diagnose the root of this problem and start fixing
it?
To be clear, my goal is to run the images listed above ideally as a
non-root user. Thank you!
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
10:37 p.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
Thank you for getting back to me! Here's what I found:
1. Non-root `podman run --rm fedora ls` runs fine and I can see the output of the `ls`
command. It looks normal.
This indicates that an OCI hook failed to run
I didn't even know there is a thing called a "hook". :)
2. Here's the output of the three places you suggested that might have hooks:
`ls /usr/libexec/oci/hooks.d/` - There is one file `oci-systemd-hook`.
`ls /etc/containers/oci/hooks.d` - Nothing.
`/usr/share/containers/oci/hooks.d` - One file called `oci-systemd-hook.json` with these
contents:
```
{
"cmd": [".*/init$" , ".*/systemd$" ],
"hook": "/usr/libexec/oci/hooks.d/oci-systemd-hook",
"stage": [ "prestart", "poststop" ]
}
```
I have zero idea what any of this means, what a hook is supposed to do, or how
`oci-systemd-hook` got there.
By the way, `oci-systemd-hook` looks like a binary file (???), but I tried to see its
contents via `cat` and there is a little bit of readable text in it. I've pasted the
output in this pastebin:
https://wtf.roflcopter.fr/paste/?1e2e32dbdccb06fa#GVpawbfVEvmJUEUzuLCu2YQ...
Does any of the above provide clues to what's happening? Is it "simply" a
matter of removing `oci-systemd-hook`? Or is that file critical to the functioning of
Podman?
Thanks!!
9 a.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
On Sat, Feb 20, 2021 at 10:38 PM boardbill_unpretended--- via Podman <
podman(a)lists.podman.io> wrote:
Thank you for getting back to me! Here's what I found:
1. Non-root `podman run --rm fedora ls` runs fine and I can see the output
of the `ls` command. It looks normal.
That's good. It's not a general issue with this specific user but maybe
specific to the hook or image.
> This indicates that an OCI hook failed to run
I didn't even know there is a thing called a "hook". :)
2. Here's the output of the three places you suggested that might have
hooks:
`ls /usr/libexec/oci/hooks.d/` - There is one file `oci-systemd-hook`.
`ls /etc/containers/oci/hooks.d` - Nothing.
`/usr/share/containers/oci/hooks.d` - One file called
`oci-systemd-hook.json` with these contents:
```
{
"cmd": [".*/init$" , ".*/systemd$" ],
"hook": "/usr/libexec/oci/hooks.d/oci-systemd-hook",
"stage": [ "prestart", "poststop" ]
}
```
I have zero idea what any of this means, what a hook is supposed to do, or
how `oci-systemd-hook` got there.
This hook *was* used to allow for running systemd inside a container.
Docker rejected to support that natively, so it was achieved by using an
OCI runtime hook. Such hooks are executed by the low-level container
runtimes such as runc and crun. In this specific case, the systemd hook
was setting up certain mounts and tmpfs's to let systemd run happily inside
the container.
However, the hook is obsolete in the Podman world since Podman supports
running systemd natively.
By the way, `oci-systemd-hook` looks like a binary file (???), but I
tried
to see its contents via `cat` and there is a little bit of readable text in
it. I've pasted the output in this pastebin:
https://wtf.roflcopter.fr/paste/?1e2e32dbdccb06fa#GVpawbfVEvmJUEUzuLCu2YQ...
Does any of the above provide clues to what's happening? Is it "simply" a
matter of removing `oci-systemd-hook`? Or is that file critical to the
functioning of Podman?
Thanks! The oci-systemd-hook is very likely causing the issue. I had a
look at the entrypoint of the image you're using and indeed it's using
"/init". Looking at the above JSON output you shared, we can see that the
hook is fired when either /init or /systemd are the cmd/entrypoint of the
container.
That being said, removing the hook should resolve the issue since Podman
does not require the root for running systemd inside a container.
Kind regards,
Valentin
Thanks!!
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
3:40 p.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
Good news! The issue disappeared after I removed the `oci-systemd-hook` and
`oci-systemd-hook.json` files.
Thank you Valentin!! :D
I've got more questions about Podman as I continue to learn about it which probably
belong in new threads. But there is one follow up to this thread:
The idea of hooks sound interesting. Is there good documentation for how they work for
Podman and how I can set up hooks? An initial Internet search did not give me a clear
result. For example, if I want a clean up script to run inside the container *as it is
stopping*, what should I do? Let's say I have that script, how do I pass it to Podman
when I start a container and tell Podman to "hook" it so that the script runs as
the container beginning to stop (when I run `podman stop [container ID]`)? And can I
conditionally set hooks to only run for certain containers?
Thanks again! Really appreciate your help, I'll probably start another thread or two
with other problems I've encountered.
4:35 p.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
On Mon, Feb 22, 2021 at 3:41 PM boardbill_unpretended--- via Podman <
podman(a)lists.podman.io> wrote:
Yes, you can do that. We employ a trick like that for the
oci-seccomp-bpf-hook [1], where we're passing down an *annotation* that
JSOn config of the hook is looking for. This way, the hook will only be
fired when the container has the specific annotation.
[1]
https://github.com/containers/podman/blob/master/pkg/hooks/docs/oci-hooks...
[2] https://github.com/containers/oci-seccomp-bpf-hook
[3]
https://github.com/containers/oci-seccomp-bpf-hook/blob/master/oci-seccom...
> Thanks again! Really appreciate your help, I'll probably start another
> thread or two with other problems I've encountered.
> _______________________________________________
> Podman mailing list -- podman(a)lists.podman.io
> To unsubscribe send an email to podman-leave(a)lists.podman.io
Good news! The issue disappeared after I removed the
`oci-systemd-hook`
and `oci-systemd-hook.json` files.
Thank you Valentin!! :D
Excellent, glad I could help.
I've got more questions about Podman as I continue to learn about
it which
probably belong in new threads. But there is one follow up to this thread:
The idea of hooks sound interesting. Is there good documentation for how
they work for Podman and how I can set up hooks?
Yes, there's a man page `man oci-hooks` [1]. OCI hooks per specification
will run unconditionally, so we introduced a kind of wrapper around them to
make them more configurable.
An initial Internet search did not give me a clear result. For
example, if
I want a clean up script to run inside the container *as it is stopping*,
what should I do? Let's say I have that script, how do I pass it to Podman
when I start a container and tell Podman to "hook" it so that the script
runs as the container beginning to stop (when I run `podman stop [container
ID]`)?
I do not know a way to do that by means of runtime hooks. There's a
"poststop" stage but you'd need a "prestart" stage. What you
could do
instead is to have a script running inside your container that installs a
signal handler for SIGTERM. This signal is sent when stopping a container
(or during kill before shooting with SIGKILL).
And can I conditionally set hooks to only run for certain containers?
11:19 p.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
Thanks for the information! I've had a look at `man oci-hooks` and its way over my
league with many terms I don't understand.
Regarding the clean up script I was asking about, I did some searching per your suggestion
to wait for the SIGTERM signal, and the closest tool I can find so far is the `trap`
command which seems to do something *if* a given signal is given.
My best guess right now is:
1. Create the clean up script, let's call it `my-cleaner.sh`. This script has at least
one line with the `trap` command waiting for the SIGTERM signal. (is there a way to tell
`trap` to wait for SIGTERM *or* SIGKILL???)
2. In the Dockerfile add a `COPY my-cleaner.sh /my-cleaner.sh` line so that the script is
put into the container during `podman build`.
3. As you know, the container images I'm working with already has `CMD /init` (BTW,
what does `/init` do anyway????), so I guess I'll have to start the script by changing
the `CMD` line this way:
```
CMD /my-cleaner.sh & \
&& /init
```
4. Cross fingers and hope that when I run `podman stop` on the container, the resulting
SIGTERM will be seen by the `trap` command in `my-cleaner.sh` and take the clean up steps.
(by the way, how can I check that the clean up actually happened? is there a canonical way
to document if the desired actions were taken?)
Does the above look right?
8:01 a.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
On Fri, Feb 26, 2021 at 11:21 PM boardbill_unpretended--- via Podman <
podman(a)lists.podman.io> wrote:
Thanks for the information! I've had a look at `man oci-hooks`
and its way
over my league with many terms I don't understand.
Regarding the clean up script I was asking about, I did some searching per
your suggestion to wait for the SIGTERM signal, and the closest tool I can
find so far is the `trap` command which seems to do something *if* a given
signal is given.
My best guess right now is:
1. Create the clean up script, let's call it `my-cleaner.sh`. This script
has at least one line with the `trap` command waiting for the SIGTERM
signal. (is there a way to tell `trap` to wait for SIGTERM *or* SIGKILL???)
SIGKILL is special since you (i.e., a user-mode process) can't really
handle this signal. When a process receives it, the kernel will go ahead
and kill/exit it.
2. In the Dockerfile add a `COPY my-cleaner.sh /my-cleaner.sh` line
so
that the script is put into the container during `podman build`.
3. As you know, the container images I'm working with already has `CMD
/init` (BTW, what does `/init` do anyway????), so I guess I'll have to
start the script by changing the `CMD` line this way:
That depends on the individual container image. In most cases, `/init`/ is
a symlink to `systemd`. When a container is started with either of the two
as the command/entrypoint, Podman will automatically do the setup to allow
for running systemd inside the container.
```
CMD /my-cleaner.sh & \
&& /init
```
4. Cross fingers and hope that when I run `podman stop` on the container,
the resulting SIGTERM will be seen by the `trap` command in `my-cleaner.sh`
and take the clean up steps. (by the way, how can I check that the clean up
actually happened? is there a canonical way to document if the desired
actions were taken?)
Does the above look right?
I suggest having a close look whether `/init` is needed in your container.
As mentioned above, Podman will setup systemd to run inside the container
but that behaviour would now change given the CMD is altered.
Kind regards,
Valentin
_____________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
8:37 a.m.
New subject: OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
CMD /my-cleaner.sh & \
&& /init
[...]
Does the above look right?
Nope, that's not how CMD works. It needs to be given a single command.
I don't know the specifics of /init, because I'm using tini instead.
But the purpose is probably the same: Have a minimal init process run
as pid 1 (process ID 1), so that signal propagation to child processes
is handled correctly. Default signal handling is different for pid 1
than for all other processes.[1]
The setup I use is:
- set ENTRYPOINT to tini, or some other minimal init system
- use CMD for what I actually want to run, usually a bash script
For example:
ENTRYPOINT ["/sbin/tini", "-g", "--"]
CMD ["/path/to/my/bash/script.sh"]
The bash script starts with this line:
#!/usr/bin/env bash
Alternatively, you could try
CMD ["/bin/bash", "/path/to/my/bash/script.sh"]
If I am mistaken and /init is not a minimal init process to be used
as entry point, but some regular command to be executed, try this:
CMD ["/bin/bash", "-c", "/my-cleaner.sh & exec /init"]
That is a single command, namely bash, which then interprets the
arguments as a command line and starts child processes as required.
Hope that helps,
Roland
[1]
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-pr...
1145
days inactive
1155
days old
8 comments
3 participants
participants (3)
-
boardbill_unpretended＠simplelogin.co -
Roland Weber -
Valentin Rothberg