OCI runtime error when starting hub.docker.com/r/rocker images as non-root - how to troubleshoot?
Dear Podman list, I am just starting to dip my toes into running containers in Podman 2.2.1 on RHEL 8.3 (Linux kernel 4.18) and would appreciate guidance on how to diagnose and solve an error I get when running as non-root. My goal is to do R language development in containers, and I used `podman pull` to get these R images from the popular Rocker project (https://www.rocker-project.org/): https://hub.docker.com/r/rocker/r-base https://hub.docker.com/r/rocker/tidyverse https://hub.docker.com/r/rocker/rstudio The `r-base` image is just the R interpreter and it starts fine where I can just attach it to a terminal to run R commands. However, the other images (`tidyverse` and `rstudio`) are built on top of `r-base` and when I run them as non-root (e.g. `podman run -ti rocker/tidyverse`) they give this error: ``` Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: ``` I have no idea how to interpret and understand this error message. The only clue I have is that these two images run without errors if I append `sudo` to my `podman run` command. I also checked the Dockerfiles (such as this one: https://github.com/rocker-org/rocker-versioned/blob/master/tidyverse/3.6.3.D...) and it's not clear what might be giving the error. Oh, and the `rocker/rstudio` image allows you to specify ports i.e. `-p 8787:8787` but even if I leave out this argument, I still get the error above when running `podman run` on this image. So I don't think it's an issue with opening or mapping ports (and Podman allows this as non-root, right?). How should I begin to diagnose the root of this problem and start fixing it? To be clear, my goal is to run the images listed above ideally as a non-root user. Thank you!
Hi there, Thanks for reaching out! On Fri, Feb 19, 2021 at 10:14 PM boardbill_unpretended--- via Podman < podman@lists.podman.io> wrote:
Dear Podman list,
I am just starting to dip my toes into running containers in Podman 2.2.1 on RHEL 8.3 (Linux kernel 4.18) and would appreciate guidance on how to diagnose and solve an error I get when running as non-root.
My goal is to do R language development in containers, and I used `podman pull` to get these R images from the popular Rocker project ( https://www.rocker-project.org/):
https://hub.docker.com/r/rocker/r-base https://hub.docker.com/r/rocker/tidyverse https://hub.docker.com/r/rocker/rstudio
The `r-base` image is just the R interpreter and it starts fine where I can just attach it to a terminal to run R commands.
However, the other images (`tidyverse` and `rstudio`) are built on top of `r-base` and when I run them as non-root (e.g. `podman run -ti rocker/tidyverse`) they give this error:
``` Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: ```
This indicates that an OCI hook failed to run. Can you do an `$ ls /usr/libexec/oci/hooks.d/` and see if there are any hooks? There might also be config files in `/etc/containers/oci/hooks.d` or `/usr/share/containers/oci/hooks.d` to control possibly installed hooks. Do other images run? For instance, `podman run --rm fedora ls`? I *suspect* that there's an OCI hook installed that requires root and may run unconditionally. Kind regards, Valentin I have no idea how to interpret and understand this error message. The only
clue I have is that these two images run without errors if I append `sudo` to my `podman run` command. I also checked the Dockerfiles (such as this one: https://github.com/rocker-org/rocker-versioned/blob/master/tidyverse/3.6.3.D...) and it's not clear what might be giving the error.
Oh, and the `rocker/rstudio` image allows you to specify ports i.e. `-p 8787:8787` but even if I leave out this argument, I still get the error above when running `podman run` on this image. So I don't think it's an issue with opening or mapping ports (and Podman allows this as non-root, right?).
How should I begin to diagnose the root of this problem and start fixing it?
To be clear, my goal is to run the images listed above ideally as a non-root user. Thank you! _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Thank you for getting back to me! Here's what I found: 1. Non-root `podman run --rm fedora ls` runs fine and I can see the output of the `ls` command. It looks normal.
This indicates that an OCI hook failed to run
I didn't even know there is a thing called a "hook". :) 2. Here's the output of the three places you suggested that might have hooks: `ls /usr/libexec/oci/hooks.d/` - There is one file `oci-systemd-hook`. `ls /etc/containers/oci/hooks.d` - Nothing. `/usr/share/containers/oci/hooks.d` - One file called `oci-systemd-hook.json` with these contents: ``` { "cmd": [".*/init$" , ".*/systemd$" ], "hook": "/usr/libexec/oci/hooks.d/oci-systemd-hook", "stage": [ "prestart", "poststop" ] } ``` I have zero idea what any of this means, what a hook is supposed to do, or how `oci-systemd-hook` got there. By the way, `oci-systemd-hook` looks like a binary file (???), but I tried to see its contents via `cat` and there is a little bit of readable text in it. I've pasted the output in this pastebin: https://wtf.roflcopter.fr/paste/?1e2e32dbdccb06fa#GVpawbfVEvmJUEUzuLCu2YQGaq... Does any of the above provide clues to what's happening? Is it "simply" a matter of removing `oci-systemd-hook`? Or is that file critical to the functioning of Podman? Thanks!!
On Sat, Feb 20, 2021 at 10:38 PM boardbill_unpretended--- via Podman < podman@lists.podman.io> wrote:
Thank you for getting back to me! Here's what I found:
1. Non-root `podman run --rm fedora ls` runs fine and I can see the output of the `ls` command. It looks normal.
That's good. It's not a general issue with this specific user but maybe specific to the hook or image.
This indicates that an OCI hook failed to run
I didn't even know there is a thing called a "hook". :)
2. Here's the output of the three places you suggested that might have hooks:
`ls /usr/libexec/oci/hooks.d/` - There is one file `oci-systemd-hook`. `ls /etc/containers/oci/hooks.d` - Nothing. `/usr/share/containers/oci/hooks.d` - One file called `oci-systemd-hook.json` with these contents:
``` { "cmd": [".*/init$" , ".*/systemd$" ], "hook": "/usr/libexec/oci/hooks.d/oci-systemd-hook", "stage": [ "prestart", "poststop" ] } ```
I have zero idea what any of this means, what a hook is supposed to do, or how `oci-systemd-hook` got there.
This hook *was* used to allow for running systemd inside a container. Docker rejected to support that natively, so it was achieved by using an OCI runtime hook. Such hooks are executed by the low-level container runtimes such as runc and crun. In this specific case, the systemd hook was setting up certain mounts and tmpfs's to let systemd run happily inside the container. However, the hook is obsolete in the Podman world since Podman supports running systemd natively.
By the way, `oci-systemd-hook` looks like a binary file (???), but I tried to see its contents via `cat` and there is a little bit of readable text in it. I've pasted the output in this pastebin:
https://wtf.roflcopter.fr/paste/?1e2e32dbdccb06fa#GVpawbfVEvmJUEUzuLCu2YQGaq...
Does any of the above provide clues to what's happening? Is it "simply" a matter of removing `oci-systemd-hook`? Or is that file critical to the functioning of Podman?
Thanks! The oci-systemd-hook is very likely causing the issue. I had a look at the entrypoint of the image you're using and indeed it's using "/init". Looking at the above JSON output you shared, we can see that the hook is fired when either /init or /systemd are the cmd/entrypoint of the container. That being said, removing the hook should resolve the issue since Podman does not require the root for running systemd inside a container. Kind regards, Valentin
Thanks!! _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Good news! The issue disappeared after I removed the `oci-systemd-hook` and `oci-systemd-hook.json` files. Thank you Valentin!! :D I've got more questions about Podman as I continue to learn about it which probably belong in new threads. But there is one follow up to this thread: The idea of hooks sound interesting. Is there good documentation for how they work for Podman and how I can set up hooks? An initial Internet search did not give me a clear result. For example, if I want a clean up script to run inside the container *as it is stopping*, what should I do? Let's say I have that script, how do I pass it to Podman when I start a container and tell Podman to "hook" it so that the script runs as the container beginning to stop (when I run `podman stop [container ID]`)? And can I conditionally set hooks to only run for certain containers? Thanks again! Really appreciate your help, I'll probably start another thread or two with other problems I've encountered.
On Mon, Feb 22, 2021 at 3:41 PM boardbill_unpretended--- via Podman < podman@lists.podman.io> wrote:
Good news! The issue disappeared after I removed the `oci-systemd-hook` and `oci-systemd-hook.json` files.
Thank you Valentin!! :D
Excellent, glad I could help.
I've got more questions about Podman as I continue to learn about it which probably belong in new threads. But there is one follow up to this thread:
The idea of hooks sound interesting. Is there good documentation for how they work for Podman and how I can set up hooks?
Yes, there's a man page `man oci-hooks` [1]. OCI hooks per specification will run unconditionally, so we introduced a kind of wrapper around them to make them more configurable.
An initial Internet search did not give me a clear result. For example, if I want a clean up script to run inside the container *as it is stopping*, what should I do? Let's say I have that script, how do I pass it to Podman when I start a container and tell Podman to "hook" it so that the script runs as the container beginning to stop (when I run `podman stop [container ID]`)?
I do not know a way to do that by means of runtime hooks. There's a "poststop" stage but you'd need a "prestart" stage. What you could do instead is to have a script running inside your container that installs a signal handler for SIGTERM. This signal is sent when stopping a container (or during kill before shooting with SIGKILL). And can I conditionally set hooks to only run for certain containers?
Yes, you can do that. We employ a trick like that for the oci-seccomp-bpf-hook [1], where we're passing down an *annotation* that JSOn config of the hook is looking for. This way, the hook will only be fired when the container has the specific annotation. [1] https://github.com/containers/podman/blob/master/pkg/hooks/docs/oci-hooks.5.... [2] https://github.com/containers/oci-seccomp-bpf-hook [3] https://github.com/containers/oci-seccomp-bpf-hook/blob/master/oci-seccomp-b...
Thanks again! Really appreciate your help, I'll probably start another thread or two with other problems I've encountered. _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
Thanks for the information! I've had a look at `man oci-hooks` and its way over my league with many terms I don't understand. Regarding the clean up script I was asking about, I did some searching per your suggestion to wait for the SIGTERM signal, and the closest tool I can find so far is the `trap` command which seems to do something *if* a given signal is given. My best guess right now is: 1. Create the clean up script, let's call it `my-cleaner.sh`. This script has at least one line with the `trap` command waiting for the SIGTERM signal. (is there a way to tell `trap` to wait for SIGTERM *or* SIGKILL???) 2. In the Dockerfile add a `COPY my-cleaner.sh /my-cleaner.sh` line so that the script is put into the container during `podman build`. 3. As you know, the container images I'm working with already has `CMD /init` (BTW, what does `/init` do anyway????), so I guess I'll have to start the script by changing the `CMD` line this way: ``` CMD /my-cleaner.sh & \ && /init ``` 4. Cross fingers and hope that when I run `podman stop` on the container, the resulting SIGTERM will be seen by the `trap` command in `my-cleaner.sh` and take the clean up steps. (by the way, how can I check that the clean up actually happened? is there a canonical way to document if the desired actions were taken?) Does the above look right?
On Fri, Feb 26, 2021 at 11:21 PM boardbill_unpretended--- via Podman < podman@lists.podman.io> wrote:
Thanks for the information! I've had a look at `man oci-hooks` and its way over my league with many terms I don't understand.
Regarding the clean up script I was asking about, I did some searching per your suggestion to wait for the SIGTERM signal, and the closest tool I can find so far is the `trap` command which seems to do something *if* a given signal is given.
My best guess right now is:
1. Create the clean up script, let's call it `my-cleaner.sh`. This script has at least one line with the `trap` command waiting for the SIGTERM signal. (is there a way to tell `trap` to wait for SIGTERM *or* SIGKILL???)
SIGKILL is special since you (i.e., a user-mode process) can't really handle this signal. When a process receives it, the kernel will go ahead and kill/exit it.
2. In the Dockerfile add a `COPY my-cleaner.sh /my-cleaner.sh` line so that the script is put into the container during `podman build`. 3. As you know, the container images I'm working with already has `CMD /init` (BTW, what does `/init` do anyway????), so I guess I'll have to start the script by changing the `CMD` line this way:
That depends on the individual container image. In most cases, `/init`/ is a symlink to `systemd`. When a container is started with either of the two as the command/entrypoint, Podman will automatically do the setup to allow for running systemd inside the container.
``` CMD /my-cleaner.sh & \ && /init ```
4. Cross fingers and hope that when I run `podman stop` on the container, the resulting SIGTERM will be seen by the `trap` command in `my-cleaner.sh` and take the clean up steps. (by the way, how can I check that the clean up actually happened? is there a canonical way to document if the desired actions were taken?)
Does the above look right?
I suggest having a close look whether `/init` is needed in your container. As mentioned above, Podman will setup systemd to run inside the container but that behaviour would now change given the CMD is altered. Kind regards, Valentin
_____________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
CMD /my-cleaner.sh & \ && /init [...] Does the above look right?
Nope, that's not how CMD works. It needs to be given a single command. I don't know the specifics of /init, because I'm using tini instead. But the purpose is probably the same: Have a minimal init process run as pid 1 (process ID 1), so that signal propagation to child processes is handled correctly. Default signal handling is different for pid 1 than for all other processes.[1] The setup I use is: - set ENTRYPOINT to tini, or some other minimal init system - use CMD for what I actually want to run, usually a bash script For example: ENTRYPOINT ["/sbin/tini", "-g", "--"] CMD ["/path/to/my/bash/script.sh"] The bash script starts with this line: #!/usr/bin/env bash Alternatively, you could try CMD ["/bin/bash", "/path/to/my/bash/script.sh"] If I am mistaken and /init is not a minimal init process to be used as entry point, but some regular command to be executed, try this: CMD ["/bin/bash", "-c", "/my-cleaner.sh & exec /init"] That is a single command, namely bash, which then interprets the arguments as a command line and starts child processes as required. Hope that helps, Roland [1] https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-probl...
participants (3)
-
boardbill_unpretended＠simplelogin.co -
Roland Weber -
Valentin Rothberg