docker registry terminates when run under podman
Hallo, Not sure whether it is a right place o ask, but still. I am trying to run docker registry in rootless podman. I have created a 'register' user and started the registry with the following command: podman run --privileged -d --name registry -p 5000:5000 -v /srv/registry:/var/lib/registry --restart=always registry:latest (/srv/registry belongs to 'registry' user, so should be visible in rootless container as belonging to root). So far so good, I test the register with podman search and it returns expected results. But as soon as I exit the ssh session in which I started the container, container terminates, not immediately but in half minute or so. There;s no diagnostic about the reason in podman logs, nor in /var/syslog, nor in dmesg output. Just in case I tried to start it with nohup - same result. I also tried to start it as a service using the following file: [Unit] Description=Containers registry After=syslog.target network.target RequiresMountsFor=/srv/registry [Service] User=registry Group=registry LimitMEMLOCK=infinity LimitNOFILE=65535 Type=simple RemainAfterExit=yes ExecStart=/usr/bin/podman start registry ExecStop=/usr/bin/podman stop -t 60 registry [Install] WantedBy=multi-user.target And it still terminates in about half a minute after service startup. How can I find at least some indication about why this happens? Best regards, -- \ / | | (OvO) | Михаил Иванов | (^^^) | | \^/ | E-mail: ivans@isle.spb.ru | ^ ^ | |
Hi Michael, Thanks for reaching out. On Sat, May 29, 2021 at 10:33 PM Michael Ivanov <ivans@isle.spb.ru> wrote:
Hallo,
Not sure whether it is a right place o ask, but still. I am trying to run docker registry in rootless podman. I have created a 'register' user and started the registry with the following command:
podman run --privileged -d --name registry -p 5000:5000 -v /srv/registry:/var/lib/registry --restart=always registry:latest
(/srv/registry belongs to 'registry' user, so should be visible in rootless container as belonging to root).
So far so good, I test the register with podman search and it returns expected results. But as soon as I exit the ssh session in which I started the container, container terminates, not immediately but in half minute or so. There;s no diagnostic about the reason in podman logs, nor in /var/syslog, nor in dmesg output. Just in case I tried to start it with nohup - same result. I also tried to start it as a service using the following file:
[Unit] Description=Containers registry After=syslog.target network.target RequiresMountsFor=/srv/registry
[Service] User=registry Group=registry LimitMEMLOCK=infinity LimitNOFILE=65535 Type=simple RemainAfterExit=yes ExecStart=/usr/bin/podman start registry ExecStop=/usr/bin/podman stop -t 60 registry
[Install] WantedBy=multi-user.target
And it still terminates in about half a minute after service startup.
How can I find at least some indication about why this happens?
You could use strace on the container process and the corresponding conmon process to see what's going on. I suspect that the container is getting killed, probably by systemd. Can you try using `podman-generate-systemd` for generating the systemd unit file? Kind regards, Valentin
Best regards, -- \ / | | (OvO) | Михаил Иванов | (^^^) | | \^/ | E-mail: ivans@isle.spb.ru | ^ ^ | | _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On Mon, May 31, 2021 at 10:34 PM Michael Ivanov <ivans@isle.spb.ru> wrote:
Hallo Valentin, thanks for response.
I'm a bit skeptical about strace. It needs to be run inside a container. strace uses shared libraries but registry container is based on busybox and required libraries are just not there, even libc is different.
I put registry container inside a pod and when register terminates after exit from ssh session, pod also is stopped cannot be restarted anymore. On attempt to start the pod the following error is returned:
error starting container cf38918cdcd10310cf0b339b429d767e03ff8b221d40d73244f918208d672582: command rootless-cni-infra [alloc cf38918cdcd10310cf0b339b429d767e03ff8b221d40d73244f918208d672582 regnet registry ] in container ff259455829ebb5035e1bd68f75517d7c8983741be347c95967817ed3e3ff6f6 failed with status 1, stdout="", stderr="failed to open netns \"/proc/3037/ns/net\": failed to Statfs \"/proc/3037/ns/net\": no such file or directory\n"
If I stop the pod with registry properly (podman pod stop) then it can be started back without problems.
When I try to cleanup the problem removing the pod, I see the following errors:
registry@cronos:~$ podman pod rm registry ERRO[0000] Error removing container 719c2f2059039d409c9ff7c35b9624f144a0853b1c85848efbdddeb06fddb416 from pod 097d2adb0d65bf8598537c0bb79d2e771fabbb0be726d010e34743075b154541: error freeing lock for container 719c2f2059039d409c9ff7c35b9624f144a0853b1c85848efbdddeb06fddb416: no such file or directory ERRO[0000] Error removing container cf38918cdcd10310cf0b339b429d767e03ff8b221d40d73244f918208d672582 from pod 097d2adb0d65bf8598537c0bb79d2e771fabbb0be726d010e34743075b154541: error freeing lock for container cf38918cdcd10310cf0b339b429d767e03ff8b221d40d73244f918208d672582: no such file or directory ERRO[0000] Error freeing pod 097d2adb0d65bf8598537c0bb79d2e771fabbb0be726d010e34743075b154541 lock: no such file or directory Error: error freeing lock for container 507b1a22cfbec0d6fe11cd513c0e7d31e6a36f5cf036844e03de02d1eda83765: no such file or directory
Are these errors related to corrupt /tmp/podman-run-20000 or /tmp/run-2000 contents? Is it possible that some files under /tmp/podman-run-20000 or /tmp/run-20000 are somehow removed on ssh session exit and crash the container?
It looks like the network namespace has disappeared. Would you open an issue upstream on github.com/containers/podman and provide the requested information (reproducer, output of podman-info, etc.)? Maybe you hit a bug that's already fixed, or a new one. Having all the information at hand will help. Kind regards, Valentin
Best regards,
On 31.05.2021 10:34, Valentin Rothberg wrote:
Hi Michael,
Thanks for reaching out.
On Sat, May 29, 2021 at 10:33 PM Michael Ivanov <ivans@isle.spb.ru> wrote:
Hallo,
Not sure whether it is a right place o ask, but still. I am trying to run docker registry in rootless podman. I have created a 'register' user and started the registry with the following command:
podman run --privileged -d --name registry -p 5000:5000 -v /srv/registry:/var/lib/registry --restart=always registry:latest
(/srv/registry belongs to 'registry' user, so should be visible in rootless container as belonging to root).
So far so good, I test the register with podman search and it returns expected results. But as soon as I exit the ssh session in which I started the container, container terminates, not immediately but in half minute or so. There;s no diagnostic about the reason in podman logs, nor in /var/syslog, nor in dmesg output. Just in case I tried to start it with nohup - same result. I also tried to start it as a service using the following file:
[Unit] Description=Containers registry After=syslog.target network.target RequiresMountsFor=/srv/registry
[Service] User=registry Group=registry LimitMEMLOCK=infinity LimitNOFILE=65535 Type=simple RemainAfterExit=yes ExecStart=/usr/bin/podman start registry ExecStop=/usr/bin/podman stop -t 60 registry
[Install] WantedBy=multi-user.target
And it still terminates in about half a minute after service startup.
How can I find at least some indication about why this happens?
You could use strace on the container process and the corresponding conmon process to see what's going on.
I suspect that the container is getting killed, probably by systemd. Can you try using `podman-generate-systemd` for generating the systemd unit file?
Kind regards, Valentin
Best regards, -- \ / | | (OvO) | Михаил Иванов | (^^^) | | \^/ | E-mail: ivans@isle.spb.ru | ^ ^ | | _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
-- \ / | | (OvO) | Михаил Иванов | (^^^) | | \^/ | E-mail: ivans@isle.spb.ru | ^ ^ | |
_______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On Sat, May 29, 2021 at 11:31:59PM +0300, Michael Ivanov wrote:
Not sure whether it is a right place o ask, but still. I am trying to run docker registry in rootless podman. I have created a 'register' user and started the registry with the following command:
podman run --privileged -d --name registry -p 5000:5000 -v /srv/registry:/var/lib/registry --restart=always registry:latest
(/srv/registry belongs to 'registry' user, so should be visible in rootless container as belonging to root).
So far so good, I test the register with podman search and it returns expected results. But as soon as I exit the ssh session in which I started the container, container terminates, not immediately but in half minute or so.
That sounds a lot like systemd-logind killing the user's processes after the user's logged out. Check the logind.conf(5) man page section on the "KillUserProcesses" setting and the loginctl(1) page's section on the "enable-linger" command for more info, but the short version is that a one time loginctl enable-linger $USER should keep it from doing that. HTH, Nalin
participants (4)
-
Gordon Messmer -
Michael Ivanov -
Nalin Dahyabhai -
Valentin Rothberg