4 p.m.
We're trying to pilot Prometheus services as a container in our enclave (along with
the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I'm using the following URL as a reference to
try and setup DHCP services from the physical network for the Prometheus container. We
are doing it this way so anyone on the network with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn't sure if there was a naming
convention) <==================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network)
<==============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket | grep -i
"podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
? io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled; vendor preset:
disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3 days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
? io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
??49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in 3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet received within
10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in 7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet received within
10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in 15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet received within
10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in 32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus --net=prod_network
-p 9090:9090 --privileged -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest
ERRO[0164] error loading cached network config: network "prod_network" not found
in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the ERRO and WARN.
Since it doesn't run, can't look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the directions from
the article step-by-step.
Also is there a better way to present a container to our prod network that is running
local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
5:20 p.m.
New subject: DHCP lease from physical network not working for container - network is down error
Here would be my approach. I would stop the dhcp service. Then I would
start it manually in a terminal so I could watch it. Then run your
container with podman --log-level=debug and grab that output.
If that does not reveal the problem, then I would start looking at things
like typos for the ethernet interface, selinux, firewalld, kernel modules
are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM Christopher.Miller(a)gd-ms.com <
Christopher.Miller(a)gd-ms.com> wrote:
We’re trying to pilot Prometheus services as a container in our enclave
(along with the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I’m using the following URL as a
reference to try and setup DHCP services from the physical network for the
Prometheus container. We are doing it this way so anyone on the network
with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn’t sure if there
was a naming convention) ç================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network) ç
============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket |
grep -i "podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
● io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled;
vendor preset: disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3
days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
● io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
└─49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in
3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet
received within 10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in
7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet
received within 10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in
15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet
received within 10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in
32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet
received within 10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in
64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled
io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus
--net=prod_network -p 9090:9090 --privileged -v
/etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest
ERRO[0164] error loading cached network config: network "prod_network" not
found in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the
ERRO and WARN. Since it doesn’t run, can’t look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the
directions from the article step-by-step.
Also is there a better way to present a container to our prod network that
is running local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller(a)gd-ms.com
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
6:52 a.m.
New subject: DHCP lease from physical network not working for container - network is down error
From your logs:
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP
packet
received within 10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in
64.627280 seconds
It looks like your system cannot get a dhcp lease. Did you confirm that
your dhcp server is working and the system can reach it?
On Tue, Mar 8, 2022 at 12:22 AM Brent Baude <bbaude(a)redhat.com> wrote:
Here would be my approach. I would stop the dhcp service. Then I
would
start it manually in a terminal so I could watch it. Then run your
container with podman --log-level=debug and grab that output.
If that does not reveal the problem, then I would start looking at things
like typos for the ethernet interface, selinux, firewalld, kernel modules
are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM Christopher.Miller(a)gd-ms.com <
Christopher.Miller(a)gd-ms.com> wrote:
>
>
> We’re trying to pilot Prometheus services as a container in our enclave
> (along with the node exporter and Grafana).
>
>
>
> This is with podman 3.4.2 on RHEL 8.1. I’m using the following URL as a
> reference to try and setup DHCP services from the physical network for the
> Prometheus container. We are doing it this way so anyone on the network
> with a web browser can reach the UI.
>
>
>
> https://www.redhat.com/sysadmin/leasing-ips-podman
>
>
>
> I setup a .conflist file under /etc/cni/net.d created the following file:
>
>
>
> 91-prometheus.conflist (just gave it a generic name, wasn’t sure if there
> was a naming convention) ç================================
>
>
>
>
>
> {
>
> "cniVersion": "0.4.0",
>
> "name": "prod_network", (name of prod_network)
ç
> ============================
>
> "plugins": [
>
> {
>
> "type":
"macvlan",
>
> "master":
"eno1",
>
> "ipam": {
>
> "type":
> "dhcp"
>
> }
>
> }
>
> ]
>
> }
>
>
>
>
>
> I enable and started the following .socket file
>
>
>
> [user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket
> | grep -i "podman"
>
> io.podman.dhcp.socket enabled
>
>
>
> [user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
>
> ● io.podman.dhcp.socket - DHCP Client for CNI
>
> Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket;
> enabled; vendor preset: disabled)
>
> Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3
> days ago
>
> Listen: /run/cni/dhcp.sock (Stream)
>
> CGroup: /system.slice/io.podman.dhcp.socket
>
>
>
>
>
> Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
>
> [user_a@computer_a net.d]$ sudo systemctl is-enabled
> io.podman.dhcp.socket
>
> enabled
>
>
>
>
>
> [user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
>
> ● io.podman.dhcp.service - DHCP Client CNI Service
>
> Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service;
> enabled; vendor preset: disabled)
>
> Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min
> ago
>
> Main PID: 49378 (dhcp)
>
> Tasks: 7 (limit: 45874)
>
> Memory: 9.1M
>
> CGroup: /system.slice/io.podman.dhcp.service
>
> └─49378 /usr/libexec/cni/dhcp daemon
>
>
>
> Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is
> down
>
> Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in
> 3.131274 seconds
>
> Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP
> packet received within 10s
>
> Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in
> 7.313039 seconds
>
> Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP
> packet received within 10s
>
> Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in
> 15.601824 seconds
>
> Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP
> packet received within 10s
>
> Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in
> 32.030425 seconds
>
> Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP
> packet received within 10s
>
> Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in
> 64.627280 seconds
>
>
>
>
>
> [user_a@computer_a net.d]$ sudo systemctl is-enabled
> io.podman.dhcp.service
>
> enabled
>
>
>
>
>
> [user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus
> --net=prod_network -p 9090:9090 --privileged -v
> /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
> docker.io/bitnami/prometheus:latest
>
> ERRO[0164] error loading cached network config: network "prod_network"
> not found in CNI cache
>
> WARN[0164] falling back to loading from existing plugins on disk
>
>
>
>
>
> The container never runs, just shows started status and then outputs the
> ERRO and WARN. Since it doesn’t run, can’t look at the logs for it.
>
>
>
> Where is the best place to start to troubleshoot this? As I followed the
> directions from the article step-by-step.
>
>
>
> Also is there a better way to present a container to our prod network
> that is running local on my RHEL workstation?
>
>
>
> Thanks
>
>
>
>
>
> Chris Miller
>
> Altron INC.
> 703-814-7647
>
> Christopher.miller(a)altroninc.com
>
> Christopher.Miller(a)gd-ms.com
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Podman mailing list -- podman(a)lists.podman.io
> To unsubscribe send an email to podman-leave(a)lists.podman.io
>
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
2:51 p.m.
New subject: DHCP lease from physical network not working for container - network is down error
Other than running status against io.podman.service.dhcp.service and
io.podman.sevice.dhcp.socket, how can I tell the DHCP server for containers is working?
Is there a way to test from the container as its trying to run?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
From: Paul Holzinger <pholzing(a)redhat.com>
Sent: Tuesday, March 8, 2022 7:52 AM
To: Brent Baude <bbaude(a)redhat.com>
Cc: Miller, Christopher (NE) <Christopher.Miller(a)gd-ms.com>; podman(a)lists.podman.io
Subject: Re: [Podman] Re: DHCP lease from physical network not working for container -
network is down error
External E-mail --- CAUTION: This email originated from outside GDMS. Do not click links
or open attachments unless you recognize the sender and know the content is safe.
From your logs:
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
It looks like your system cannot get a dhcp lease. Did you confirm that your dhcp server
is working and the system can reach it?
On Tue, Mar 8, 2022 at 12:22 AM Brent Baude
<bbaude@redhat.com<mailto:bbaude@redhat.com>> wrote:
Here would be my approach. I would stop the dhcp service. Then I would start it manually
in a terminal so I could watch it. Then run your container with podman --log-level=debug
and grab that output.
If that does not reveal the problem, then I would start looking at things like typos for
the ethernet interface, selinux, firewalld, kernel modules are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
<Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>> wrote:
We're trying to pilot Prometheus services as a container in our enclave (along with
the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I'm using the following URL as a reference to
try and setup DHCP services from the physical network for the Prometheus container. We
are doing it this way so anyone on the network with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn't sure if there was a naming
convention) <==================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network)
<==============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket | grep -i
"podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
? io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled; vendor preset:
disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3 days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
? io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
??49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in 3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet received within
10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in 7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet received within
10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in 15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet received within
10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in 32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus --net=prod_network
-p 9090:9090 --privileged -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest<http://docker.io/bitnami/prometheus:latest>
ERRO[0164] error loading cached network config: network "prod_network" not found
in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the ERRO and WARN.
Since it doesn't run, can't look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the directions from
the article step-by-step.
Also is there a better way to present a container to our prod network that is running
local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller@altroninc.com<mailto:Christopher.miller@altroninc.com>
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
_______________________________________________
Podman mailing list -- podman@lists.podman.io<mailto:podman@lists.podman.io>
To unsubscribe send an email to
podman-leave@lists.podman.io<mailto:podman-leave@lists.podman.io>
_______________________________________________
Podman mailing list -- podman@lists.podman.io<mailto:podman@lists.podman.io>
To unsubscribe send an email to
podman-leave@lists.podman.io<mailto:podman-leave@lists.podman.io>
3:26 p.m.
New subject: DHCP lease from physical network not working for container - network is down error
Chris,
I want to just double check something basic. You understand the dhcp
service for podman is just an advanced dhcp proxy server. It does not
generate the leases and pass them to the containers. There needs to be a
dhcp server on the network you are connecting the macvlan to. Are we on
the same page?
On Tue, Mar 8, 2022 at 2:51 PM Christopher.Miller(a)gd-ms.com <
Christopher.Miller(a)gd-ms.com> wrote:
Other than running status against io.podman.service.dhcp.service and
io.podman.sevice.dhcp.socket, how can I tell the DHCP server for containers
is working? Is there a way to test from the container as its trying to run?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller(a)gd-ms.com
*From:* Paul Holzinger <pholzing(a)redhat.com>
*Sent:* Tuesday, March 8, 2022 7:52 AM
*To:* Brent Baude <bbaude(a)redhat.com>
*Cc:* Miller, Christopher (NE) <Christopher.Miller(a)gd-ms.com>;
podman(a)lists.podman.io
*Subject:* Re: [Podman] Re: DHCP lease from physical network not working
for container - network is down error
*External E-mail *--- CAUTION: This email originated from outside GDMS.
Do not click links or open attachments unless you recognize the sender and
know the content is safe.
From your logs:
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet
received within 10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in
64.627280 seconds
It looks like your system cannot get a dhcp lease. Did you confirm that
your dhcp server is working and the system can reach it?
On Tue, Mar 8, 2022 at 12:22 AM Brent Baude <bbaude(a)redhat.com> wrote:
Here would be my approach. I would stop the dhcp service. Then I would
start it manually in a terminal so I could watch it. Then run your
container with podman --log-level=debug and grab that output.
If that does not reveal the problem, then I would start looking at things
like typos for the ethernet interface, selinux, firewalld, kernel modules
are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM Christopher.Miller(a)gd-ms.com <
Christopher.Miller(a)gd-ms.com> wrote:
We’re trying to pilot Prometheus services as a container in our enclave
(along with the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I’m using the following URL as a
reference to try and setup DHCP services from the physical network for the
Prometheus container. We are doing it this way so anyone on the network
with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn’t sure if there
was a naming convention) ç================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network) ç
============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket |
grep -i "podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
● io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled;
vendor preset: disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3
days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
● io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
└─49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in
3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet
received within 10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in
7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet
received within 10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in
15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet
received within 10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in
32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet
received within 10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in
64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled
io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus
--net=prod_network -p 9090:9090 --privileged -v
/etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest
ERRO[0164] error loading cached network config: network "prod_network" not
found in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the
ERRO and WARN. Since it doesn’t run, can’t look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the
directions from the article step-by-step.
Also is there a better way to present a container to our prod network that
is running local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller(a)gd-ms.com
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
7:16 a.m.
New subject: DHCP lease from physical network not working for container - network is down error
Brent, I'm not understanding the Podman Networking documentation for what I'm
trying to achieve. With the issues I'm having, sounds like the DHPC solution
won't work for what I'm wanting to do.
Let me ask this. Is there a way for me to assign IP addresses to containers (maybe
manually with the net command?), so they show up on my dev network (not the 10.88.xx.xx
network)?
Thanks
From: Brent Baude <bbaude(a)redhat.com>
Sent: Tuesday, March 8, 2022 4:26 PM
To: Miller, Christopher (NE) <Christopher.Miller(a)gd-ms.com>; podman(a)lists.podman.io
Subject: Re: [Podman] Re: DHCP lease from physical network not working for container -
network is down error
External E-mail --- CAUTION: This email originated from outside GDMS. Do not click links
or open attachments unless you recognize the sender and know the content is safe.
Chris,
I want to just double check something basic. You understand the dhcp service for podman
is just an advanced dhcp proxy server. It does not generate the leases and pass them to
the containers. There needs to be a dhcp server on the network you are connecting the
macvlan to. Are we on the same page?
On Tue, Mar 8, 2022 at 2:51 PM
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
<Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>> wrote:
Other than running status against io.podman.service.dhcp.service and
io.podman.sevice.dhcp.socket, how can I tell the DHCP server for containers is working?
Is there a way to test from the container as its trying to run?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller@altroninc.com<mailto:Christopher.miller@altroninc.com>
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
From: Paul Holzinger <pholzing@redhat.com<mailto:pholzing@redhat.com>>
Sent: Tuesday, March 8, 2022 7:52 AM
To: Brent Baude <bbaude@redhat.com<mailto:bbaude@redhat.com>>
Cc: Miller, Christopher (NE)
<Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>>;
podman@lists.podman.io<mailto:podman@lists.podman.io>
Subject: Re: [Podman] Re: DHCP lease from physical network not working for container -
network is down error
External E-mail --- CAUTION: This email originated from outside GDMS. Do not click links
or open attachments unless you recognize the sender and know the content is safe.
From your logs:
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
It looks like your system cannot get a dhcp lease. Did you confirm that your dhcp server
is working and the system can reach it?
On Tue, Mar 8, 2022 at 12:22 AM Brent Baude
<bbaude@redhat.com<mailto:bbaude@redhat.com>> wrote:
Here would be my approach. I would stop the dhcp service. Then I would start it manually
in a terminal so I could watch it. Then run your container with podman --log-level=debug
and grab that output.
If that does not reveal the problem, then I would start looking at things like typos for
the ethernet interface, selinux, firewalld, kernel modules are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
<Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>> wrote:
We're trying to pilot Prometheus services as a container in our enclave (along with
the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I'm using the following URL as a reference to
try and setup DHCP services from the physical network for the Prometheus container. We
are doing it this way so anyone on the network with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn't sure if there was a naming
convention) <==================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network)
<==============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket | grep -i
"podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
? io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled; vendor preset:
disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3 days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
? io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
??49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in 3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet received within
10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in 7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet received within
10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in 15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet received within
10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in 32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus --net=prod_network
-p 9090:9090 --privileged -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest<http://docker.io/bitnami/prometheus:latest>
ERRO[0164] error loading cached network config: network "prod_network" not found
in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the ERRO and WARN.
Since it doesn't run, can't look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the directions from
the article step-by-step.
Also is there a better way to present a container to our prod network that is running
local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller@altroninc.com<mailto:Christopher.miller@altroninc.com>
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
_______________________________________________
Podman mailing list -- podman@lists.podman.io<mailto:podman@lists.podman.io>
To unsubscribe send an email to
podman-leave@lists.podman.io<mailto:podman-leave@lists.podman.io>
_______________________________________________
Podman mailing list -- podman@lists.podman.io<mailto:podman@lists.podman.io>
To unsubscribe send an email to
podman-leave@lists.podman.io<mailto:podman-leave@lists.podman.io>
9:41 a.m.
New subject: DHCP lease from physical network not working for container - network is down error
On Wed, 9 Mar 2022 13:16:29 +0000
"Christopher.Miller(a)gd-ms.com" <Christopher.Miller(a)gd-ms.com> wrote:
Let me ask this. Is there a way for me to assign IP addresses to
containers (maybe manually with the net command?), so they show up on
my dev network (not the 10.88.xx.xx network)?
Shameless plug that won't help with your issue right now: that's a use
case that's covered by passt (https://passt.top, not widely packaged, no
official Podman integration yet): there you get (by default) a
non-translated address, assigned by its own DHCP/DHCPv6/NDP server.
I don't know enough about Podman in general to answer this, just that
if your container is rootless, you can't skip NAT at the moment (as far
as I know).
--
Stefano
4:41 p.m.
New subject: DHCP lease from physical network not working for container - network is down error
On 9. Mar 2022, at 16:41, Stefano Brivio <sbrivio(a)redhat.com>
wrote:
On Wed, 9 Mar 2022 13:16:29 +0000
"Christopher.Miller(a)gd-ms.com" <Christopher.Miller(a)gd-ms.com> wrote:
> Let me ask this. Is there a way for me to assign IP addresses to
> containers (maybe manually with the net command?), so they show up on
> my dev network (not the 10.88.xx.xx network)?
Shameless plug that won't help with your issue right now: that's a use
case that's covered by passt (https://passt.top, not widely packaged, no
official Podman integration yet): there you get (by default) a
non-translated address, assigned by its own DHCP/DHCPv6/NDP server.
I don't know enough about Podman in general to answer this, just that
if your container is rootless, you can't skip NAT at the moment (as far
as I know).
This is possible using a macvlan network. The following instructions are for CNI.
First create the configuration file for your new network
/etc/cni/net.d/{your-network}.conflist
(please adapt to your network settings,
especially. you need to adjust your master device and set a name for your-network.)
{
"cniVersion": "0.4.0",
"name": “{your-network}",
"plugins": [
{
"type": "macvlan",
"master": "bn0",
"ipam": {
"type": "host-local",
"ranges": [
[
{
"subnet": "172.25.45.0/24",
"rangeStart": "172.25.45.1",
"rangeEnd": "172.25.45.254",
"gateway": "172.25.45.254"
}
]
],
"routes": [
{"dst": "0.0.0.0/0"}
]
}
},
{
"type": "tuning",
"capabilities": {
"mac": true
}
}
]
}
Start a container using network mac and ip, e.g.:
$ podman run --name foo \
--rm --network {your-network} --mac-address 52:54:cf:cf:cf:10 --ip
172.25.45.197
Best
Christian
--
Stefano
_______________________________________________
Podman mailing list -- podman(a)lists.podman.io
To unsubscribe send an email to podman-leave(a)lists.podman.io
10:09 a.m.
New subject: DHCP lease from physical network not working for container - network is down error
Adding -log-level=dubug to podman run shows the following ERRO and WARN
ERRO[0164] error loading cached network config: network "prod-network" not found
in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
INFO[0164] Deleting pod prod_prometheus_prod_prometheus from CNI network
"prod-network" (type=macvlan)
DEBU[0164] unmounted container
"7f4127775ce3d3575f9877566d478328bc0e8c7ff00f8d80761a98ab9254b66c"
DEBU[0164] Network is already cleaned up, skipping...
DEBU[0164] Cleaning up container
7f4127775ce3d3575f9877566d478328bc0e8c7ff00f8d80761a98ab9254b66c
DEBU[0164] Network is already cleaned up, skipping...
DEBU[0164] Container 7f4127775ce3d3575f9877566d478328bc0e8c7ff00f8d80761a98ab9254b66c
storage is already unmounted, skipping...
DEBU[0164] ExitCode msg: "error configuring network namespace for container
7f4127775ce3d3575f9877566d478328bc0e8c7ff00f8d80761a98ab9254b66c: error adding pod
prod_prometheus_prod_prometheus to cni network \"prod-network\": error calling
dhcp.allocate: no more tries"
Error: error configuring network namespace for container
7f4127775ce3d3575f9877566d478328bc0e8c7ff00f8d80761a98ab9254b66c: error adding pod
prod_prometheus_prod_prometheus to CNI network "prod-network": error calling
DHCP.Allocate: no more tries
Noticed its trying to create a pod, automatically? Is this normal?
Googling the first ERRO line shows this:
https://github.com/containers/podman/issues/12651
My kernel (4.18.0-147.5.1.el8_1.x86_64) shows veth loaded into the kernel:
sudo lsmod | grep -I "veth"
veth 28672 0
SELinux and firewalld are disabled (this is a development env). I think the .conflist
file I created is okay, no typos.
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller(a)altroninc.com
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
From: Brent Baude <bbaude(a)redhat.com>
Sent: Monday, March 7, 2022 6:21 PM
To: Miller, Christopher (NE) <Christopher.Miller(a)gd-ms.com>
Cc: podman(a)lists.podman.io
Subject: Re: [Podman] DHCP lease from physical network not working for container - network
is down error
External E-mail --- CAUTION: This email originated from outside GDMS. Do not click links
or open attachments unless you recognize the sender and know the content is safe.
Here would be my approach. I would stop the dhcp service. Then I would start it manually
in a terminal so I could watch it. Then run your container with podman --log-level=debug
and grab that output.
If that does not reveal the problem, then I would start looking at things like typos for
the ethernet interface, selinux, firewalld, kernel modules are loaded, etc...
On Mon, Mar 7, 2022 at 4:02 PM
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
<Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>> wrote:
We're trying to pilot Prometheus services as a container in our enclave (along with
the node exporter and Grafana).
This is with podman 3.4.2 on RHEL 8.1. I'm using the following URL as a reference to
try and setup DHCP services from the physical network for the Prometheus container. We
are doing it this way so anyone on the network with a web browser can reach the UI.
https://www.redhat.com/sysadmin/leasing-ips-podman
I setup a .conflist file under /etc/cni/net.d created the following file:
91-prometheus.conflist (just gave it a generic name, wasn't sure if there was a naming
convention) <==================================
{
"cniVersion": "0.4.0",
"name": "prod_network", (name of prod_network)
<==============================
"plugins": [
{
"type": "macvlan",
"master": "eno1",
"ipam": {
"type":
"dhcp"
}
}
]
}
I enable and started the following .socket file
[user_a@computer_a net.d]$ sudo systemctl list-unit-files --type=socket | grep -i
"podman"
io.podman.dhcp.socket enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.socket
? io.podman.dhcp.socket - DHCP Client for CNI
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.socket; enabled; vendor preset:
disabled)
Active: active (running) since Fri 2022-02-25 13:41:44 EST; 1 weeks 3 days ago
Listen: /run/cni/dhcp.sock (Stream)
CGroup: /system.slice/io.podman.dhcp.socket
Feb 25 13:41:44 computer_a systemd[1]: Listening on DHCP Client for CNI.
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.socket
enabled
[user_a@computer_a net.d]$ sudo systemctl status io.podman.dhcp.service
? io.podman.dhcp.service - DHCP Client CNI Service
Loaded: loaded (/usr/lib/systemd/system/io.podman.dhcp.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2022-03-07 15:11:45 EST; 1h 35min ago
Main PID: 49378 (dhcp)
Tasks: 7 (limit: 45874)
Memory: 9.1M
CGroup: /system.slice/io.podman.dhcp.service
??49378 /usr/libexec/cni/dhcp daemon
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 network is down
Mar 07 15:16:19 computer_a dhcp[49378]: 2022/03/07 15:16:19 retrying in 3.131274 seconds
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 no DHCP packet received within
10s
Mar 07 15:16:32 computer_a dhcp[49378]: 2022/03/07 15:16:32 retrying in 7.313039 seconds
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 no DHCP packet received within
10s
Mar 07 15:16:49 computer_a dhcp[49378]: 2022/03/07 15:16:49 retrying in 15.601824 seconds
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 no DHCP packet received within
10s
Mar 07 15:17:15 computer_a dhcp[49378]: 2022/03/07 15:17:15 retrying in 32.030425 seconds
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 no DHCP packet received within
10s
Mar 07 15:17:58 computer_a dhcp[49378]: 2022/03/07 15:17:58 retrying in 64.627280 seconds
[user_a@computer_a net.d]$ sudo systemctl is-enabled io.podman.dhcp.service
enabled
[user_a@computer_a net.d]$ sudo podman run -dit --name tcs_prometheus --net=prod_network
-p 9090:9090 --privileged -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
docker.io/bitnami/prometheus:latest<http://docker.io/bitnami/prometheus:latest>
ERRO[0164] error loading cached network config: network "prod_network" not found
in CNI cache
WARN[0164] falling back to loading from existing plugins on disk
The container never runs, just shows started status and then outputs the ERRO and WARN.
Since it doesn't run, can't look at the logs for it.
Where is the best place to start to troubleshoot this? As I followed the directions from
the article step-by-step.
Also is there a better way to present a container to our prod network that is running
local on my RHEL workstation?
Thanks
Chris Miller
Altron INC.
703-814-7647
Christopher.miller@altroninc.com<mailto:Christopher.miller@altroninc.com>
Christopher.Miller@gd-ms.com<mailto:Christopher.Miller@gd-ms.com>
_______________________________________________
Podman mailing list -- podman@lists.podman.io<mailto:podman@lists.podman.io>
To unsubscribe send an email to
podman-leave@lists.podman.io<mailto:podman-leave@lists.podman.io>
984
days inactive
993
days old
8 comments
5 participants
participants (5)
-
Brent Baude -
Christopher.Miller＠gd-ms.com -
Felder, Christian -
Paul Holzinger -
Stefano Brivio