7:24 a.m.
Hi,
My server has multiple network interfaces.
- bond0.3 <- default route via 192.168.3.1
- bond0.7 <- lower priority interface via 192.168.7.1
In this case I want my containers to be using bond0.7.
I add a separate routing table and prepare default route:
echo "7 CONTAINERS" >> /etc/iproute2/rt_tables
sudo ip route add default via 192.168.7.1 table CONTAINERS
sudo ip route add 192.168.7.1 dev bond0.7 table CONTAINERS
sudo ip rule add from 10.89.0.0/24 lookup CONTAINERS
For this I am using rootful containers, though I plan to adapt these to
be rootless when I get them working.
sudo podman network create -d bridge net1
sudo podman run -dt --name test --network net1 --cap-add NET_RAW --rm busybox
sudo podman exec -it test ash
/ # wget -q -O - ifconfig.me
Nothing.........
/ # ip route get 1.1.1.1
1.1.1.1 via 10.89.0.1 dev eth0 src 10.89.0.2
/ # ip route
default via 10.89.0.1 dev eth0
10.89.0.0/24 dev eth0 scope link src 10.89.0.2
/ # traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 46 byte packets
1 128bedbab28b (10.89.0.2) 3034.227 ms !H 3039.811 ms !H 3039.875 ms !H
Now if I delete the rule:
sudo ip rule del from 10.89.0.0/24 lookup CONTAINERS
and do something like a traceroute (which takes the default route out):
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 46 byte packets
1 host.containers.internal (10.89.0.1) 0.006 ms 0.005 ms 0.004 ms
2 192.168.3.1 (192.168.3.1) 0.562 ms 0.815 ms 0.766 ms
...
then add the rule:
sudo ip rule add from 10.89.0.0/24 lookup CONTAINERS
/ # traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 46 byte packets
1 host.containers.internal (10.89.0.1) 0.008 ms 0.005 ms 0.004 ms
2 192.168.7.1 (192.168.7.1) 0.533 ms 0.558 ms 0.440 ms
it works.
The issue is I want to add the routing rules when my machine boots up.
I was able to reproduce the same problem with Docker.
--
Daniel Gray 0x41911F722B0F9AE3
https://mastodon.social/@dngray
11:02 a.m.
New subject: Problem adding a routing rule for containers before they start
So it seems the problem is something to do with ARP replies for my host
not making it into the container when I have a routing rule. For
example:
sudo podman network create -d bridge net1
sudo podman run -dt --name test --network net1 --cap-add NET_RAW --rm busybox
On container:
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if542: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether d2:a6:90:7b:60:80 brd ff:ff:ff:ff:ff:ff
inet 10.89.0.4/24 brd 10.89.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d0a6:90ff:fe7b:6080/64 scope link
valid_lft forever preferred_lft forever
Then in the container:
/ # ping 1.1.1.1
Then on the host:
sudo tcpdump -nn -i cni-podman1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cni-podman1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:20:02.428435 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:20:04.411105 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:20:05.415224 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:20:06.428555 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
/ # arp -a
host.containers.internal (10.89.0.1) at <incomplete> on eth0
Seems that my container doesn't know who 10.89.0.1 is.
If I remove the rule:
sudo ip rule del from 10.89.0.0/24 lookup CONTAINERS
the ARP reply comes through:
sudo tcpdump -nn -i cni-podman1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cni-podman1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:22:09.563747 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:22:09.563801 ARP, Reply 10.89.0.1 is-at ce:b3:e0:ab:b0:ff, length 28
02:22:09.563831 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 14, seq 0, length 64
02:22:09.812966 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 14, seq 0, length 64
02:22:10.563915 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 14, seq 1, length 64
02:22:10.807300 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 14, seq 1, length 64
02:22:14.935078 ARP, Request who-has 10.89.0.4 tell 10.89.0.1, length 28
02:22:14.935128 ARP, Reply 10.89.0.4 is-at d2:a6:90:7b:60:80, length 28
and remains for as long as the arp cache is valid
/ # arp -a
host.containers.internal (10.89.0.1) at ce:b3:e0:ab:b0:ff [ether] on eth0
if I add the rule again on my host
sudo ip rule add from 10.89.0.0/24 lookup CONTAINERS
things continue to work for a while until, the arp cache expires, for
example ping stops:
/ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=59 time=8.701 ms
64 bytes from 1.1.1.1: seq=1 ttl=59 time=8.810 ms
64 bytes from 1.1.1.1: seq=2 ttl=59 time=9.335 ms
64 bytes from 1.1.1.1: seq=3 ttl=59 time=9.660 ms
64 bytes from 1.1.1.1: seq=4 ttl=59 time=8.742 ms
64 bytes from 1.1.1.1: seq=5 ttl=59 time=8.242 ms
64 bytes from 1.1.1.1: seq=6 ttl=59 time=8.940 ms
64 bytes from 1.1.1.1: seq=7 ttl=59 time=8.987 ms
64 bytes from 1.1.1.1: seq=8 ttl=59 time=9.302 ms
^C
--- 1.1.1.1 ping statistics ---
27 packets transmitted, 9 packets received, 66% packet loss
round-trip min/avg/max = 8.242/8.968/9.660 ms
sudo tcpdump -nn -i cni-podman1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cni-podman1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:24:47.781760 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 0, length 64
02:24:47.790349 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 0, length 64
02:24:48.781992 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 1, length 64
02:24:48.790710 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 1, length 64
02:24:49.782086 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 2, length 64
02:24:49.791350 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 2, length 64
02:24:50.782279 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 3, length 64
02:24:50.791865 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 3, length 64
02:24:51.782334 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 4, length 64
02:24:51.791010 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 4, length 64
02:24:52.782396 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 5, length 64
02:24:52.790556 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 5, length 64
02:24:52.801752 ARP, Request who-has 10.89.0.4 tell 10.89.0.1, length 28
02:24:52.801792 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:24:52.801803 ARP, Reply 10.89.0.4 is-at d2:a6:90:7b:60:80, length 28
02:24:53.782472 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 6, length 64
02:24:53.791345 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 6, length 64
02:24:53.815092 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:24:54.782662 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 7, length 64
02:24:54.791579 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 7, length 64
02:24:54.828530 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:24:55.782856 IP 10.89.0.4 > 1.1.1.1: ICMP echo request, id 18, seq 8, length 64
02:24:55.792086 IP 1.1.1.1 > 10.89.0.4: ICMP echo reply, id 18, seq 8, length 64
02:24:56.783054 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:24:57.791743 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:24:58.801868 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:00.783485 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:01.788525 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:02.801872 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:04.784141 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:05.788525 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:06.801764 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:08.784601 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:09.788526 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:10.801864 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
02:25:12.785253 ARP, Request who-has 10.89.0.1 tell 10.89.0.4, length 28
^C
36 packets captured
36 packets received by filter
0 packets dropped by kernel
and then we get back to:
/ # arp -a
host.containers.internal (10.89.0.1) at <incomplete> on eth0
on the container.
So it seems I need another solution to route out via bond0.7, that
doesn't intefer with ARP requests between the containers and
cni-podman1.
--
Daniel Gray 0x41911F722B0F9AE3
https://mastodon.social/@dngray
2:29 p.m.
New subject: Problem adding a routing rule for containers before they start
Okay so what's interesting is if I remove that rule.
sudo ip rule del from 10.89.0.0/24 lookup CONTAINERS
Set a mark
sudo iptables -t mangle -A PREROUTING -s 10.89.0.0/24 -j MARK --set-xmark 0x7/0xffffffff
Enable rp-filter: https://sysctl-explorer.net/net/ipv4/rp_filter/
sudo sysctl -w net.ipv4.conf.all.rp_filter=2
sudo ip rule add fwmark 7 table CONTAINERS prio 700
things seem to work, there seems to be ARP replies making their way into
the container.
--
Daniel Gray 0x41911F722B0F9AE3
https://mastodon.social/@dngray
820
days inactive
820
days old
2 comments
1 participants
participants (1)
-
Daniel Gray