0

I have a Linux box that I have setup with 3 network interfaces, connected to 3 different networks, each with their own subnet. At a high level I am trying to achieve communications between devices on each of these networks to devices on each of the other networks.

The 3 network interfaces are configured as described:

eth0 "WAN Port"

eth0 is a fairly standard interface configured as a DHCP client and connected to an upstream network which has a DHCP server and provides a gateway to the internet.

IP: 192.168.100.105/24 (address assigned by DHCP server on network)

eth1 "LAN Port"

eth1 has been configured as a DHCP Server and acts as the gateway for a small LAN. The interface has been configured with the following command:

nmcli c add con-name eth1-dhcp-server type ethernet ifname eth1 ipv4.method shared ipv6.method ignore ipv4.addresses 10.79.0.1

IP: 10.79.0.1/24

wlan0 "WiFi AP"

wlan0 is configured similarly to eth1, providing a small LAN, but instead using a WiFi Access Point. The Access Point, DHCP server and static IP, is setup with the following commands:

nmcli d wifi hotspot ifname wlan0 ssid <ss_id> password <password>
nmcli c modify Hotspot connection.autoconnect yes ipv4.addresses 10.80.0.1

IP: 10.80.0.1/24

Ping Test Setup

To test communication across networks I've connected the following devices to each of the networks.

Device Network (Identified by Linux box's network interface) IP
PC1 eth0 192.168.100.145
PC2 eth1 10.79.0.77
PC3 wlan0 10.80.0.16

Ping Test Results

PC1 (192.168.100.145):

ping 10.79.0.77 ✕ Fail

ping 10.80.0.16 ✓ Success

PC2 (10.79.0.77):

ping 192.168.100.145 ✓ Success

ping 10.80.0.16 ✓ Success

PC3 (10.80.0.16):

ping 192.168.100.145 ✓ Success

ping 10.79.0.77 ✕ Fail

The results of this tests indicate some positive results, with successful routing of packets with destinations of 192.168.100.0/24 and 10.80.0.0/24, but not to 10.79.0.0/24.

Troubleshooting Attempts

After battling to try and get all packets routed successfully I've still made very little progress. However I have come across one interesting find to do with modifying the route metric of the 10.80.0.0/24 route, changing it from 150 to 101 (lower than the 10.79.0.0/24 route). This changes the ping behaviour so that packets with destination 10.79.0.0/24 get successfully routed, and now 10.80.0.0/24 are not. Setting them to the same metric did not get them to both work, so I'm not sure there is a related permanent fix here...

Original routing table:

$ ip route
default via 192.168.100.1 dev eth0 proto dhcp metric 100 
10.79.0.0/24 dev eth1 proto kernel scope link src 10.79.0.1 metric 101 
10.80.0.0/24 dev wlan0 proto kernel scope link src 10.80.0.1 metric 150
169.254.0.0/16 dev eth0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.105 metric 100

Modified routing table:

$ ip route
default via 192.168.100.1 dev eth0 proto dhcp metric 100 
10.79.0.0/24 dev eth1 proto kernel scope link src 10.79.0.1 metric 101 
10.80.0.0/24 dev wlan0 proto kernel scope link src 10.80.0.1 metric 100 ** reduced from 150
169.254.0.0/16 dev eth0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.105 metric 100

Is there anyone out there with a solid understanding on what is going on here? And even better yet a solution!

Thanks for taking a look.

Improve this question
7
  • It's unclear why this changes the behavior, as routes aren't sorted by metric in the first place. Routes are chosen based on whether they match the destination (and two different /24 routes cannot possibly match the same destination), then based on prefix length (longest prefix having higher priority), and only then -- i.e. if you somehow had two "10.80.0.0/24" routes -- would the metric become relevant.
    – grawity_u1686
    Commented Nov 22, 2021 at 21:04
  • Yeah, I was surprised to find this as well. Any thoughts on other changes to try?
    – this_is_not_a_test
    Commented Nov 22, 2021 at 22:08
  • Check whether sysctl net.ipv4.conf.{all,eth0,eth1,wlan0}.rp_filter is set to 0, not 1 or 2. Use ip route get <addr> to query which route is used for what destination, in case there's something else (like policies in ip rule) that alters the behavior.
    – grawity_u1686
    Commented Nov 23, 2021 at 5:23
  • I checked my rp_filters and I had a mix of interfaces set to 1s and 2s: $ sysctl net.ipv4.conf.{all,eth0,eth1,wlan0}.rp_filter net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.eth0.rp_filter = 2 net.ipv4.conf.eth1.rp_filter = 1 net.ipv4.conf.wlan0.rp_filter = 1 Unfortunately after setting them all to 0, I still was seeing the same behaviour.
    – this_is_not_a_test
    Commented Nov 23, 2021 at 14:51
  • The ip route get <addr> also returned as expected for sample addresses on each subnet: ip route get 10.79.0.77 10.79.0.77 dev eth1 src 10.79.0.1 uid 1000 and ip route get 10.80.0.16 10.80.0.16 dev wlan0 src 10.80.0.1 uid 1000
    – this_is_not_a_test
    Commented Nov 23, 2021 at 14:55

1 Answer 1

Reset to default
0

I think I've finally figured out the issue, and it had to do with iptable rules. I should have known.

I noticed in my iptable that there was REJECT policies with reject-with icmp-port-unreachable. This matched up with the Destination Port Unreachable response I was receiving from my failed pings.

You can see the original iptables output here:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.79.0.0/24         state RELATED,ESTABLISHED
ACCEPT     all  --  10.79.0.0/24         anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             10.80.0.0/24         state RELATED,ESTABLISHED
ACCEPT     all  --  10.80.0.0/24         anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

I have changed the order such that the REJECT all-- anywhere anywhere reject-with icmp-port-unreachable rules were below all of the 10.79.0.0/24 and 10.80.0.0/24 rules.

Once I had changed the iptables accordingly everything seems to be working now! Updated iptables:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.79.0.0/24         state RELATED,ESTABLISHED
ACCEPT     all  --  10.79.0.0/24         anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             10.80.0.0/24         state RELATED,ESTABLISHED
ACCEPT     all  --  10.80.0.0/24         anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

I want to also thank @user1686 for their input and providing some helpful suggestions along the way!

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .