ping -I only works on the interface with the default gateway?

Question

I've got a couple of systems that both act the same way. Each has an interface on two different subnets, each of which has internet access, but I can only use ping -I successfully on one of the interfaces. For instance, a Linux box running Debian 8.11

Here's /etc/network/interfaces:

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.0.94
netmask 255.255.255.0
gateway 192.168.0.1

# VLAN 782 'PUB', Public
auto eth0.782
iface eth0.782 inet static
      address 192.168.2.94
      netmask 255.255.255.0

and my routing table:

netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0.782

I can:

root@pbx3:~# ping google.com
PING google.com (172.217.2.78) 56(84) bytes of data.
64 bytes from mia09s01-in-f14.1e100.net (172.217.2.78): icmp_seq=1 ttl=48 time=63.6 ms

I can:

ping -I eth0 google.com
PING google.com (172.217.2.78) from 192.168.0.94 eth0: 56(84) bytes of data.
64 bytes from ord08s13-in-f14.1e100.net (172.217.2.78): icmp_seq=1 ttl=48 time=70.2 ms

but I can't:

ping -I eth0.782 google.com
PING google.com (172.217.2.78) from 192.168.2.94 eth0.782: 56(84) bytes of data.
From 192.168.2.94 icmp_seq=1 Destination Host Unreachable

Though I can nmap and ping devices on that subnet:

ping -I eth0.782 192.168.2.1
PING 192.168.2.1 (192.168.2.1) from 192.168.2.94 eth0.782: 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.385 ms

(same problem with another system, Centos 6.8 using eth0 and eth1)

My good friend Google seems to hint that this has something to do with my routing table, but I can't seem to discover exactly what's wrong, or exactly how to fix it. I've tried adding the 192.168.2.1 gateway to the eth0.782 network, but then the eth0 network disappears, so clearly there can be only one gateway...

Thanks in advance for any hints!

Answer 1

Partial answer:

For starters, you need the gateway for the subnet. If you can't get DHCP to work, do it manually for now:

ip route add 192.168.2.0/24 via 192.168.2.1 dev eth0.782

(You may have to delete the route without the gateway first; I never tried updating a route without a gateway. Use ip route to get all routes, then ip route del ... with the info from the previous step to delete it.)

Then verify with ip route (or netstat -rn) that the gateway is correctly set.

Next try the ping again. I am actually not sure if binding to an interface will make it ignore the main routing table. If that doesn't work, you can use policy routing to set up two default rules that are distinguished by source address.

If you plan to use this setup as a way to reach the internet via two different interfaces: This is not going to work (and it's a FAQ, every week or so someone tries a variant of this).

So if this is an XY question, where your X is "I want to reach the internet in two different ways", and your Y is "I want to ping without a default route", you'll need to elaborate your X (e.g. "will all applications you use be able to bind to a specific interface?" vs e.g. "why not use network namespaces"?)

Edit

So if the use case "check connectivity in other LAN, do something based on the result", the probably simplest way is to create a different network namespace, configure it in /etc/netns/your_namespace/network/interfaces (this is a feature of ip netns exec, read up on it), use inet dynamic instead of inet static to make sure it gets a DHCP address and default route, and then run ping in this namespace using ip netns exec.

You no more have conflicted default routes, problem solved. You'll have to read up on network namespaces, there are e.g. plenty questions here and on unix.stackexchange.

You could also try to combine policy routing with ping -I binding, but that's likely going to be more difficult, in particular if you want DHCP configuration for your second interface.

Answer 2

Maybe is something related with rp_filter (https://www.theurbanpenguin.com/rp_filter-and-lpic-3-linux-security/). This setting could drop an incoming message (let say, a ping reply) if the source address is not routable (let say, there is no default rule for the outgoing interface). This works for me:

Temporal change:

echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

Permanent change: Edit /etc/sysctl.conf with:

net.ipv4.conf.all.rp_filter=2

Answer 3

OK, so I got it to mostly work, thanks to dirkt for the hints, unfortunately I don't have enough reputation to mark your answer useful, sigh. First, I started by updating a different machine (Raspberry Pi running Debian 9.9) and following the instructions at https://www.sbprojects.net/projects/raspberrypi/vlan.php and now I can (for instance)

ping -I eth0.3750 google.com

or

traceroute -i eth0.3750 google.com

Unfortunately it stops working if I set a static IP, but I don't need to know the IP address if I'm using eth0.3750 so I guess that's close enough. FWIW, my other machines were older and didn't have all the right software, so upgrading/updating them was more trouble than using a Raspberry Pi. 8*)