1

Running into an odd issue where pinging from a VLAN interface specifying the IP address works, but when specifying the interface name it does not.

This system is running Rocky Linux 8.5 (4.18.0-348.el8.0.2.x86_64). It has a single network interface plugged in which is set up with 4 bridge interfaces using the old style network-scripts (see scripts below).

When pinging out to 8.8.8.8 without specifying any interface/IP at all, everything works as expected. If I specify br0, all works as expected. If I specify br1, br2 or br3, I get no reply. If I specify the IP address associated with br1, etc, then it works as expected.

Ping output

# PING WORKS
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=104 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 104.224/104.224/104.224/0.000 ms

# PING WORKS
$ ping -I br0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.3.15.22 br0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=101 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=101 ms

# PING FAILS
$ ping -I br1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.4.15.22 br1: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms

# PING WORKS
$ ping -I 10.4.15.22 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.4.15.22 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=4.72 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=4.65 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=4.65 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 4.645/4.672/4.721/0.065 ms

As you can see from the ip addr output below, interface br1 has IP address 10.4.15.22. As well, it is set up to use routing table vlan4 and has the proper rules associated with it.

Output from ip addr show

# ip addr show
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
    inet 10.3.15.22/16 brd 10.3.255.255 scope global br0
       valid_lft forever preferred_lft forever
7: eno1.4@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br1 state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
8: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
    inet 10.4.15.22/16 brd 10.4.255.255 scope global br1
       valid_lft forever preferred_lft forever
9: eno1.5@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br2 state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
10: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
    inet 10.5.15.22/16 brd 10.5.255.255 scope global br2
       valid_lft forever preferred_lft forever
11: eno1.6@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br3 state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
12: br3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:8e:b4:98 brd ff:ff:ff:ff:ff:ff
    inet 10.6.15.22/16 brd 10.6.255.255 scope global br3
       valid_lft forever preferred_lft forever

sysconfig/network-scripts

$ cat /etc/sysconfig/network-scripts/ifcfg-eno1
TYPE=Ethernet
BOOTPROTO=none
DEVICE=eno1
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
MTU=9000

$ cat /etc/sysconfig/network-scripts/ifcfg-eno1.4 
DEVICE=eno1.4
BRIDGE=br1
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
MTU=9000
VLAN=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=10.3.15.22
GATEWAY=10.3.0.1
DNS1=10.3.0.1
NETMASK=255.255.0.0
ONBOOT=yes
NM_CONTROLLED=no
DEFROUTE=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-br1
DEVICE=br1
IPADDR=10.4.15.22
GATEWAY=10.4.0.1
DNS1=10.4.0.1
DEFROUTE=no
TYPE=Bridge
BOOTPROTO=static
NETMASK=255.255.0.0
ONBOOT=yes
NM_CONTROLLED=no

$ cat /etc/sysconfig/network-scripts/route-br1 
10.4.0.0/16 dev br1 src 10.4.15.22 table vlan4
default via 10.4.0.1 dev br1 table vlan4

$ cat /etc/sysconfig/network-scripts/rule-br1 
from 10.4.15.22/32 table vlan4
to 10.4.15.22/32 table vlan4

/etc/iproute2/rt_tables

$ cat /etc/iproute2/rt_tables
3 vlan3
4 vlan4
5 vlan5
6 vlan6

ip route and ip rule output

$ ip route show
default via 10.3.0.1 dev br0 
10.3.0.0/16 dev br0 proto kernel scope link src 10.3.15.22 
10.4.0.0/16 dev br1 proto kernel scope link src 10.4.15.22 
10.5.0.0/16 dev br2 proto kernel scope link src 10.5.15.22 
10.6.0.0/16 dev br3 proto kernel scope link src 10.6.15.22 
169.254.0.0/16 dev br0 scope link metric 1006 
169.254.0.0/16 dev br1 scope link metric 1008 
169.254.0.0/16 dev br2 scope link metric 1010 
169.254.0.0/16 dev br3 scope link metric 1012

$ ip route show table vlan4
default via 10.4.0.1 dev br1 
10.4.0.0/16 dev br1 scope link src 10.4.15.22

$ ip rule
0:  from all lookup local
32758:  from all to 10.6.15.22 lookup vlan6
32759:  from 10.6.15.22 lookup vlan6
32760:  from all to 10.5.15.22 lookup vlan5
32761:  from 10.5.15.22 lookup vlan5
32762:  from all to 10.4.15.22 lookup vlan4
32763:  from 10.4.15.22 lookup vlan4
32764:  from all to 10.3.15.22 lookup vlan3
32765:  from 10.3.15.22 lookup vlan3
32766:  from all lookup main
32767:  from all lookup default
Improve this question

1 Answer 1

Reset to default
1

To have routing tables be adequately selected by routing rules with an interface bound using SO_BINDTODEVICE, rules with the keyword oif should also be used:

oif NAME

select the outgoing device to match. The outgoing interface is only available for packets originating from local sockets that are bound to a device.

Else during initial lookup, when no source address is also specified, the lookup is using the default INADDR_ANY (aka 0.0.0.0/0) as source selector before figuring out the correct source IP address later (eg from hinted source or other algorithms). As no added routing rule will match INADDR_ANY all the additional tables are skipped and this will end up evaluating only with the main routing table with an additional filter by the bound device: no default route is defined and thus no adequate gateway is used, except for br0 which has the default route also in the main table.

So these commands should be added, each line in its respective /etc/sysconfig/network-scripts/rule-brX file (minus ip rule add ):

ip rule add oif br0 lookup 3
ip rule add oif br1 lookup 4
ip rule add oif br2 lookup 5
ip rule add oif br3 lookup 6

so that the adequate routing table is looked up when binding to an interface, thus providing the correct gateway in each case.

All the kernel evaluation can be verified with ip route get:

ip route get

get a single route

this command gets a single route to a destination and prints its contents exactly as the kernel sees it.

Before the additional rules

  • br0

    Assuming that for the sake of symmetry there's also a routing table like this (but it doesn't really matter: the main routing table is good enough for br0):

    default via 10.3.0.1 dev br0 table 3 
10.3.0.0/16 dev br0 table 3 scope link src 10.3.15.22 


# ip route get from 10.3.15.22 to 8.8.8.8
8.8.8.8 from 10.3.15.22 via 10.3.0.1 dev br0 table 3 uid 0 
    cache 
# ip route get oif br0 to 8.8.8.8
8.8.8.8 via 10.3.0.1 dev br0 src 10.3.15.22 uid 0 
    cache

    In the 2nd case, when just binding to the interface without selecting a source address, the main table gets used, but that's still correct, because the main table also has the default route for br0: it works

  • br1

    # ip route get from 10.4.15.22 to 8.8.8.8
8.8.8.8 from 10.4.15.22 via 10.4.0.1 dev br1 table 4 uid 0 
    cache 
# ip route get oif br1 to 8.8.8.8
8.8.8.8 dev br1 src 10.4.15.22 uid 0 
    cache

    In the 2nd case, as there's nothing defining a gateway in the main table, the route uses no gateway and because the interface is forced, it's assumed that 8.8.8.8 is directly connected (tcpdump would probably show ARP queries emitted about 8.8.8.8 if anything).

    It doesn't work.

After the additional rules for the bound interface case

  • br0

    # ip route get oif br0 to 8.8.8.8
8.8.8.8 via 10.3.0.1 dev br0 table 3 src 10.3.15.22 uid 0 
    cache

    While the overall result is the same: it works, this time the additional table was used.

  • br1

    # ip route get oif br1 to 8.8.8.8
8.8.8.8 via 10.4.0.1 dev br1 table 4 src 10.4.15.22 uid 0 
    cache

    This time, the oif br1 selector selected routing table 4 which does define a proper default route with a proper gateway: it works.

Note: A similar setup using L3 interfaces (eg: VPN interfaces like WireGuard or OpenVPN in tun mode) wouldn't require this, because L3 interfaces don't need any gateway (even if defined it's not used since there's no L2 layer below IP to resolve) and thus no specific route definition is needed for traffic to be emitted when binding to such L3 interface. Anyway it's still a good idea to use it too, especially if the routing table contains routing negations.

Improve this answer
1
  • To be clear: this is not related to VLANs, it's related to L2 interfaces and losing the gateway information when (only) binding to such interface.
    – A.B
    Commented Mar 18, 2023 at 14:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .