1

Set up/configuration:

I have a RHEL 8 server, running Asterisk 15.x, that has 2 NICs. NMCLI is used for networking

NIC0 (eno5np0) is on the trusted network and is configured as a static IPv4 and NIC1 (ens1f0) is on the untrusted side as a DHCP IPv4. Both are UP,BROADCAST,RUNNING,MULTICAST

NIC0 is where I access the server from, is an internal network and has an IP of 10.38.149.244/32 (GW is 10.38.149.241) NIC1 is supposed to allow access to the internet (for SIP calling) and has an IP of 10.0.0.91 (GW is 10.0.0.1)

Firewall status - inactive(dead) SE Linux status - disabled

Server #1 interface configs:

TYPE=Ethernet
DEVICE=eno5np0
UUID=77c33e7a-7dba-4785-b749-dc0883b46cef
ONBOOT=yes
IPADDR=10.38.149.244
NETMASK=255.255.255.240
GATEWAY=10.38.149.241
NM_CONTROLLED=yes
BOOTPROTO=none
DOMAIN=comcast.net
DNS1=69.252.80.80
DNS2=69.252.81.81
DEFROUTE=yes
USERCTL=no
IPV4_FAILURE_FATAL=yes

TYPE=Ethernet
BOOTPROTO=dhcp
NM_CONTROLLED=yes
PEERDNS=no
DEFROUTE=no
NAME=ens1f0
UUID=249b95f0-d490-4402-b654-43695317d738
DEVICE=ens1f0
ONBOOT=yes
PROXY_METHOD=none
BROWSER_ONLY=no
IPV4_FAILURE_FATAL=no
IPV6_DISABLED=yes
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no

Kernel IP routing table:

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.38.149.241 0.0.0.0 UG 100 0 0 eno5np0
10.0.0.0 0.0.0.0 255.255.255.0 U 101 0 0 ens1f0
10.38.149.240 0.0.0.0 255.255.255.240 U 100 0 0 eno5np0

I do not have any nft tables/IP tables configured

I am SSH'd to the 10.38.149.244 interface (NIC0, aka eno5np0), have sudo access

I run the following command for NIC0: sudo traceroute -i eno5np0 8.8.8.8 and get a nice, completed trace to 8.8.8.8

I run the following command for NIC1: sudo traceroute -i ens1f0 8.8.8.8 and it times out, no packets received

I cannot ping/traceroute to any ip address through NIC1 (sudo ping -I and sudo traceroute -i) except 10.0.0.1, which is the gateway. It is almost like if it isn't the gateway the packets are not making it back into the server for processing?

Issue/Problem

So, after trying both ping and traceroute and not receiving a response, I opened a second SSH session to the server and did a tcpdump while running a ping to 8.8.8.8 over the NIC1 interface in my first SSH session:

TCP Dump

sudo tcpdump -vv --interface ens1f0 -c 10
dropped privs to tcpdump
tcpdump: listening on ens1f0, link-type EN10MB (Ethernet), capture size 262144 bytes

15:21:09.450739 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms
          rdnss option (25), length 40 (5):  lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net
            0x0000:  0000 0000 00b4 2001 0558 feed 0000 0000
            0x0010:  0000 0000 0001 2001 0558 feed 0000 0000
            0x0020:  0000 0000 0002
          prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s
            0x0000:  40c0 0000 012c 0000 012c 0000 0000 2601
            0x0010:  0000 0200 0080 0000 0000 0000 0000
          route info option (24), length 24 (3):  ::/0, pref=medium, lifetime=180s
            0x0000:  0000 0000 00b4 0000 0000 0000 0000 0000
            0x0010:  0000 0000 0000
          source link-address option (1), length 8 (1): 10:56:11:86:6e:92
            0x0000:  1056 1186 6e92


15:21:10.415419 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28


15:21:11.439570 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28

15:21:12.453262 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms
          rdnss option (25), length 40 (5):  lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net
            0x0000:  0000 0000 00b4 2001 0558 feed 0000 0000
            0x0010:  0000 0000 0001 2001 0558 feed 0000 0000
            0x0020:  0000 0000 0002
          prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s
            0x0000:  40c0 0000 012c 0000 012c 0000 0000 2601
            0x0010:  0000 0200 0080 0000 0000 0000 0000
          route info option (24), length 24 (3):  ::/0, pref=medium, lifetime=180s
            0x0000:  0000 0000 00b4 0000 0000 0000 0000 0000
            0x0010:  0000 0000 0000
          source link-address option (1), length 8 (1): 10:56:11:86:6e:92
            0x0000:  1056 1186 6e92


15:21:12.463417 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28


15:21:13.487416 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28

15:21:13.546246 IP (tos 0x0, ttl 4, id 8382, offset 0, flags [DF], proto UDP (17), length 219)
    169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 191


15:21:13.546273 IP (tos 0x0, ttl 4, id 8383, offset 0, flags [DF], proto UDP (17), length 223)
    169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 195


15:21:13.546320 IP (tos 0x0, ttl 4, id 8384, offset 0, flags [DF], proto UDP (17), length 227)
    169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 199


15:21:13.546419 IP (tos 0x0, ttl 4, id 8385, offset 0, flags [DF], proto UDP (17), length 220)
    169.254.100.1.50759 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 192


10 packets captured
10 packets received by filter
0 packets dropped by kernel

I am not understanding why, if the server is doing an ARP request, am I not getting a response? Is the issue on my server not knowing how to respond back to NIC0 with my ping request (where I am SSH'd into)? Is it the gateway being misconfigured? Do I need a NFT table/IP Table configured?

I am familiar with how to do this in RHEL 6.x, but not in RHEL 8 (configuration using IP route and IP tables was simpler I think?)

At the end of the day (for a broader picture) - I have Softphone clients to register to the Asterisk PBX on the internal/trusted network coming in over NIC0 (which works). They need to make phone calls to endpoints on the Internet, but only over NIC1 - and right now I cannot even ping to any location on the internet over the NIC1 interface.

Any help/guidance would be very much appreciated at this point - I am lost and desperate.

Edit/additional clarification: I have a RHEL 6.x server, with exact same physical connections and NICs that this does work on. I have tried to use the iptable and routing table from this Server #2 on Server #1 above and it will not work (I get booted when I turn the interface back up, and have to reboot the device to clear out any unsaved changes before I can get back in) I did use the iptables to nft translate function just as an FYI. I have plugged my Server #1 NIC1 into the known good modem/internet access port that Server #2 is using and still no change.

Server #2 interface configs:

DEVICE=eth0
BOOTPROTO=none
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
UUID="da71293d-4351-481e-a794-bc5850e29391"
IPADDR=10.38.149.243
DNS1=10.168.241.223
DOMAIN=comcast.net
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
#HWADDR=00:1C:23:CF:BC:E3
HWADDR=00:1c:23:cf:bc:e3
NETMASK=255.255.255.240
USERCTL=no
PEERDNS=yes
GATEWAY=10.38.149.241

DEVICE=eth1
BOOTPROTO=dhcp
HWADDR=00:1c:23:cf:bc:e5
NM_CONTROLLED=yes
ONBOOT=yes
DEFROUTE=yes
TYPE=Ethernet
UUID="78bc69cb-80ca-41d1-af9c-66703eb952d5"
USERCTL=no
PEERDNS=yes
IPV6INIT=no

Kernel Routing Table on Server #2

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.1 255.255.255.255 UGH 0 0 0 eth1
10.38.149.240 0.0.0.0 255.255.255.240 U 0 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.0.0.0 10.38.149.241 255.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth1

iptables -L on Server #2

Chain INPUT (policy ACCEPT)

target prot opt source destination status?
DROP all -- c-67-164-235-175.devivce1.mi.inetprovider.net anywhere
DROP all -- c-67-164-235-175.devivce1.mi.inetprovider.net anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT udp -- anywhere anywhere udp dpt:sip
ACCEPT udp -- anywhere anywhere udp dpts:ndmp:dnp
DROP all -- 106.0.0.0/8 anywhere
DROP all -- 106.0.0.0/8 anywhere
DROP all -- host-87-0-0-0.retail.blockeddomain.notus/8 anywhere
DROP all -- 113.0.0.0/8 anywhere
DROP all -- 117.0.0.0/8 anywhere
DROP all -- p5b000000.dip0.blockeddomain.notus/8 anywhere

Chain FORWARD (policy ACCEPT)

target prot opt source destination
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)

target prot opt source destination
Improve this question
3
  • Please take a moment to edit your question and fix the formatting problems. In particular, the tcpdump output is completely useless as currently presented, and the kernel routing table isn't a table.
    – larsks
    Commented May 19 at 12:33
  • Hope the TCPDUMP is easier to read now? Not sure of the best format for it?
    – ripvw32
    Commented May 20 at 15:36
  • iptables -L is incomplete - it won't show if a particular rule applies to all network interfaces, or to a specific interface only. Use iptables -Lvn to see the complete definition of each rule.
    – telcoM
    Commented May 21 at 19:56

1 Answer 1

Reset to default
0

A gateway with a genmask of 0.0.0.0 is a "default gateway". In other words, it means "unless otherwise specified, the rest of the world is this way." In a simple multi-homed host configuration, there should be just one default gateway in the entire system at a time. You cannot really use two NATted internet connections in parallel, unless you at least have an exact control of how the NAT is done. The best you can probably do with two average consumer-grade Internet connections (with a provider-dictated NAT on each) is to use one as a primary, with an automatic fall-back to the second one if the first one loses a link.

You have a default gateway configured on eno5np0 interface, but not on the ens1f0 interface. There are no more specific routes either, just the auto-generated network entries for the local network segment of each interface. This is probably because your system's DHCP client detects you already have a statically-configured default gateway on eno5np0, so it won't mess things up by adding another.

As a result, the system has no clue that it should send outgoing traffic addressed to 8.8.8.8 via 10.0.0.1 if sending it out through ens1f0. By your routing table, only addresses in the form of 10.0.0.* should be reachable through that interface.

But because you are explicitly telling traceroute to try and reach 8.8.8.8 via ens1f0, it assumes you are trying to debug a possibly misconfigured server in your local network segment, and sends out direct ARP requests for that IP address.

You should never see an ARP request for 8.8.8.8 in your own network (unless you are next-door to a Google's datacenter and have somehow managed to get a neighborly direct-link to their network :-) unless something is misconfigured. Instead, you should see an ARP request for the default gateway in that segment, and then this system should send any outgoing traffic bound to 8.8.8.8 to the gateway.

Your system also probably has a IP Reverse Path Filtering in effect. Basically, since your routing table says that the ens1f0 interface has connectivity to the 10.0.0.* addresses only, any packets with source addresses not in that range coming in via that interface would get discarded as fakes. That would cause any responses from 8.8.8.8 coming in via 10.0.0.1 to be discarded as long as your current routing table is in effect.

Improve this answer
4
  • Dude... seriously, thank you so much. I truly appreciate it. I really needed that. SO - if you wouldn't mind - in order to duplicate what is on my RHEL 6 server, the below is the way to do it?
    – ripvw32
    Commented May 21 at 22:43
  • Keep routes: ens1f0 10.0.0.0/24 0.0.0.0 eno5np0 10.38.149.240/28 0.0.0.0
    – ripvw32
    Commented May 21 at 22:43
  • Remove route: sudo nmcli connection modify eno5np0 -ipv4.routes "0.0.0.0/0 10.38.149.241" (remove default route from NIC0)
    – ripvw32
    Commented May 21 at 22:45
  • Add routes: sudo nmcli connection modify ens1f0 +ipv4.routes "0.0.0.0/0 10.0.0.1" (add default route to NIC1) sudo nmcli connection modify eno5np0 +ipv4.routes "10.0.0.0/8 10.38.149.241" sudo nmcli connection modify ens1f0 +ipv4.routes "0.0.0.0/32 10.0.0.1"
    – ripvw32
    Commented May 21 at 22:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .