Set up/configuration:
I have a RHEL 8 server, running Asterisk 15.x, that has 2 NICs. NMCLI is used for networking
NIC0 (eno5np0) is on the trusted network and is configured as a static IPv4 and NIC1 (ens1f0) is on the untrusted side as a DHCP IPv4. Both are UP,BROADCAST,RUNNING,MULTICAST
NIC0 is where I access the server from, is an internal network and has an IP of 10.38.149.244/32 (GW is 10.38.149.241) NIC1 is supposed to allow access to the internet (for SIP calling) and has an IP of 10.0.0.91 (GW is 10.0.0.1)
Firewall status - inactive(dead) SE Linux status - disabled
Server #1 interface configs:
TYPE=Ethernet
DEVICE=eno5np0
UUID=77c33e7a-7dba-4785-b749-dc0883b46cef
ONBOOT=yes
IPADDR=10.38.149.244
NETMASK=255.255.255.240
GATEWAY=10.38.149.241
NM_CONTROLLED=yes
BOOTPROTO=none
DOMAIN=comcast.net
DNS1=69.252.80.80
DNS2=69.252.81.81
DEFROUTE=yes
USERCTL=no
IPV4_FAILURE_FATAL=yes
TYPE=Ethernet
BOOTPROTO=dhcp
NM_CONTROLLED=yes
PEERDNS=no
DEFROUTE=no
NAME=ens1f0
UUID=249b95f0-d490-4402-b654-43695317d738
DEVICE=ens1f0
ONBOOT=yes
PROXY_METHOD=none
BROWSER_ONLY=no
IPV4_FAILURE_FATAL=no
IPV6_DISABLED=yes
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
Kernel IP routing table:
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
0.0.0.0 | 10.38.149.241 | 0.0.0.0 | UG | 100 | 0 | 0 | eno5np0 |
10.0.0.0 | 0.0.0.0 | 255.255.255.0 | U | 101 | 0 | 0 | ens1f0 |
10.38.149.240 | 0.0.0.0 | 255.255.255.240 | U | 100 | 0 | 0 | eno5np0 |
I do not have any nft tables/IP tables configured
I am SSH'd to the 10.38.149.244 interface (NIC0, aka eno5np0), have sudo access
I run the following command for NIC0: sudo traceroute -i eno5np0 8.8.8.8 and get a nice, completed trace to 8.8.8.8
I run the following command for NIC1: sudo traceroute -i ens1f0 8.8.8.8 and it times out, no packets received
I cannot ping/traceroute to any ip address through NIC1 (sudo ping -I and sudo traceroute -i) except 10.0.0.1, which is the gateway. It is almost like if it isn't the gateway the packets are not making it back into the server for processing?
Issue/Problem
So, after trying both ping and traceroute and not receiving a response, I opened a second SSH session to the server and did a tcpdump while running a ping to 8.8.8.8 over the NIC1 interface in my first SSH session:
TCP Dump
sudo tcpdump -vv --interface ens1f0 -c 10
dropped privs to tcpdump
tcpdump: listening on ens1f0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:21:09.450739 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120
hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms
rdnss option (25), length 40 (5): lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net
0x0000: 0000 0000 00b4 2001 0558 feed 0000 0000
0x0010: 0000 0000 0001 2001 0558 feed 0000 0000
0x0020: 0000 0000 0002
prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s
0x0000: 40c0 0000 012c 0000 012c 0000 0000 2601
0x0010: 0000 0200 0080 0000 0000 0000 0000
route info option (24), length 24 (3): ::/0, pref=medium, lifetime=180s
0x0000: 0000 0000 00b4 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000
source link-address option (1), length 8 (1): 10:56:11:86:6e:92
0x0000: 1056 1186 6e92
15:21:10.415419 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28
15:21:11.439570 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28
15:21:12.453262 IP6 (flowlabel 0x9b9b7, hlim 255, next-header ICMPv6 (58) payload length: 120) fe80::1256:11ff:fe86:6e92 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 120
hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 180s, reachable time 0ms, retrans timer 0ms
rdnss option (25), length 40 (5): lifetime 180s, addr: device1.inetprovider.net addr: device2.inetprovider.net
0x0000: 0000 0000 00b4 2001 0558 feed 0000 0000
0x0010: 0000 0000 0001 2001 0558 feed 0000 0000
0x0020: 0000 0000 0002
prefix info option (3), length 32 (4): 2601:0:200:80::/64, Flags [onlink, auto], valid time 300s, pref. time 300s
0x0000: 40c0 0000 012c 0000 012c 0000 0000 2601
0x0010: 0000 0200 0080 0000 0000 0000 0000
route info option (24), length 24 (3): ::/0, pref=medium, lifetime=180s
0x0000: 0000 0000 00b4 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000
source link-address option (1), length 8 (1): 10:56:11:86:6e:92
0x0000: 1056 1186 6e92
15:21:12.463417 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28
15:21:13.487416 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has dns.google tell 10.0.0.91, length 28
15:21:13.546246 IP (tos 0x0, ttl 4, id 8382, offset 0, flags [DF], proto UDP (17), length 219)
169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 191
15:21:13.546273 IP (tos 0x0, ttl 4, id 8383, offset 0, flags [DF], proto UDP (17), length 223)
169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 195
15:21:13.546320 IP (tos 0x0, ttl 4, id 8384, offset 0, flags [DF], proto UDP (17), length 227)
169.254.100.1.50760 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 199
15:21:13.546419 IP (tos 0x0, ttl 4, id 8385, offset 0, flags [DF], proto UDP (17), length 220)
169.254.100.1.50759 > 239.255.255.250.ssdp: [udp sum ok] UDP, length 192
10 packets captured
10 packets received by filter
0 packets dropped by kernel
I am not understanding why, if the server is doing an ARP request, am I not getting a response? Is the issue on my server not knowing how to respond back to NIC0 with my ping request (where I am SSH'd into)? Is it the gateway being misconfigured? Do I need a NFT table/IP Table configured?
I am familiar with how to do this in RHEL 6.x, but not in RHEL 8 (configuration using IP route and IP tables was simpler I think?)
At the end of the day (for a broader picture) - I have Softphone clients to register to the Asterisk PBX on the internal/trusted network coming in over NIC0 (which works). They need to make phone calls to endpoints on the Internet, but only over NIC1 - and right now I cannot even ping to any location on the internet over the NIC1 interface.
Any help/guidance would be very much appreciated at this point - I am lost and desperate.
Edit/additional clarification: I have a RHEL 6.x server, with exact same physical connections and NICs that this does work on. I have tried to use the iptable and routing table from this Server #2 on Server #1 above and it will not work (I get booted when I turn the interface back up, and have to reboot the device to clear out any unsaved changes before I can get back in) I did use the iptables to nft translate function just as an FYI. I have plugged my Server #1 NIC1 into the known good modem/internet access port that Server #2 is using and still no change.
Server #2 interface configs:
DEVICE=eth0
BOOTPROTO=none
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
UUID="da71293d-4351-481e-a794-bc5850e29391"
IPADDR=10.38.149.243
DNS1=10.168.241.223
DOMAIN=comcast.net
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
#HWADDR=00:1C:23:CF:BC:E3
HWADDR=00:1c:23:cf:bc:e3
NETMASK=255.255.255.240
USERCTL=no
PEERDNS=yes
GATEWAY=10.38.149.241
DEVICE=eth1
BOOTPROTO=dhcp
HWADDR=00:1c:23:cf:bc:e5
NM_CONTROLLED=yes
ONBOOT=yes
DEFROUTE=yes
TYPE=Ethernet
UUID="78bc69cb-80ca-41d1-af9c-66703eb952d5"
USERCTL=no
PEERDNS=yes
IPV6INIT=no
Kernel Routing Table on Server #2
Destination | Gateway | Genmask | Flags | Metric | Ref | Use | Iface |
---|---|---|---|---|---|---|---|
0.0.0.0 | 10.0.0.1 | 255.255.255.255 | UGH | 0 | 0 | 0 | eth1 |
10.38.149.240 | 0.0.0.0 | 255.255.255.240 | U | 0 | 0 | 0 | eth0 |
10.0.0.0 | 0.0.0.0 | 255.255.255.0 | U | 0 | 0 | 0 | eth1 |
10.0.0.0 | 10.38.149.241 | 255.0.0.0 | UG | 0 | 0 | 0 | eth0 |
0.0.0.0 | 10.0.0.1 | 0.0.0.0 | UG | 0 | 0 | 0 | eth1 |
iptables -L on Server #2
Chain INPUT (policy ACCEPT)
target | prot | opt | source | destination | status? |
---|---|---|---|---|---|
DROP | all | -- | c-67-164-235-175.devivce1.mi.inetprovider.net | anywhere | |
DROP | all | -- | c-67-164-235-175.devivce1.mi.inetprovider.net | anywhere | |
ACCEPT | all | -- | anywhere | anywhere | |
ACCEPT | all | -- | anywhere | anywhere | state RELATED,ESTABLISHED |
ACCEPT | tcp | -- | anywhere | anywhere | tcp dpt:ssh |
ACCEPT | udp | -- | anywhere | anywhere | udp dpt:sip |
ACCEPT | udp | -- | anywhere | anywhere | udp dpts:ndmp:dnp |
DROP | all | -- | 106.0.0.0/8 | anywhere | |
DROP | all | -- | 106.0.0.0/8 | anywhere | |
DROP | all | -- | host-87-0-0-0.retail.blockeddomain.notus/8 | anywhere | |
DROP | all | -- | 113.0.0.0/8 | anywhere | |
DROP | all | -- | 117.0.0.0/8 | anywhere | |
DROP | all | -- | p5b000000.dip0.blockeddomain.notus/8 | anywhere |
Chain FORWARD (policy ACCEPT)
target | prot | opt | source | destination |
---|---|---|---|---|
ACCEPT | all | -- | anywhere | anywhere |
Chain OUTPUT (policy ACCEPT)
target | prot | opt | source | destination |
---|
tcpdump
output is completely useless as currently presented, and the kernel routing table isn't a table.iptables -L
is incomplete - it won't show if a particular rule applies to all network interfaces, or to a specific interface only. Useiptables -Lvn
to see the complete definition of each rule.