2

Asymmetric routing problems are driving me crazy!

I am trying to build a multi-homed server with 3 NICs. Each NIC is connected to 3 different subnets:

eth0: LAN connected, 10.99.72.38; function: Management traffic
eth1: LAN connected, 10.99.70.150; function: User traffic
eth2: WAN connected, 10.99.74.85; function: User traffic

I am using an Amazon Linux image (Linux 3.14.20-20.44.amzn1.x86_64 x86_64) but I can switch to CentOS if it really makes a difference.

The functionality I am trying to implement:

eth0: Management traffic

  • address: 10.99.72.38
  • I want this to be an 'isolated' interface, only accepting and replying to addresses in 10.0.0.0/8
  • Think of this as a 'ssh only from local LAN' NIC.
  • Can ONLY respond back through eth0
  • All other destinations are blocked, meaning, no other destinations can be reached through this interface
  • WILL NOT use eth1 or eth2 for reply traffic for traffic arriving on eth0.

eth1: User traffic to/from LAN

  • Address: 10.99.70.150
  • accepts any user traffic from the LAN to any destination
  • routes packets out through eth1 for traffic destined for 10.x.x.x.
  • routes packets out through eth2 for any other destination (default route)
  • will NOT route incoming packets through eth0

eth2: User traffic to/from WAN

  • address: 10.99.74.85
  • accepts any user traffic from eth1 and will send packets out eth2 for any destination not in 10.x.x.x
  • accepts reply packets on eth2 and routes through eth1 for any traffic destined for 10.x.x.x
  • will NOT route incoming packets through eth0

I have created a iproute2 table in rt_tables called 'mgmt' and added policy based routing rules with high priority to try and isolate this interface, but no matter what I try, the main routing table still seems to be called as eth0 is the default route. Problems include:

  • Incoming packets from eth1 are routed through eth0 (I don't want this!)
  • Incoming packets from eth2 destined for 10.x.x.x are routed through eth0 (I don't want this!)
  • When I delete the default eth0 route from main, I lose eth0 functionality completely, even with a route in the mgmt table, but eth1 then replies correctly.

Starting from the beginning, the unmodified route table (route -n):

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface  
0.0.0.0         10.99.72.1      0.0.0.0         UG    0      0        0 eth0  
0.0.0.0         10.99.70.1      0.0.0.0         UG    10001  0        0 eth1  
0.0.0.0         10.99.74.1      0.0.0.0         UG    10002  0        0 eth2  
10.99.70.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1  
10.99.72.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0  
10.99.74.0      0.0.0.0         255.255.255.0   U     0      0        0 eth2

Unmodified IP rules are:

0:      from all lookup local  
32766:  from all lookup main  
32767:  from all lookup default

Any assistance in correctly formulating the specific routing rules, tables, and routes, and showing me what the final routing tables and rules would look like would be greatly appreciated!

Improve this question
9
  • 1
    What is the netmask for the three interfaces? Are they connected to the same LAN?
    – Hauke Laging
    Commented Nov 17, 2014 at 6:28
  • Don't use the legacy route command, esp. if you're also using rules. Show ip ro sh output, ip ru sh output, and ip ro sh table mgmt output and the same for any other tables you may have created. You may also need iptables rules to prevent packets going out to eth0 that originated from eth1.
    – wurtel
    Commented Nov 17, 2014 at 12:20
  • @HaukeLaging: all three interfaces have a netmask of 255.255.255.0. Each interface is on a different subnet. Since this machine is being built on Amazon Web Services (AWS) within a single VPC, they share a common (virtual) VPC router.
    – SteveT
    Commented Nov 18, 2014 at 2:40
  • Will all connections to eth0 be from 10.99.72.x?
    – Hauke Laging
    Commented Nov 18, 2014 at 22:54
  • "accepts any user traffic from eth1" does not make sense to me. You must configure on eth1 what is to be accepted. There is no internal routing between interfaces. Do I understand it right: eth2 shall not accept new incoming connections at all? If you use ip rule then you should show us its output.
    – Hauke Laging
    Commented Nov 18, 2014 at 23:02

1 Answer 1

Reset to default
0

You should get rid of the default gateway setting for eth0 and eth1. How that is done depends on your distro.

You probably don't need policy routing. I guess what you want is best achieved with Netfilter / iptables.

iptables -I INPUT 1 -i eth0 ! -s 10.99.72.0/24 -j DROP
iptables -I FORWARD 1 -i eth0 -j DROP
iptables -I FORWARD 2 -o eth0 -j DROP

iptables -I INPUT 2 -i eth1 ! -s 10.0.0.0/8 -j DROP
iptables -I FORWARD 3 -i eth1 -j ACCEPT

iptables -I INPUT 3 -i eth2 -m conntrack --ctstate NEW -j DROP
iptables -I FORWARD 4 -i eth2 -j ACCEPT

That can probably be done without Netfilter but with policy routing instead.

Improve this answer
3
  • Thanks for this approach. I haven't spent much time using iptables, so I need to do a little homework to see if the above will do what I am hoping for. Could you please clarify your last sentence? ---"That can probably be done without Netfilter but with policy routing instead."
    – SteveT
    Commented Nov 20, 2014 at 2:21
  • @SteveT You talked about routing only thus I thought you might prefer a routing solution. You can set up a route of type unreachable, blackhole, throw, or prohibit. But as I said: In contrast to Netfilter that affects responses only.
    – Hauke Laging
    Commented Nov 20, 2014 at 2:39
  • I finally had a chance to go through the rules listed above...I like the approach of using this along with policy routing to solve the original problem. Thank you very much for your help.
    – SteveT
    Commented Nov 26, 2014 at 2:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .