0

Overview

I am trying to deal with the following scenario:

+------+                 (Internet)  50.a.b.c +---------+
| edge + eth0 <-----------------------------> + outpost |
+--+---+  25.u.v.w       (Secured by IPSec)   +---------+
   | enp                                      ^
   |  10.0.0.1                                |
   |                                          |
   | enp                                      |
   |  10.0.0.2                                |
+--+---+                                      |
| node + eth0 <----- XXX (FORBIDDEN) XXX -----+
+------+  160.r.s.t

There is an edge host with public IP 25.u.v.w on interface eth0 and a private IP 10.0.0.1 in LAN 10.0.0.0/24 on interface enp.

Likewise, there's a node host in the same LAN 10.0.0.0/24 which has a distinct public IP 160.r.s.t on eth0 and 10.0.0.2 for the LAN.

edge maintains an IPSec secured link between edge and outpost.

outpost is only reachable via public IP 50.a.b.c. outpost must never be reachable via internet, only IPSec connections from edge (iptables on outpost ensure that constraint).

node is not the single node in the network, there is a multitude of them. I only depict one node to keep things simple (should work the same for all nodes).

All hosts run Ubuntu 22.04.

Task

How can I make sure that node accesses outpost by routing its packets via edge over enp and then to outpost via eth0?

About edge

Routing table of edge

With comments below

# ip r
default via 172.16.1.1 dev eth0 proto dhcp src 25.u.v.w metric 100
# Default route intended for unspecific internet access. 172.16.1.1 seems to be the VM bridge of the cloud service provider.
10.0.0.0/16 via 10.0.0.254 dev enp
# access to LAN via gateway
10.0.0.254 dev enp scope link
# no idea why this is needed
172.16.1.1 dev eth0 proto dhcp scope link src 25.u.v.w metric 100
# Seems to define the next hop to the VM bridge gateway

Interfaces of edge

Only relevant entries shown:

# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 25.u.v.w/32 metric 100 scope global dynamic eth0
       valid_lft 57218sec preferred_lft 57218sec
    inet6 e:f:g:h::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:2ff:fea1:8068/64 scope link
       valid_lft forever preferred_lft forever
3: enp: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/32 brd 10.0.0.1 scope global dynamic enp
       valid_lft 86280sec preferred_lft 86280sec
    inet6 fe80::8400:ff:fe61:cc98/64 scope link
       valid_lft forever preferred_lft forever

About node

Pristine routing table of node

The routing table before change attempt

# ip r
default via 172.16.1.1 dev eth0 proto dhcp src 160.r.s.t metric 100
# Default route intended for unspecific internet access. 172.16.1.1 seems to be the VM bridge of the cloud service provider.
10.0.0.0/16 via 10.0.0.254 dev enp
# access to LAN via gateway
10.0.0.254 dev enp scope link
# no idea why this is needed
172.16.1.1 dev eth0 proto dhcp scope link src 160.r.s.t metric 100
# Seems to define the next hop to the VM bridge gateway

As you can see, node is configured equally to edge in that it route internet traffic via the default gateway (as intended).

Interfaces of node

# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether mm:mm:mm:mm:mm:mm brd ff:ff:ff:ff:ff:ff
    inet 160.r.s.t/32 metric 100 scope global dynamic eth0
       valid_lft 55654sec preferred_lft 55654sec
    inet6 i:j:k:l::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:2ff:fea1:91a3/64 scope link
       valid_lft forever preferred_lft forever
3: enp: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:00:00:61:d1:48 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/32 brd 10.0.0.2 scope global dynamic enp
       valid_lft 52529sec preferred_lft 52529sec
    inet6 fe80::8400:ff:fe61:d148/64 scope link
       valid_lft forever preferred_lft forever

Attempt to route traffic to outpost from node over edge

I inserted a seemingly harmless rule into node's routing table:

ip r a 50.a.b.c via 10.0.0.1 dev enp proto static onlink

For any traffic destined for outpost, send it to edge. edge's routing table will perform the actual routing to outpost.

However, this rule made node completely inaccessible from edge as well as its public IP 160.r.s.t.

I don't understand why this rule affects incoming traffic. It should affect outgoing traffic targeted at outpost.

Do I need SNAT Postrouting?

After doing some more research it seems I might need a rule on edge like

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j SNAT --to 25.u.v.w

I haven't yet applied this rule as it does not resolve the actual issue of making node inaccessible on the routing table change denoted above.

How would the correct routing table look like for node?

Referring to the task described above, how would a proper routing table for node look like?

Constraints:

  • Access to outpost must be routed to edge (via LAN 10.0.0.0/24), from there NATed with public source IP 25.u.v.w to outpost
  • Access to any other internet host must go via the default gateway and use node's public IP 160.r.s.t.
  • Access to nodes within LAN 10.0.0.0/24 must be routed locally.

Do I have to consider something else in order to make routing to outpost work?

Can someone with more networking experience shed light on this issue?

Improve this question
0

1 Answer 1

Reset to default
0

Finally I found the culprit. The iptables rules and routing table entries were essentially correct. However, the cloud-provider had an opaque gateway 10.0.0.254 in the private LAN.

I was missing an entry in the cloud-provider's web configuration that redirected all traffic from the opaque gateway 10.0.0.254 to edge on 10.0.0.1.

Once I did that, the rule

ip r a 50.a.b.c via 10.0.0.1 dev enp proto static onlink

still made node immediately unreachable. One still had to funnel the traffic over the opaque gateway which redirected it back to edge somehow.

So the correct route was

ip r a 50.a.b.c via 10.0.0.254 dev enp proto static onlink

As this solution is cloud-provider specific and does not answer the actual question, I will not mark my own answer as a solution.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .