Routing localhost and only localhost over another network on linux?

Question

Problem description:

I have a rather complicated issue where I can't seem to figure out how to effectively tackle it; Suppose there's the following setup:

There is 1 linux machine, connected to 3 sets of networks, let's call them A, B, and C.

The linux machine can get through the internet through network A. It has a private ip address on this network, but NO NAT is enabled on it; e.g. any other machines that are routed through this network which have a public ip given out by it can connect to the internet through it, but the computer in question cannot do so.

Network B is a public ip network set; a set of several public networks. By default, packets from B are either routed back into B, and for all other destinations not in B are routed via A.

Network C is, like A, also a network where the machine has a private address. However, here NAT is enabled. Network C can be used to communicate with the internet, but devices on network B are not allowed to route through C, they have to route through A.

Currently, the linux machine can't talk to the internet. I want it to be able to do so (e.g. ping www.stackoverflow.com to not return timeout).

How to tell linux to use another gateway for its own traffic, and only its own traffic (source or destination == localhost)?

Important

I do not want to route all the traffic to the internet over just one network. The traffic coming/going to the internet needs to use two networks, depending on source: Everything from the linux machine over one network (C), everything that it routes (that isn't on one of its local ranges), it routes over the other one (A).

Tentative solution

First, I thought to just add 'from/to localhost' to the ip rule table, yet this is not allowed. What's the easiest way to allow this router to use another router on network C as its gateway, but only from traffic originating from the machine itself?

Essentially, what I'd want is something that would be equivalent to the example below, if that would actually work in the naïve, intuitive way:

0: from all lookup local
32000: from localhost lookup myroute
32100: from all lookup main 
32767: from all lookup default

Answer 1

I tested the following configuration with two network namespaces, and two veth pairs:

+-- Main --+
|          | 192.168.2.21/24
|    wlan0 | ------------------- ...
|          |
|          | 10.0.0.254/24   10.0.0.1/24 +-- NS0 --+
|   veth0a | --------------------------- | veth0b  |
|          |                             +---------+
|          |
|          | 10.0.1.254/24   10.0.1.1/24 +-- NS1 --+
|   veth1a | --------------------------- | veth1b  |
|          |                             +---------+
+----------+

The goal is to route all traffic from Main to wlan0 (network C), while routing everything from veth0 (network A) through veth1 (network B).

There's a normal default route on wlan0, acquired via DHCP:

$ ip route
default via 192.168.2.1 dev wlan0 
...

Forwarding is enabled:

echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward

We add a default route for veth1a in a new table, assuming 10.0.1.1 is the gateway:

sudo ip route add 10.0.1.0/24 table 250 dev veth1a src 10.0.1.254
sudo ip route add default table 250 dev veth1a via 10.0.1.1

All traffic from and to veth0a (10.0.0.0/24) uses this table:

sudo ip rule add from 10.0.0.0/24 table 250
sudo ip rule add to 10.0.0.0/24 table 250

If somebody behind veth1b (10.0.1.0/24) has the stupid idea to try to use us as a gateway, prevent forwarding to wlan0 and forward right back using the same table. This will usually generate ICMP error packets:

sudo ip rule add from 10.0.1.0/24 table 250
sudo ip rule add to 10.0.1.0/24 table 250

Adapt to your setup as necessary.

No localhost was even remotely mentioned anywhere.

Answer 2

Try ip ru add from 0.0.0.0

I tested on my router, locally originated connections match the rule, routed connections don't.