How to make the incoming and outgoing interfaces the same by packet?

Question

My server has two interfaces.

First interface is directly connected to local ISP and second interface is connected to other region that connected with GRE tunnel.

So it looks like:

┌──────────────────┐ ────────────────── ISP 1
│ Server (x.x.x.1) │ 
└──────────────────┘ ─── GRE Tunnel ─── ISP 2

Each ISP, we announce same prefix(anycast) so each interface has same IP address. (x.x.x.1) Our default route is ISP 1, so currently, all traffic is routed to ISP 1(Interface 1), regardless of incoming interface is Interface 1 or Interface 2.
I want to configure it so that packets coming in from Interface 1 are sent out through Interface 2, and packets coming in from Interface 2 are sent out through Interface 2. (Like sticky session in load balancer)

Our operating system is CentOS 7. Is there any way to configure this? I tried to import route table from ISP 2 and override route to ISP 2 that matches ISP 2 route table, but it failed, because ISP 2's route table is asymmetric. (So I sent packet to ISP 1 but it returns from ISP 2.)

Answer 1

For outgoing replies to incoming requests, you can set up policy routing:

1. Import default routes from each ISP into separate kernel routing tables.

   For example, in Bird:

   ipv4 table ispone4;
   
   protocol bgp 'ispone' {
       ipv4 {
           table ispone4;
           igp table master4;
           [...]
       };
   };
   
   protocol kernel 'krt_ispone4' {
       ipv4 {
           table ispone4;
       };
       kernel table 10;
   };
   

2. Create policy routing rules (e.g. via ip rule) to use the kernel tables depending on the packet mark (aka fwmark):

   ip -4 rule add fwmark 0x10 lookup 10
   

   For example, in systemd-networkd:

   [RoutingPolicyRule]
   Family=both
   Priority=501
   FirewallMark=82
   Table=82
   

3. Create iptables or nftables rules that would mark incoming packets depending on the interface they arrive on, and would store the packet mark as conntrack mark for incoming packets, and restore it from conntrack for outgoing ones.

However, this will do absolutely nothing for incoming replies to the requests that your server has made on its own (e.g. DNS queries). The only reliable way to avoid asymmetric routing there is to not advertise the same prefix through both ISPs.