SSH connection to Raspberry Pi using 2 internet and 2 routers

Question

I have two internet connection and both of them have public static IPs. Both of this internet connections are connected to their own separate routers and these two routers are connected to each other from LAN to LAN. One of the router has DHCP disabled.
First router has gateway of 192.168.0.1 and second router has gateway of 192.168.0.2. First Router with gateway 192.168.0.1 has DHCP enabled and Raspberry Pi is connected to this first router via LAN cable and WiFi is disabled in Raspberry Pi. In both of the router, I have done port forwarding for port 22 pointing to Raspberry Pi local IP.
The problem is that I am only able to connect to Raspberry Pi via SSH from First router's public IP from the internet. I am not able to connect from second router public IP even though both of them have port forwarding enabled for port 22 pointing to Raspberry Pi local IP address. My raspberry PI do not have an static IP configuration but I have done IP reservation on router so Raspberry Pi will always receive same IP address.
What I noticed is that this depends on what gateway Raspberry Pi is using. I ran the route -n on my Raspberry Pi and saw that it was pointing to First router's 192.168.0.1 gateway hence I was able to connect it from First router's public static IP. If I change this gateway to 192.168.0.2, then I can connect from second router's public static IP but not from the first one.
What configuration change that I can do to make sure my Raspberry Pi can be connected from either of the public static IP via SSH from internet? I need this to make sure I will at least have one way to connect when one of the internet is down.

Answer 1

You can make use of ct mark in nftables:

table ip alt_gateway {
        chain input {
                type filter hook input priority mangle; policy accept;
                ether saddr YOUR_SECOND_ROUTER_MAC ct mark set 0x00616c74
        }

        chain output {
                type route hook output priority mangle; policy accept;
                ct mark 0x00616c74 meta mark set 0x00616c74
        }
}

ether saddr YOUR_SECOND_ROUTER_MAC matches traffics that were forwarded by your second router (i.e. those that have its MAC Address as the source L2 address). Technically the rule also covers traffics originated from the router itself, so you may want to either add ip saddr != 192.168.0.2 before ct mark set or a direct route for 192.168.0.2 in the alternate route table below.

The mark 0x00616c74 is arbitary, and the meta mark to set can be different from the ct mark (on the other hand, you can also have something like meta mark set ct mark, which is practically equivalent to the above).

The meta mark can then be matched against an ip rule (that looks up an alternate route table for default route, where you can have 192.168.0.2 as the nexthop / gateway) with fwmark:

ip route add default via 192.168.0.2 table 123
ip rule add iif lo fwmark 0x00616c74 lookup 123

iif lo means that the rule only applies on traffics originated from this host. It's sort of optional and is meant to correspond with the fact that the above table does not deal with traffics forwarded by this host anyway, that is, you'll need additional chain/rules if this host also serves as some kind of router / gateway and that this trick is also needed for traffics from / to the hosts that it serves.

The table number 123 is also arbitrary.