VPS Port forwarding without SNAT/Masquerade using source-based routing?

Question

I'm trying to test out using a VPS to port forward 80,443 into a local web server over a VPN tunnel while keeping the original source public IP intact. The main problem here is routing, as the edge routing device will effectively be seeing 2 interfaces with public IPs coming in (the true WAN, and the VPN interface).

Obviously, the simple solution is to just SNAT traffic going across the vpn tunnel, but I would like insight into the true source of the web traffic, plus the more NAT I do, the more overhead I introduce into the system. In addition I am avoiding putting any reverse proxy on the VPS as I refuse to decrypt the web traffic at any time while traversing the VPS.

Now on to the testing so far: IPs used: Web SRV: 192.168.0.2, VPS Wireguard Addr: 192.168.200.1, Router Wireguard Addr: 192.168.200.2.

I have a virtual test network set up with a few Debian 11 boxes serving as the VPS server, the local router, and a web server.

On the VPS, I DNAT traffic coming in on the interface on port 80,443:

iptables -t nat -A PREROUTING -i eth0 -p tcp -m multiport –dports 80,443 -j DNAT –to-destination 192.168.0.2

I already set sysctl.conf to enable ipv4 forwarding. I have a static route on the VPS telling where to get to the 192.168.0.2:

ip route add 192.168.0.2/32 via 192.168.200.2 dev wg0

On the local router, I found some reply based rules online based on interfaces:

echo 200 vpsrt >> /etc/iproute2/rt_tables

ip rule add from 192.168.200.2 table vpsrt prio 1
ip route add default via 192.168.200.1 dev wg0 table vpsrt

Since the VPN tunnel is on the router, the web server already has its default gateway configured, so return traffic between the local router and web server doesn't change at all.

wg0.conf on the VPS (acting as server):

[Interface]
PrivateKey=*
Address=192.168.200.1/30
ListenPort=50000

[Peer]
PublicKey=*
AllowedIPs=192.168.200.2/32,192.168.0.2/32

wg0.conf on the router (acting as the client):

[Interface]
Address=192.168.200.2/32
PrivateKey=*

[Peer]
Endpoint=PUBIPVPS:50000
PublicKey=*
AllowedIPs=172.20.200.0/30

I've verified the Wireguard tunnel works. I can ping 192.168.200.1 and .2 on each device respectively, and wg show says handshake is complete and bidirectional data is flowing.

Doing some PCAPs, traffic going through the VPS is correct. Tcpdump on wg0 on the VPS shows packets addressed to 192.168.0.2 port 443. WG Show on the VPS says its transmitting 1.3Kb, but the WG Show on the router says its only received a 500 bytes (the test ping). Tcpdump on the routers WAN interface shows the Wireguard packets are reaching the router. But the Tcpdump on the routers wg0 shows its not receiving any packets.

I am pretty sure the problem is Wireguard is silently dropping packets with a source IP that is not configured in the AllowedIPs (on the router). But the problem is I can't put 0.0.0.0/0 on routers wg0.conf because it would start using the tunnel as the default gateway for the entire network!

Is it impossible to do source based routing with Wireguard as a non-default gateway? Are there any potential work-arounds using another tunneling software? I really like the speed of Wireguard, but it appears without SNAT/Masquerade it becomes extremely limited.

Update: just found this article: https://techoverflow.net/2021/07/09/what-does-wireguard-allowedips-actually-do/ It explained that wireguards "firewalling" and routing are tied together using AllowedIPs.

A little more testing, I can simply put AllowedIPs on the router to 0.0.0.0/0 and add a postup rule in wg0.conf to remove the default gateway route and instead add a static route to the tunnel IPs only. (Unfortunate that wireguard chose to roll "firewalling" and routing into a single option)

A Tcpdump on the router's wg0 interface shows the packets are now traversing the tunnel. Tcpdump on the routers LAN interface shows the packets are transmitted out, and the web server is responding. But now the issue is those interface source-based routing rules appear to not be working as the web servers response is not being passed into the wireguard tunnel.

A Tcpdump on the routers WAN interface shows the web servers syn-acks being pushed out that gateway. So now my problem lies in the source-based routing rules I mentioned above. Source-based routing is out of my knowledge area, so any help with those is much appreciated.

Answer 1

You're 90% of the way there. When set up your custom vpsrt routing table on the router:

echo 200 vpsrt >> /etc/iproute2/rt_tables

ip rule add from 192.168.200.2 table vpsrt prio 1
ip route add default via 192.168.200.1 dev wg0 table vpsrt

You should use the web server's IP address, not the router's IP address, for the second line:

ip rule add from 192.168.0.2 table vpsrt prio 1

This will direct your router to send all traffic from your web server to the WireGuard tunnel. As long as you have AllowedIPs=0.0.0.0/0 as part of the peer entry for the VPS server in your router's WireGuard config, the WireGuard tunnel will accept that traffic, and send it back to the VPS server.

Also, you don't need the third line from above if you specify the table in the router's WireGuard config:

[Interface]
Address=192.168.200.2/32
PrivateKey=*
Table=vpsrt

[Peer]
Endpoint=PUBIPVPS:50000
PublicKey=*
AllowedIPs=0.0.0.0/0

With that setting, wg-quick will add the default route to your custom vpsrt table instead of messing with your default routing globally.

And also, if you don't want to route all your traffic from the web server to the VPS -- only responses from TCP port 80 and 443 -- you can use these rules in place of the second line above:

ip rule add from 192.168.0.2 ipproto tcp sport 80 table vpsrt priority 1
ip rule add from 192.168.0.2 ipproto tcp sport 443 table vpsrt priority 2

Answer 2

You could look into using a VRF (Virtual Routing and Forwarding). A VRF is used to configure a separate routing table with it's own default gateway. You can connect your wireguard interface and your web server to the VRF and the traffic will be separate from the currently existing connection.

Here is a great example on using VRF's: https://interpip.es/linux/creating-a-vrf-and-running-services-inside-it-on-linux/