Cannot reach web server in vm from bare metal using DNS after iptables rules forward packets to virtual bridge

Question

This is a simplified description of the development environment at work and I am trying to configure it to facilitate developers using it.

Topology:

The topology of my setup is illustrated below. Description:

1. I have 2 virtual machines (vm1 and vm2) running on my bare metal server bm1.
2. I have a web server (listening on port 8080) running on vm1.
3. I have a public facing router i.e. a public IP with a DNS name, say, www.mysite.com (IP: 10.1.2.3)
4. The public facing router has been configured to forward any traffic coming for port 9080 to bare metal server bm1 (192.168.2.51)

Requirement:

I need to be able to reach the web server (vm1-ws1) using the DNS name from both the bare metal server (bm1) and the virtual machines (vm1, vm2) over port 9080 i.e. curl http://www.mysite.com:9080 should work from both the bare metal server as well as the VMs.

Solution attempted:

1. I verified that the web serer is reachable from both the bare metal and the VMs using the private IP address i.e. curl http://192.168.121.29:8080 works from both the bare metal server and the VMs.
2. I configured IP table rules as:

iptables -t nat -I PREROUTING -p tcp --dport 9080 -j DNAT --to 192.168.121.29:8080 
iptables -I FORWARD -d  192.168.121.29 -o virbr1 -j ACCEPT

3. After adding the IP table rules, I can access the web server from the VMs but not from the bare metal server. In other words, curl http://www.mysite.com:9080 works from the VMs but not from the bare metal. It hangs and then times out.

Kindly help me understand where I might be going wrong with the iptables rules config.

Answer 1

Traffics originated from the host itself is governed by the OUTPUT chain, so you need the following rule additionally:

iptables -t nat -I OUTPUT -p tcp --dport 9080 -j DNAT --to-destination 192.168.121.29:8080

(For the record, since you don't have a -d match in the DNAT rule(s), any TCP traffics with destination port of 9080 would be redirected to 192.168.121.29:8080, i.e., regardless of the destination IP address.)

You may need the following rule additionally, depending on whether you already have a MASQUERADE rule that applies on all traffics that egress via virbr1 (well, more precisely, as long as it cover traffics with the preferred source IP in your route for 10.1.2.3, i.e., most likely the default route):

iptables -t nat -I POSTROUTING -p tcp --dport 8080 -d 192.168.121.29 -j MASQUERADE

Theoretically a MASQUERADE rule should not be needed when the host is the VM's default gateway, that is, usually it doesn't matter whether the source IP used is the one configured on the interface the traffics egress via. However, since DNAT in OUTPUT is really a "hacky" thing, that is, unlike the case in PREROUTING, it is done after a routing decision has been made instead of before. Therefore, apparently in reality, you'll need to make sure the traffics have their source IP NAT'd as well.

It might also be worth mentioning that, in case you have filtering at OUTPUT, the -o virbr0 match would be irrelevant for the traffics even when they egress via the interface after the "hacky" DNAT. It is probably either because the filtering is performed before it, OR, the egress interface in the "states" of the traffics hasn't been "rewritten" yet, because maybe the "re-routing" is done after the filtering. (I don't know much about the "internals".)

Hence you'll need to either have an accept rule for e.g. -p tcp --dport 8080 -d 192.168.121.29, or one for (all) traffics that egress via the "original" interface (e.g., -o eno1).