UPDATE: This happened when using docker containers inside the network. I'm not sure that matters, but it might be affecting the topology somehow...
I'm struggling to understand a behavior I can't explain in a sample network. Basically I want to be able to switch some of my servers by redirecting to a different server (so that I can point my applications to some database server for example and then switch the database connections to another server without changing the applications). I'm probably going to use something like rinetd for that, but I'm curious why some iptables rules are not working as I would expect.
The sample network had 3 servers: gw (192.168.0.1/24), s1 (192.168.0.2/24) and s2 (192.168.0.3/24). gw is the default gateway for s1 and s2. Then I set up in gw's PREROUTING in the nat table to redirect traffic to 192.168.1.1:5432 (which is outside the main network, which means it goes to the gateway) to s2:5432 (192.168.0.2). The 192.168.1.1 address doesn't have to exist at all, it could be any address outside of the routed internal network (192.168.0.0/24).
Now, I understand why I can't access 192.168.1.1:5432 from the gw gateway since the traffic doesn't go through NAT PREROUTING (where DNAT happens) and I know how to make it possible by adding a DNAT rule to nat's OUTPUT chain and an SNAT rule to POSTROUTING. As expected, pointing to 192.168.1.1:5432 will redirect to 192.168.0.2:5432 when accessed from s1. Now, what I don't understand is why the connection isn't accepted or refused when I try to connect to 192.168.1.1:5432 from s2. I think the connection should be redirected back to s2 (192.168.0.2).
Performing SNAT or MASQUERADE in nat's POSTROUTING doesn't make any difference either in my tests. Why doesn't redirecting back doesn't work with iptables?
UPDATE: I looked for some help in #Netfilter channel on Freenode and it turns out this should work as confirmed by "evilman_work" when replicated in a lab environment. Also, we noticed that it worked in my server too if the interface was put in promiscous mode. We were not able to figure out the reason for that, so if you have any ideas I'd love to hear them. This is not much of an issue because this was a test environment using Docker while the host was the gateway. I want to use this in real VM's so I suspect I won't have troubles in that scenario. Maybe this was caused by something related to Docker...
