2

Following a short/simplified diagram of my setup (sorry for the title):

Internet ----- eth0 (1.2.3.4) --- br0 (10.0.0.1)
                                      |
                      +---------------+---------------+
                      |                               |
                veth0 (10.0.0.2)                veth1 (10.0.0.3)
                    httpd                            app

I forward external traffic via the usual DNAT/SNAT (using nftables) from eth0 to the service that is appropriate (e.g. to httpd, if someone talks to port 443 on eth0). I can also talk between containers (e.g. from app to httpd), or between containers and the "host" (via br0), if they use the internal IP addresses (10.0.0.X). This all works without having to turn on route_localnet.

What I can't seem to make work is, if for example the container app is trying to talk to httpd using the external address (1.2.3.4). So e.g. if app is sending a request to 1.2.3.4:443. That it is using 1.2.3.4 and not the internal 10.0.0.2 is beyond my control (I might be able to fix some cases by adding a split dns view, but certainly not all, so it's not really worth it).

When I try to talk from container app to httpd using the external address, for example using socat (socat -4 - TCP4:1.2.3.4:443), I get EHOSTUNREACH ("No route to host").

I tried to add a separate DNAT for this case, e.g.: nft add rule ip contBr prerouting iif "br0" ip daddr 1.2.3.4 tcp dport { 80, 443 } dnat to 10.0.0.2. The idea was that as it only changes the destination address before routing, it will know where to send it during routing, and the source address keeps being 10.0.0.3, so returning the answer should be fine. Doesn't work though. Now, when trying to talk to httpd from app, I get ETIMEDOUT ("Connection timed out") - which shouldn't be any filter from me, since I reject everything I don't want, and don't just drop it.

Not sure what exactly to search for on the internet, or here.

Improve this question
2
  • 1
    I think you need to set up hairpin NAT for that particular scenario.
    – Preston Maness
    Commented Apr 17, 2022 at 20:47
  • 1
    That's exactly what it is. Thanks, I didn't knew the term, but with it I quickly found a solution and the reason why mine didn't work.
    – mageta
    Commented Apr 18, 2022 at 12:14

1 Answer 1

Reset to default
2

After a helpful comment from Preston to my question I found out that the term/technique I'm looking for is called Hairpinning/Hairpin NAT (https://en.wikipedia.org/wiki/Hairpinning, https://newspaint.wordpress.com/2018/09/13/hairpin-for-lxc-containers-using-iptables/).

The reason why my singular rule nft add rule ip contBr prerouting iif "br0" ip daddr 1.2.3.4 tcp dport { 80, 443 } dnat to 10.0.0.2 didn't work is that while the destination address is changed to the correct one, and the answer will also go back to the expected source (app, 10.0.0.3); the application running in app will expect an answer from the host with address 1.2.3.4, not 10.0.0.2 - but that will be the case because of the DNAT.

The solution is to mark the connection, and when sending an answer back to app, we either need to add a SNAT, or MASQUERADE; so that the answer package gets the expected source address 1.2.3.4.

What works for me is:

nft add rule ip contBr prerouting iif "br0" ip daddr 1.2.3.4 tcp dport { 80, 443 } ct mark set 0x61687269
nft add rule ip contBr prerouting iif "br0" ip daddr 1.2.3.4 tcp dport { 80, 443 } dnat to 10.0.0.2
nft add rule ip contBr postrouting oif "br0" ct mark 0x61687269 snat to 1.2.3.4

                   /------------------------------------\
                  /                                      \
  RESP 10.0.0.3  /              REQ  1.2.3.4:443          \
  SNAT  1.2.3.4 /               SRC 10.0.0.3               \
                |  /------------------------------------\   \
                |  |                                     \  |
Internet ----- eth0 (1.2.3.4) --- br0 (10.0.0.1)         |  |
                |  |                  |                  |  |
                |  | DNAT 10.0.0.2    |                  |  |
                |  |  SRC 10.0.0.3    |                  |  |
                |  |                  |                  |  |
  RESP 10.0.0.3 |  |  +---------------+---------------+  |  |
   SRC 10.0.0.2 |  |  |                               |  |  |
                |  |  |                               |  |  |
                |  |  |                               |  |  |
                ^  v  |                               |  ^  v
                veth0 (10.0.0.2)                veth1 (10.0.0.3)
                    httpd                            app
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .