0

This is a supplement for iptables SNAT for UDP rule is only applied to some traffic

In fact, I have three machines:

  • the client
  • the target server
  • the relay server

What's my motivation for doing this?

The upload channel from the client to the target server is very unstable, but the download channel from the target server to the client is very good.

Therefore, I want the relay server to relay the upload traffic from the client to the target server, but not to relay the download traffic that the target server responses to the client. Instead, the target server responses directly to the client.

Here is the traffic flow

What I do

In order to achieve this requirement.

  • On the relay server, I set up DNAT rules for request packets from the client, to translate the destination address of these packets to the target server, so that the target server thinks that the request packets are directly sent to it by the client.

  • On the target server, I add SNAT rules for the response packets , to translate the source address of the response packets to the relay server, in order to make the client think that the response packets come from the relay server.

Configuration:

  • IP Addresses

    • Relay Server: 192.168.169.129
    • Client: 192.168.169.131
    • Target Server: 192.168.169.132

  • General purpose

    • Client sends request traffic to RelayServer:10080
    • RelayServer:10080 relay request traffic from client to TargetServer:80 (Achieved by DNAT rules on the relay server)
    • TargetServer:80 response client as if it is RelayServer:10080 (Response packets failed to apply SNAT rules on the target server)
    • Client receives response traffic from RelayServer:10080, which is actually from TargetServer:80

  • iptables rules on the relay server:

    # DNAT-RULES                     relay-server       protocol            relay-port                target-server:target:port 
iptables -t nat -A PREROUTING -d 192.168.169.129/32 -p tcp -m tcp --dport 10080 -j DNAT --to-destination 192.168.169.132:80
iptables -t nat -A PREROUTING -d 192.168.169.129/32 -p udp -m udp --dport 10080 -j DNAT --to-destination 192.168.169.132:80

  • iptables rules on the target server:

    # NOTRACK-RULES                  target-server      protocol            target-port    
iptables -t raw -A PREROUTING -d 192.168.169.132/32 -p udp -m udp --dport 80 -j NOTRACK
iptables -t raw -A PREROUTING -d 192.168.169.132/32 -p tcp -m tcp --dport 80 -j NOTRACK

# SNAT-RULES                      target-server      protocol            target-port            relay-server:relay:port 
iptables -t nat -A POSTROUTING -s 192.168.169.132/32 -p udp -m udp --sport 80 -j SNAT --to-source 192.168.169.129:10080
iptables -t nat -A POSTROUTING -s 192.168.169.132/32 -p tcp -m tcp --sport 80 -j SNAT --to-source 192.168.169.129:10080

Works perfectly with UDP protocol

Problem with TCP protocol

  • The request packets can reach the target server successfully, which means that the DNAT rules on the relay server do work.

  • And the target server can give responses for request packets, but the source address of the response packets can't apply the SNAT rules, which means that the client do receive response packets, however, the source address of response packets is the target server instead of the relay server. So the client will reject those response packets.

  • Further testing showed that, the TCP packets sent directly from the target server to the client do apply the SNAT rules, but the TCP response packets don't.

What is the problem?

  • Is this still a problem with conntrack? I have add NOTRACK and SNAT rules for TCP as per UDP's configuration.
  • Is there a solution for my requirement?

Update

SNAT works on the target server, however client reset tcp connection

After replacing iptables with nftables on the target server, SNAT works for both request and reponse packets on the target server.

However, the client will reply a RESET packet to the SYNC packet from the target server. The SYNC packet is the response packet to the request SYNC packet from the client, and it is translated by SNAT rules on the target server so that the client should think it is from the relay server.

New Problem

  • Why do the client reset connection after receiving response SYNC packet from the target?

  • Can client setup a TCP connection to the target server by the relay server in this way?

Update Again

Replace iptables with nftables on the relay server, it works as what I expect

After capturing packets, I know that the problem is that the relay server RESET TCP connection once it receives the ACK packet from the client, which is the the third packet of TCP protocol.

By the DNAT rules, the relay server should translate the address of the ACK packet to the target server, then send it to the target server. I don't know why the ACK packet failed to apply the DNAT rules.

When I replace iptables with nftables on the relay server, it works as what I expect. Maybe iptables is too hard to use, I don't know the problem with iptables.

Anyway I ended up achieving my needs with nftables!

Improve this question
8
  • Can you show packet captures for packets on the wire at each point in the flow for a single connection attempt? (i.e. wire from client to relay, wire from relay to target, and wire from target to client). My off-the-top-of-my-head guess is that sequence numbers aren't matching, but packets should make that pretty clear (you should be able to see things from the client's perspective, which will probably explain the client's behavior)
    – Slartibartfast
    Commented Mar 23, 2022 at 18:15
  • I have solved this problem with nftables. Maybe iptables is too difficult to use! I checked the packets and the sequence did match. The problem is that the relay server RESET TCP connection after the client replied the third packet of TCP protocol (ACK to the target server). By the DNAT rules, the relay server should translate the address of the ACK packet to the target. I don't know why the ACK packet failed to apply the DNAT rules. By nftables, I succesfully connect the client and the target server with TCP protocol.
    – Peiyuan
    Commented Mar 23, 2022 at 18:33
  • If the relay server was considering that ACK as part of a connection (and the fact that it responded to a packet with a RST implies that it was), note that it didn't see the corresponding SYN/ACK from the target server (different path), so in that context it was correct to do so.
    – Slartibartfast
    Commented Mar 23, 2022 at 19:36
  • I wonder if the relay server should RESET that ACK packet if there are DNAT rules for that ACK packet. The fact is when I switch to nftables, it works as I expect.
    – Peiyuan
    Commented Mar 23, 2022 at 19:41
  • I posted an answer, maybe you can take a look at the packets in the end, it seems that everything is ok.
    – Peiyuan
    Commented Mar 23, 2022 at 19:49

1 Answer 1

Reset to default
1

Solved by replacing iptables with nftables

Note: It cannot be implemented in the real-world network due to the source address verification.

Purpose:

the traffic flow

  • Client sends request traffic to RelayServer:10080
  • RelayServer:10080 relay request traffic from client to TargetServer:10080 (Achieved by DNAT rules on the relay server)
  • TargetServer:10080 response client as if it is RelayServer:10080 (Achieved by SNAT rules on the target server)
  • Client receives response traffic from RelayServer:10080, which is actually from TargetServer:10080
  • it works for both TCP and UDP protocol

Environment

  • Devices: 4 Vmware Machines with bridged network

    • Router
    • Client
    • Relay Server
    • Target Server

  • OS: Alpine Linux 3.15.1

Devices IP Addresses

  • Client
    • IP: 192.168.10.2/24
    • Gatewat: 192.168.10.1
    • MAC: 00:0c:29:06:c7:7e
  • Relay Server
    • IP: 192.168.20.2/24
    • Gatewat: 192.168.20.1
    • MAC: 00:0c:29:5b:89:3e
  • Target Server
    • IP: 192.168.30.2/24
    • Gatewat: 192.168.30.1
    • MAC: 00:0c:29:15:da:6a

Router Config

Network Interfaces

  • eth0
    • MAC: 00:0c:29:65:3c:a3
    • IP: 192.168.10.1/24
  • eth1
    • MAC: 00:0c:29:65:3c:ad
    • IP: 192.168.20.1/24
  • eth2
    • MAC: 00:0c:29:65:3c:b7
    • IP: 192.168.30.1/24

Enable ipv4_forward

echo 1 > /proc/sys/net/ipv4/ip_forward

Disable source address verification for all interfaces

For linux-based router, refer to kernel sysctl parameter rp_filter

echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter

Relay/Target Server Config

Both

  • uninstall iptables & install nftables
  • enable ipv4_forward option for the kernel 
    echo 1 > /proc/sys/net/ipv4/ip_forward

nftables rules for the relay server

table ip route {
      chain prerouting {
              type filter hook prerouting priority dstnat + 1; policy accept;
              ip daddr 192.168.20.2 udp dport 10080 ip daddr set 192.168.30.2
              ip daddr 192.168.20.2 tcp dport 10080 ip daddr set 192.168.30.2
      }
}

nftables rules for the target server:

table ip raw {
      chain prerouting {
              type filter hook prerouting priority raw; policy accept;
              ip daddr 192.168.30.2 udp dport 10080 notrack return
              ip daddr 192.168.30.2 tcp dport 10080 notrack return
      }
}
table ip route {
      chain output {
              type filter hook postrouting priority srcnat + 1; policy accept;
              ip saddr 192.168.30.2 udp sport 10080 ip saddr set 192.168.20.2
              ip saddr 192.168.30.2 tcp sport 10080 ip saddr set 192.168.20.2
      }
}

How to test

  • On the target server, start a tcp/udp server

    nc -s 192.168.30.2 -l -p 10080 # for tcp
nc -s 192.168.30.2 -l -u -p 10080 # for udp

  • On the client, start a tcp/udp client to server

    nc -s 192.168.10.2 -p 12345 192.168.20.2 10080 # for tcp
nc -s 192.168.10.2 -u -p 12345 192.168.20.2 10080 # for udp

My Testing

Steps

  1. The client establishes a TCP connection with the server
  2. The client sends "Hello" to the server
  3. The server replies "World" to the client
  4. The client terminates the TCP connection

Packets flow captured by the Router

router:~# tcpdump '(dst 192.168.10.2 && dst port 12345) || (src 192.168.10.2 && src port 12345)' -e
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:33:45.712751 00:0c:29:06:c7:7e (oui Unknown) > 00:0c:29:65:3c:a3 (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [S], seq 4018363570, win 64240, options [mss 1460,sackOK,TS val 1510067563 ecr 0,nop,wscale 7], length 0
17:33:45.712870 00:0c:29:65:3c:ad (oui Unknown) > 00:0c:29:5b:89:3e (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [S], seq 4018363570, win 64240, options [mss 1460,sackOK,TS val 1510067563 ecr 0,nop,wscale 7], length 0
17:33:45.713459 00:0c:29:5b:89:3e (oui Unknown) > 00:0c:29:65:3c:ad (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [S], seq 4018363570, win 64240, options [mss 1460,sackOK,TS val 1510067563 ecr 0,nop,wscale 7], length 0
17:33:45.713460 00:0c:29:65:3c:b7 (oui Unknown) > 00:0c:29:15:da:6a (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [S], seq 4018363570, win 64240, options [mss 1460,sackOK,TS val 1510067563 ecr 0,nop,wscale 7], length 0
17:33:45.713557 00:0c:29:15:da:6a (oui Unknown) > 00:0c:29:65:3c:b7 (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [S.], seq 4008910075, ack 4018363571, win 65160, options [mss 1460,sackOK,TS val 3299425652 ecr 1510067563,nop,wscale 7], length 0
17:33:45.713572 00:0c:29:65:3c:a3 (oui Unknown) > 00:0c:29:06:c7:7e (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [S.], seq 4008910075, ack 4018363571, win 65160, options [mss 1460,sackOK,TS val 3299425652 ecr 1510067563,nop,wscale 7], length 0
17:33:45.713802 00:0c:29:06:c7:7e (oui Unknown) > 00:0c:29:65:3c:a3 (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [.], ack 1, win 502, options [nop,nop,TS val 1510067564 ecr 3299425652], length 0
17:33:45.713907 00:0c:29:65:3c:ad (oui Unknown) > 00:0c:29:5b:89:3e (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [.], ack 1, win 502, options [nop,nop,TS val 1510067564 ecr 3299425652], length 0
17:33:45.714096 00:0c:29:5b:89:3e (oui Unknown) > 00:0c:29:65:3c:ad (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [.], ack 4008910076, win 502, options [nop,nop,TS val 1510067564 ecr 3299425652], length 0
17:33:45.714207 00:0c:29:65:3c:b7 (oui Unknown) > 00:0c:29:15:da:6a (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [.], ack 1, win 502, options [nop,nop,TS val 1510067564 ecr 3299425652], length 0
17:33:49.369321 00:0c:29:06:c7:7e (oui Unknown) > 00:0c:29:65:3c:a3 (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [P.], seq 1:7, ack 1, win 502, options [nop,nop,TS val 1510071219 ecr 3299425652], length 6
17:33:49.369451 00:0c:29:65:3c:ad (oui Unknown) > 00:0c:29:5b:89:3e (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [P.], seq 1:7, ack 1, win 502, options [nop,nop,TS val 1510071219 ecr 3299425652], length 6
17:33:49.369819 00:0c:29:5b:89:3e (oui Unknown) > 00:0c:29:65:3c:ad (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [P.], seq 0:6, ack 1, win 502, options [nop,nop,TS val 1510071219 ecr 3299425652], length 6
17:33:49.369820 00:0c:29:65:3c:b7 (oui Unknown) > 00:0c:29:15:da:6a (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [P.], seq 0:6, ack 1, win 502, options [nop,nop,TS val 1510071219 ecr 3299425652], length 6
17:33:49.370069 00:0c:29:15:da:6a (oui Unknown) > 00:0c:29:65:3c:b7 (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 7, win 510, options [nop,nop,TS val 3299429309 ecr 1510071219], length 0
17:33:49.370087 00:0c:29:65:3c:a3 (oui Unknown) > 00:0c:29:06:c7:7e (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 7, win 510, options [nop,nop,TS val 3299429309 ecr 1510071219], length 0
17:33:51.689563 00:0c:29:06:c7:7e (oui Unknown) > 00:0c:29:65:3c:a3 (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [P.], seq 7:13, ack 1, win 502, options [nop,nop,TS val 1510073539 ecr 3299429309], length 6
17:33:51.689687 00:0c:29:65:3c:ad (oui Unknown) > 00:0c:29:5b:89:3e (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [P.], seq 7:13, ack 1, win 502, options [nop,nop,TS val 1510073539 ecr 3299429309], length 6
17:33:51.690131 00:0c:29:5b:89:3e (oui Unknown) > 00:0c:29:65:3c:ad (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [P.], seq 6:12, ack 1, win 502, options [nop,nop,TS val 1510073539 ecr 3299429309], length 6
17:33:51.690131 00:0c:29:65:3c:b7 (oui Unknown) > 00:0c:29:15:da:6a (oui Unknown), ethertype IPv4 (0x0800), length 72: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [P.], seq 6:12, ack 1, win 502, options [nop,nop,TS val 1510073539 ecr 3299429309], length 6
17:33:51.690297 00:0c:29:15:da:6a (oui Unknown) > 00:0c:29:65:3c:b7 (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 13, win 510, options [nop,nop,TS val 3299431629 ecr 1510073539], length 0
17:33:51.690305 00:0c:29:65:3c:a3 (oui Unknown) > 00:0c:29:06:c7:7e (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 13, win 510, options [nop,nop,TS val 3299431629 ecr 1510073539], length 0
17:33:55.442820 00:0c:29:06:c7:7e (oui Unknown) > 00:0c:29:65:3c:a3 (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [F.], seq 13, ack 1, win 502, options [nop,nop,TS val 1510077293 ecr 3299431629], length 0
17:33:55.442935 00:0c:29:65:3c:ad (oui Unknown) > 00:0c:29:5b:89:3e (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.20.2.10080: Flags [F.], seq 13, ack 1, win 502, options [nop,nop,TS val 1510077293 ecr 3299431629], length 0
17:33:55.443307 00:0c:29:5b:89:3e (oui Unknown) > 00:0c:29:65:3c:ad (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [F.], seq 12, ack 1, win 502, options [nop,nop,TS val 1510077293 ecr 3299431629], length 0
17:33:55.443307 00:0c:29:65:3c:b7 (oui Unknown) > 00:0c:29:15:da:6a (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.10.2.12345 > 192.168.30.2.10080: Flags [F.], seq 12, ack 1, win 502, options [nop,nop,TS val 1510077293 ecr 3299431629], length 0
17:33:55.486509 00:0c:29:15:da:6a (oui Unknown) > 00:0c:29:65:3c:b7 (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 14, win 510, options [nop,nop,TS val 3299435425 ecr 1510077293], length 0
17:33:55.486527 00:0c:29:65:3c:a3 (oui Unknown) > 00:0c:29:06:c7:7e (oui Unknown), ethertype IPv4 (0x0800), length 66: 192.168.20.2.10080 > 192.168.10.2.12345: Flags [.], ack 14, win 510, options [nop,nop,TS val 3299435425 ecr 1510077293], length 0
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .