In your situation, the "DMZ" setting is actually nothing more than a very broad port-forwarding (DNAT) rule on the router. If you want a single port to be forwarded elsewhere, usually you can simply add a port-forwarding rule to the router as you normally would – and it will take priority over the DMZ rule.
If your router allows this, you should do it on the router. Not only will it reduce load on the current DMZ-machine by simply bypassing it, but you will also avoid the hairpinning problem described below.
What you're trying to achieve on your DMZ-machine is still the same as your router's port-forwarding does (that is, simple DNAT).
But because you only have one interface, the main problem you're having is the same "NAT hairpinning" issue as with accessing port-forwarded services from inside; only it's the other way around. (Your router sends packets to the DMZ machine but receives packets from the other machine and nothing matches up.)
The workaround to this usually requires a SNAT (masquerade) rule in addition, and will result in the destination machine thinking that all connections come from your DMZ-machine (the original source IPs being lost).
The second problem is that it is currently unknown 1) whether your DMZ-machine actually has IP routing (forwarding) even enabled (you should check that), and 2) what firewall rules it already has (you should check that too – you mention adding rules, but you don't mention checking or deleting the old ones).
No matter if it's one interface or two, you're still asking the DMZ-machine to route IP packets, and therefore it should have the net.ipv4.conf.all.forwarding
sysctl set to 1.
iptables rules are kept in a list (chain), checked from top to bottom. Using -A
adds a rule to the bottom of the specified table/chain. You should always look at your current ruleset (as shown by iptables-save
or at least iptables [-t nat] -S
) to make sure the rules are in the correct order and so on.
The network is not a black box. You can use packet-capture tools such as Wireshark or tcpdump to see what packets come in and what packets come out. Maybe your DMZ-machine is already correctly forwarding all those packets to the other system – but the replies aren't coming through? That's a different problem with a different solution.