iptables is treating traffic from different hosts differently in spite of rules

Question

Something spooky is happening here and I don't know how to fix it. Summary: traffic that appears identical to tcpdump is being treated by iptables differently. Details below.

Test setup:

• device 0 - A linux machine sitting on IP 192.168.0.121

• device 1 - a dumb device that just sends packets to port 4000 on 192.168.0.121. The IP address of this device is set with the command: sudo arp -s 192.168.0.27 MAC_ADDRESS, because it does not make DHCP requests and can't be made to do anything other than its job.

• device 2 - A linux machine sending packets to port 4000 on 192.168.121 with the command:

  watch -n 1 "date | nc -4u -w1 -v 192.168.0.121 4000"

tcpdump output of the situation:

22:00:01.845359 IP 192.168.3.30.50705 > 192.168.0.121.4000: UDP, length 1
22:00:01.845391 IP 192.168.3.30.50705 > 192.168.0.121.4000: UDP, length 29
22:00:02.022257 IP 192.168.0.27.27 > 192.168.0.121.4000: UDP, length 12
22:00:03.022797 IP 192.168.0.27.27 > 192.168.0.121.4000: UDP, length 12
...

In this circumstance, the output of socat udp-l:4000,fork stdout is:

�����XX@�XWed Sep  2 22:02:18 PDT 2015
������XX��XWed Sep  2 22:02:22 PDT 201
...

And the output of socat udp-l:2700,fork stdout is empty.

The weird junk is from the sensor, and the date is from the above command. Note that the destination IP/port of this traffic is the exact same according to tcpdump. Now if I forward all traffic from UDP port 4000 to 2700:

sudo iptables -t nat -A PREROUTING -p udp --dst 192.168.0.121 --dport 4000 -j DNAT --to-destination 192.168.0.121:2700

tcpdump looks the exact same, but now, the output of socat udp-l:4000,fork stdout is:

���@�@�@�������@�@�@�������@������������@�@@����@��� ...

and the output of socat udp-l:2700,fork stdout is:

XXXWed Sep  2 22:05:42 PDT 2015
XXXWed Sep  2 22:05:46 PDT 2015
XXXWed Sep  2 22:05:50 PDT 2015
...

As you can see, traffic from device 1 is getting ignored by iptables and continuing to the host on port 4000, but traffic from device 2 is getting caught by the rule and getting forwarded to port 2700. I can clear iptables and reissue the rule all day and it just switches between those two states.

Anyone have some insight into what might be happening?

Answer 1

It turns out that two devices were both sending the linux host at 192.168.0.121 from 192.168.0.27, just with different MAC addresses. Obviously this is not compliant behavior, and iptables should not be expected to cope with it.

I would like some suggestions on how to let these two devices coexist on the same network without having the vendor write us a firmware upgrade. So if anybody has an idea, let me know.