I got a very strange problem and I am out of ideas to solve it. I hope anyone over here has a helpful idea.
Some info about the virtual machines:
- VM0: Debian 10 - 192.168.10.102 [+ 2 more interfaces]
- VM1: Debian 10 - 192.168.10.103
- Both are virtualized using proxmox (the proxmox firewall for both is not online, the cluster firewall allows all traffic in the local network)
- They are directly connected via a bridge interface
- Both machines have correct Routes (192.168.10.0/24 via the correct bridge and no route with a higher prevalence pointing at wrong places)
- Both machines have the correct entry for the other one in their ARP table
- Both machines have empty iptables rules and no other firewall configured
Behaviour when connecting from VM0 to VM1:
- VM0:[random high port] -> VM1:1234 is working properly via TCP
- SSH connections to VM1 can be made
Behaviour when connecting from VM1 to VM0:
- VM1:[random high port] -> VM0:1234 SYN packet arrives but no TCP handshake - no response
- VM1:[random high port] -> VM0:[ssh] Working
- VM1 -> ping -> VM0 Working
- VM1 -> traceroute -> VM0 Not working
Behaviour when connecting from VM0 to VM1:
- VM0:[random high port] -> VM1:1234 SYN packet arrives, TCP handshake happens as expected, connection OK
- VM0:[random high port] -> VM1:[ssh] Working
- VM0 -> ping -> VM1 Working
- VM0 -> traceroute -> VM1 Working
The thing happening on port 1234 is happening with any other TCP port as well. Except for SSH. The web server is not reachable etc.
My stomach tells me, it's a firewall issue for some reason but it doesn't appear very logical to me. The machines are - virtually - connected via Layer 2 because they share the same bridge and there should be no Layer 2 firewall. Proxmox shouldn't interfere with the Layer 2 traffic as well (I guess?).
Their ARP records are correct and detected on the correct interface.
Any ideas are greatly appreciated. Thank you in advance.
PS:
There's the failing response to the SYN packet captured with wireshark (on all interfaces so wrong routing shouldn't be the reason)