[EDIT] This seems to be an OpenVPN bug - bridging with the server-side of the openvpn tap tunnel results in lost ARP packets, but bridging with the client side (of any client connected to the vpn) works perfectly. I've worked around the bug by bridging with a client instead of the server. The bug reporting process is quite a bit of work so it will be some time before I can write it all up to submit to the Openvpn project.
I'm observing Openvpn consistently dropping certain packets...
Scenario:
I set up a wide area Layer-2 network (using OpenVPN) to support Minecraft pocket-edition (MCPE) players (just family) to see each other in the network friends list and play together. There are three remote endpoints, all in different locations, and a central openvpn server. Each remote endpoint broadcasts Wi-Fi bridged to the Openvpn tap interface. This works great and players can see each other.
Recently I wanted to add an additional Wi-Fi endpoint locally, here at the server location. So I Added an ethernet port to the bridge and attached a Wi-Fi bridge to get Layer-2 connectivity to the existing openvpn bridge. At a glance this appears to work well; clients can get to the internet and L2 traffic looks normal.
However when players on a remote client endpoint try to play against those connected to the local wi-fi bridge, players can't see each other.
The local Wi-Fi bridges to the SERVER end of the openvpn tunnel, and this seems to be a factor, but this is unexpected.
After many hours of troubleshooting I've narrowed the problem down to one peciluar fact - Wi-Fi bridged to the server's openvpn tap interface (named tapmc) are not able to play against players on the other side of the VPN.
In other words if any two players are on the same Wi-Fi or on a client openvpn Wi-Fi endpoint, they can see eachother no matter how far away. BUT players connected to the Wi-Fi that's bridged to the SERVER-side openvpn tap interface run into problems - they cannot see players on the client side of the tunnel, and remote players can't see them.
To see each other, the game sends out a UDP broadcast packet every 1-2 seconds to port 19132 (ipv4). All players on the network receive these broadcasts, and if their game is the server then their game responds with a unicast packet to the requestor. This unicast response packet contains game information, so the player who is searching for active games on the network will see them show up in their friends list so they can join the game.
Attached is an analysis of a small period of time where packets are being lost. Packets go in one side of the tap tunnel, and don't come out the other side. I've captured the packets by running tcpdump against each side of the tunnel, on the openvpn tap interface itself, so there are no bridges in the path, although the interfaces are each members of a bridge.
What I see is that PLAYER2, while searching for games on the network sends the search broadcast, which is received by PLAYER1's game, which wants to respond with a unicast game-info packet but first needs to resolve the MAC address of PLAYER2, so it sends an ARP who-has. the who-has packet, and all subsequent retransmits of it, are received and responded to by PLAYER1 but those responses are NOT transmitted across the Openvpn tunnel to PLAYER1. Thus the L2 ARP resolution never succeeds, and the unicast game-info packet is never sent, and PLAYER2 never sees PLAYER1.
Also lost across the tunnel is the second copy of the game search broadcast packet, however this is not as detrimental to the process because the first of the two copies is transmitted successfully. But why only one?
Openvpn Server configuration
server 192.168.251.0 255.255.255.0
verb 3
key ***
ca ***
cert ***
dh ***
tls-auth ***
key-direction 0
keepalive 10 60
persist-key
persist-tun
client-to-client
proto udp
port ***
dev tapmc
status ***
ifconfig-pool-persist ***
user nobody
group nobody
Openvpn Client configuration
client
nobind
dev tapmc
remote-cert-tls server
remote ***
<key>
***
</key>
<cert>
***
</cert>
<ca>
***
</ca>
<tls-auth>
***
</tls-auth>
key-direction 1
#redirect-gateway def1
Openvpn versions: server: 2.4.8-1, clients: 2.4.7-1
