I've developed an implementation for UDP hole punching that works for most use cases that I've come across. However, one of our offices has a network configuration where we have two ISPs providing us with internet for redundancy. This is set up to load balance between the two lines, so that whenever you make an outbound connection to something, it's pot luck which line you make that connection on. Obviously each line has its own external IP address, and my software has no knowledge of what that address is, or which it's using.
I believe this is stopping UDP hole punching from working. I connect to the known connection server that I'm running on AWS via one IP address, but when my software is told to connect to a peer somewhere else in the world, it may try and connect to that peer on the same IP as it did to the connection server, or it may use the other one, in which case it won't connect.
Is there a solution to this, or does the dual external IP addresses mean that UDP hole punching will never be reliable?