Transport mode IPsec between machines behind Nat devices not working

Question

My scenario does not use any kind of VPN and it does not use RRAS.

I am trying to establish an IPsec connection between two Windows 10 machines (both with 1709 fully patched) that are both behind different Nat devices.

Through Windows firewall, I created two transport rules on both machines, the rules are related to the Smb protocol.

The machines are both behind a Nat, the two Nat devices are connected through Internet.

I created on both machines the registry key “AssumeUDPEncapsulationContextOnSendRule” and I set its value on both pcs to “2”.

On the Nat device in front of the server I am trying to reach I forwarded Udp ports 500 and 4500 and all Smb related ports to the server itself.

In both pcs I allowed, through Windows firewall, inbound and outbound Udp ports 500, 4500 and even allowed “Edge traversal” for Smb in connection on the server pc.

When I try to access remotely the server the quick mode security associations get created but the connection is not working. (The output of command Get-NetIpsecQuickmodeSa seems ok, that is the Udp encapsulation has been correctly created)

The connection does work when the client is behind a Nat and the server has a public Ip but it does not work when the client has a public Ip and the server is behind a Nat or when both the client and server are behind a Nat.

There must be some setting I am ignoring.

Please, help me solving this problem.

Thanks a lot

Answer 1

I was sure that there was some setting I was ignoring and there it is:

Set-NetFirewallSetting -AllowIPsecThroughNAT Both

The registry setting “AssumeUDPEncapsulationContextOnSendRule” is related to the old Windows IPsec policy agent service.

The above Powershell command allows the same behavior for Windows firewall, in particular It sets the registry value “IPsecThroughNAT” to “2”.

Now everything is working as expected and as it should have from the beginning, this should be the default setting.

Answer 2

Windows may not support the Server to be behind a NAT Device. It requires the Server to have Public IP address. There is a workaround by adding the following registry key.

Registry subkey location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent DWORD Value Name: AssumeUDPEncapsulationContextOnSendRule DWORD Value Data: 2

Source:

https://support.microsoft.com/en-us/help/926179/how-to-configure-an-l2tp-ipsec-server-behind-a-nat-t-device-in-windows

http://www.itdroplets.com/l2tp-vpn-not-working-in-windows/