SACK support in Windows is controlled through Registry at:
- Key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
- Value:
SackOpts
- Type: REG_DWORD
- Data: 0 to disable
SACK is not the only feature that uses TCP options; TCP1323Opts controls timestamping and window scaling, and I don't think there's any way to disable the "Max segment size" option.
Reference: https://www.fs-net.de/assets/download/docu/netdcu/en/NetDCU_TCPIP_ParametersConfigurableByUser.pdf
From there you can see the SACK_PERM constantly. So I want to disable SACKs on the Windows XP machine to see if there is any improvement.
Very unlikely.
First of all, 'SACK_PERM' is not the actual usage of SACK – 'SACK_PERM' is SACK capability negotiation, a TCP handshake option that indicates willingness to use SACK if the other peer agrees. A device that doesn't support SACK would just answer the handshake without including its own 'SACK_PERM' option and the connection would automatically proceed without SACK usage.
However, your device does include 'SACK_PERM' in its answer, indicating that it does support SACK – and not only that, but it also produces actual SACKs in the packets it sends (such as in packet 954 of your capture). So that disproves your initial guess that device 2 does not support SACK, and it means that disabling SACK on the client side is very unlikely to do anything useful.
(What you're describing as "constant SACK_PERM" is not what SACK usage looks like; it's more like "constant TCP handshake attempts" when device 1 keeps trying to establish a new connection, and even though Wireshark emphasizes it a lot in its GUI, the SACK offer is not the main content of these packets.)
Even if a device was so tight in memory space that it couldn't implement SACK, it would still be capable of accepting TCP SYN with options attached and just ignoring those options that it doesn't understand, because pretty much every OS in the last 35+ years has been adding those in every handshake – it's unrealistic that any manufacturer would produce a device that requires system-wide disabling SACK in each and every client.
Furthermore, it is not only SACK that uses TCP options; it is also TCP window size scaling; TCP MSS negotiation; TCP timestamping... all of which your device apparently supports just fine, judging from its SYN_ACK reply including all those options.
Second, in your description:
At package 963 Something happened with the application on Device 2. Apparently, it freezes. You can see that Device 2 doesn't replay, it only sends an ACK (different from what it was doing ACK and then answer).
At package 969 Device 1 resets the connection.
…is not the same connection. These packets are sent for different TCP port numbers.
The connection with packet 963 (tcp.stream eq 4) does not freeze at all, as far as I can see in the capture – the device actually answers in the next packet, and it continues working all the way until the end of capture.
The connection with packet 969 (tcp.stream eq 5) is a very short-lived connection that was established in packet 879, and it does freeze, but not due to the use of SACK; there is only one segment that needs ACKing at that point, and the device just doesn't ACK it.
The "SACK_PERM" packets (all of them tcp.stream eq 6) are a single attempt to establish a fresh connection after the reset (which is ignored as much as the client tries to retransmit the SYN); however, the SACK_PERM option in them is exactly the same as in the SYN packets of working TCP connections, so it's unlikely to be the cause of the connection attempt being ignored.
