How to investigate why a computer stops a TCP connection?

Question

I've just written this question on another StackExchange site, but I was advised to ask the same question here.

I would like to broaden the question: how can I know why a host closes a TCP socket connection, requested by another host? ((Event) logs, network monitoring, ...)

Answer 1

Some broad answers for the broad question, as these are typical steps I perform in the same situation:

1. Tail /var/log/{messages,secure} during the client connection attempt
2. Check /etc/hosts.deny for the IP of the client
3. Perform a packet capture (tcpdump, wireshark) at both ends to ensure the Server receives the Client SYN, and it's not a middle hop rejecting/blocking the request
4. Look for open and listening sockets on the Server end (netstat or similar) before client connection attempts
5. Check Client and Server firewall rules (including security groups et al if in cloud) to ensure inbound/outbound connections are permitted on the attempted IP/port pair
6. Try the same connection but with a well established tool (e.g. nc/netcat) - is it your code/tool that fails to establish a connection on that IP/port or other tools too?

Regarding your original question on the other site, are you able to establish a full 3-way tcp handshake, but the disconnect (RST/FIN packet?) occurs once sending data? If so, that might point to the application protocol above TCP not liking the data sent and deciding to kill the connection.