ssh to remote server hangs, but works locally (putty / cygwin / windows 10 to linux)

Question

I've got a linux server running openssh. I can connect to it both from the local LAN and remotely. However, there is one client (a Windows 10 laptop) that can only connect to it locally. When I try to connect remotely, authentication is accepted, but the ssh client on the laptop hangs and must be killed with Process Explorer. I thought the problem might be:

1. Windows Firewall - Nope. Turned it off, got same behavior.
2. ssh client (cygwin) - Nope. Got the same behavior with putty.
3. Windows 10 - Nope. I can successfully connect remotely from another Win10 machine.

I've tried a fresh install of both cygwin & putty.

I tried running ssh with several -v options, and comparing the output to the other Win10 machine that is able to connect. The output was identical, up to a point:

Authenticated to <<IP REMOVED>>.
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768

>>>  "bad" machine hangs here

debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to Linux Mint 17.3 Rosa (GNU/Linux 3.19.0-32-generic x86_64)

Welcome to Linux Mint

On rare occasions, it has gotten further - once or twice even to the Welcome message - but the connection is never responsive to typing input.

I tried running sshd -d manually on the server, and comparing output between a "bad" remote session and a "good" one from another client. Output is identical.

To summarize: it doesn't seem to be Windows Firewall, or the client software, or Win10, or the port forwarding to the server, or DNS, or the server itself. The problem is only this one client machine, and only when connecting from outside the local LAN. It is authenticating successfully. And the client machine is running the same OS/ssh client as another machine that doesn't have the problem, and I can't see anything in the logs that distinguishes it, either.

EDIT: I should also mention, ssh connection to other remote servers works fine from all machines. It seems to be just this server/client pair, and only when connecting remotely.

UPDATE: See my comments immediately below for more info - the problem seems to be specific to the local net.

What further steps can I take to debug it?

Answer 1

It looks to me like at the point that it hangs, TCP packets from the server stop reaching the client. The reason I think this is because it sometimes hangs at different points, and because the issue varies by changing the network configuration. For example, it could be some undesired interaction between port forwarding, NAT and/or firewalls. But the question is how to diagnose why this is happening on one client but not another. Two approaches I can think of:

• You could try packet monitoring on server and client and at points along the route to check if packets are indeed getting lost and at what point.

• You could experiment to try to find a relationship between network settings and the existence or non-existence of the problem on different clients. Swap all or some network settings and IP addresses between a working and non-working client to see if it swaps the problem.