29

We have a some hardware we install at our customers' locations, that hardware connects to our ssh server and establishes a reverse ssh tunnel so we can gain access to several client systems for monitoring purposes.

Everything works fine until there is an unclean disconnect of the SSH session.

When that happens, on our SSH server the ports that were used by the reverse tunnel stay stuck in LISTENING mode and when our remote hardware eventually tries to auto reconnect and re-establish its tunnels it fails with the error

Warning: remote port forwarding failed for listen port XXXX

I tested if there was an issue with our SSH server or client by trying a clean disconnect and course that releases the ports just fine. When I simulate a connection failure (disconnect the Ethernet port of the client hardware for example) then we have the same issue I described above.

What is the proper way of handling this situation? Keep in mind these are reversed tunnels so whatever happens needs to be done on the SSH server. Ideally i need the ssh server to realize instantly that the SSH session hosting the tunnels is down and release the ports it was using. I guess the solution could involve killing the concerned SSH process but I need to be careful with that cause we have multiple clients connecting to the same ssh server and I wouldn't want to kick them offline.

Being so mature, I'm sure SSHD has some sort of built in feature to handle this but I just can't figure it out.

Please advise so I don't have to go back to administering Windows boxes ...

FYI: I'm running this on a Debian based distro.

Improve this question

1 Answer 1

Reset to default
24

You have to use ClientAliveInterval in your sshd_config. 

ClientAliveInterval 15

Ref: man sshd_config

ClientAliveCountMax
         Sets the number of client alive messages (see below) which may be
         sent without sshd(8) receiving any messages back from the client.
         If this threshold is reached while client alive messages are
         being sent, sshd will disconnect the client, terminating the
         session.  It is important to note that the use of client alive
         messages is very different from TCPKeepAlive (below).  The client
         alive messages are sent through the encrypted channel and
         therefore will not be spoofable.  The TCP keepalive option
         enabled by TCPKeepAlive is spoofable.  The client alive mechanism
         is valuable when the client or server depend on knowing when a
         connection has become inactive.

         The default value is 3.  If ClientAliveInterval (see below) is
         set to 15, and ClientAliveCountMax is left at the default,
         unresponsive SSH clients will be disconnected after approximately
         45 seconds.  This option applies to protocol version 2 only.

 ClientAliveInterval
         Sets a timeout interval in seconds after which if no data has
         been received from the client, sshd(8) will send a message
         through the encrypted channel to request a response from the
         client.  The default is 0, indicating that these messages will
         not be sent to the client.  This option applies to protocol
         version 2 only.
Improve this answer
5
  • Thank you Clement. I will look into setting that up on our server.
    – TCZ
    Commented Apr 21, 2014 at 20:12
  • Done and tested, works beautifully. Thank you very much.
    – TCZ
    Commented Apr 21, 2014 at 20:55
  • 1
    I have already implemented this exact solution, to the exactly same problem. However, I sometimes still find a connection that is in LISTEN state on the server, even though according to the logs on the client, the client already terminated the SSH tunnel executable after detecting a timeout (with ServerAliveInterval/ServerAliveCountMax). Most of the times it works as expected and the configuration seems good. But in rare cases, I encounter a connection which stays in the LISTEN state. I know it shouldn't be possible, is it a bug, or am I missing a case in which this solution does not work?
    – Yeti
    Commented Jan 26, 2020 at 7:34
  • 2
    In continuation of my previous comment, this is a related question without any answers, and this and this bug reports describe a similar problem, again without any answers. Even while also this answer proposes the same solution for this use-case, as a solution that ought to work, but doesn't
    – Yeti
    Commented Jan 26, 2020 at 7:45
  • I don't know what to tell you Yeti. When stuff that should work just doesn't, I start testing with different client/server hardware or software to confirm where the problem is.
    – TCZ
    Commented Nov 19, 2021 at 3:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .