Hanging sshd process on remote host after reverse ssh tunnel connection gets broken

Question

I set up reverse tunnel this way:

function startconn () {
    ssh -N -b ${SRCIP} -X -R ${REMOTEIP}:${REMOTEPORT}:${LOCALIP}:${LOCALPORT} root@${REMOTEIP} &
    SSHPID=$!
    echo "$SSHPID" > $PIDFILE
    echo "Forwarding port ${REMOTEPORT} at ${REMOTEIP} to ${LOCALIP}:${LOCALPORT}"
}

This forwards REMOTEPORT@REMOTEIP via backup link available via SRCIP network (I have auxilliary NIC with SRCIP==192.168.5.2 that connects to backup link gateway at 192.168.5.1).

It works nicely, but there's a problem: if ssh connection gets broken, e.g. by restart of gateway, etc, in general anything that breaks TCP/IP connection of reverse tunnel ssh session, sshd process at REMOTEIP is hanging, preventing re-establishing reverse tunnel to this port (30200 below is REMOTEPORT):

netstat -anp | grep 30200
tcp        0      0 0.0.0.0:30200           0.0.0.0:*               LISTEN      8772/sshd: root

The only viable solution I see for now is the reconnect script logging on lost connection to REMOTEIP and killing sshd process "manually" before trying to re-establish reverse tunnel.

Is there any smarter/less cumbersome way to prevent remote sshd process blocking REMOTEIP port?

Answer 1

Yes, that is autossh, a wrapper around ssh designed exactly for that. The autossh man page states:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

I use it on a dozen machines or so, and its uptime is 100%.

I use it as follows: I place this line

   su my_name -c /home/my_name/bin/auto

in the /etc/rc.local file of the machine initiating the reverse tunnel (call it A). The executable (*chmod 755 auto) file auto is:

  #!/bin/sh

 /usr/lib/autossh/autossh -M 6527 -f -p 5298 -2 -N -R 8400:localhost:2222 userB@remote_hostB -i /home/my_name/.ssh/crypto-keyB

Here 5298 is the ssh port of the remote host (I called it B), 2222 the ssh port of the calling machine A. The port 6527 is used to check on the status of the connection; it will need to be opened on the remote server's (A) firewall for this to work.

This sets up the reverse tunnel. From a terminal on the remote server (B), I can now connect to the machine initiating the reverse tunnel (A) by means of the command

 sh -Y userA@localhost -p 8400 -i /home/userB/.ssh/crypto-keyA

You must remember to connect once from either machine to the other one before setting up the tunnel, because it will be necessary to answer interactively the usual question: Machine A has replied with cryptographic key ...; should it be written down in known-hosts?.

Once that is done, you are good to go. It is basically fire and forget.