How can a "dropped" but still connected reverse-SSH tunnel be picked up again?

Question

I have a remote Raspberry Pi computer that connects from behind its restrictive firewall to my server by reverse SSH tunnel. Imagine that this Raspberry Pi is on the difficult-to-access roof of a building in another country (perhaps while a storm is in progress), so I want its connection to be reliable.

To connect to the server, the Raspberry Pi runs a process like this:

while true; do
    ssh -R 19999:localhost:22 www.sern.pro
    sleep 30
done

The part 19999:localhost:22 basically means that all traffic on port 19999 on the server should be forwarded to port 22 of the Raspberry Pi. So, I can connect to the Raspberry Pi by SSHing to the server and then running a command like the following:

ssh localhost -p 19999

This normally works just fine and, on the server, a command like netstat --all --timers --program --numeric | egrep '127.0.0.1:*(LISTEN|.*)' | sort lists the reverse SSH connection as something like this:

tcp        0      0 127.0.0.1:19999         0.0.0.0:*               LISTEN      2972/5           off (0.00/0/0)

However, occasionally I have found that this listing on the server vanishes while the Raspberry Pi maintains its SSH connection to the server. That is, the command ssh localhost -p 19999 results in an error message like the following:

ssh: connect to host localhost port 19999: Connection refused

Yet, the Raspberry Pi SSH connection remains perfectly fine, capable of running commands on the server. I don't know the right words to describe what is going wrong when this happens.

So... short of booking a flight and climbing to the roof of a building, how could I pick up this active SSH connection again as a proper reverse SSH tunnel? How can I get access to the local port again?

Answer 1

More than a decade ago autossh utility was written for such purposes.

You can use the following connection script

#!/bin/sh
export AUTOSSH_GATETIME=0 

autossh -M 0
-o "PubkeyAuthentication=yes" \
-o "StrictHostKeyChecking=false" \
-o "PasswordAuthentication=no" \
-o "ExitOnForwardFailure=yes" \
-o "ServerAliveInterval 60" \
-o "ServerAliveCountMax 3" \
-fNR 19999:localhost:22 www.sern.pro

Of course, you need to establish key-based authentication before starting this script.

Answer 2

The problem you're seeing means that the connection is not available, if you check at the end of the command that list all the connections you maybe see that the connection for that port is not any more on LISTEN and now is TIME-WAIT or CLOSE-WAIT, that means that the connections still 'alive' but only as a running process in the raspberry.

To get this working you can create an script that checks if there is a reversessh connection if not set it up.

To solve the issue you can add code to the script to check if there is a ssh process id on the unit and if you have a marker file then you kill that process to restart the tunnel again.

Foe example you have an empty file called RESTARTSSH, in your script you can check if that file exists, if yes then search the process id of the ssh and kill it, then restart the tunnel.

To check the process id you can use:

ps -ef | grep ServerAliveInterval | grep -v grep | awk '{print $2}'