How to set up an automatic (re)start of a background ssh tunnel

Question

I am a beginner user of linux, and also quite newbie at ssh and tunnels.

Anyway, my goal is to maintain a ssh tunnel open in background.

In order to do that, I wrote the following batch that I then added into crontab (the batch is automatically processed every 5 minutes during workdays and from 8am to 9pm). I read in some other thread in stackoverflow that one should use autossh that will ensure the ssh will always be ok through a recurrent check. So did I....

#!/bin/bash
LOGFILE="/root/Tunnel/logBatchRestart.log"
NOW="$(date +%d/%m/%Y' - '%H:%M)" # date & time of log

if ! ps ax | grep ssh | grep tunnelToto &> /dev/null
then
    echo "[$NOW] ssh tunnel not running : restarting it" >> $LOGFILE
    autossh -f -N -L pppp:tunnelToto:nnnnn [email protected] -p qqqq
    if ! ps ax | grep ssh | grep toto &> /dev/null
    then
            echo "[$NOW] failed starting tunnel" >> $LOGFILE
    else
            echo "[$NOW] restart successfull" >> $LOGFILE
    fi
fi

My problem is that sometimes the tunnel stops working, although every thing looks ok (ps ax | grep ssh > the result shows the two expected tasks : autossh main task and the ssh tunnel itself). I actually know about the problem cause the tunnel is used by a third party software that triggers an error as soon as the tunnel is no more responding.

SO I am wondering how I should improve my batch in order It will be able to check the tunnel and restart it if it happens to be dead. I saw some ideas in there, but it was concluded by the "autossh" hint... which I already use. Thus, I am out of ideas... If any of you have, I'd gladly have a look at them!

Thanks for taking interest in my question, and for your (maybe) suggestions!

Answer 1

Instead of checking the ssh process with ps you can do the following trick

create script, that does the following and add it to your crontab via crontab -e

#!/bin/sh

REMOTEUSER=username
REMOTEHOST=remotehost 

SSH_REMOTEPORT=22
SSH_LOCALPORT=10022

TUNNEL_REMOTEPORT=8080
TUNNEL_LOCALPORT=8080

createTunnel() {
    /usr/bin/ssh -f -N  -L$SSH_LOCALPORT:$REMOTEHOST:SSH_REMOTEPORT -L$TUNNEL_LOCALPORT:$REMOTEHOST:TUNNEL_REMOTEPORT $REMOTEUSER@$REMOTEHOST
    if [[ $? -eq 0 ]]; then
        echo Tunnel to $REMOTEHOST created successfully
    else
        echo An error occurred creating a tunnel to $REMOTEHOST RC was $?
    fi
}

## Run the 'ls' command remotely.  If it returns non-zero, then create a new connection
/usr/bin/ssh -p $SSH_LOCALPORT $REMOTEUSER@localhost ls >/dev/null 2>&1
if [[ $? -ne 0 ]]; then
    echo Creating new tunnel connection
    createTunnel
fi

In fact, this script will open two ports

• port 22 which will be used to check if the tunnel is still alive
• port 8080 which is the port you might want to use

Please check and send me further questions via comments

Answer 2

You don't need bash for this!

Nowadays most linux boxes come with systemd which has solid support for starting/stopping/restarting services with configurable conditions and timeouts.

1. As root create a systemd service unit:

sudo vim /etc/systemd/system/ssh-tunnel.service

2. Paste the following config into the new file:

[Unit]
Description=ssh
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=5
StartLimitBurst=1

[Service]
# NOTE: you MUST start ssh *without!* the -f (forking) switch, 
# so that systemd can monitor it and detect when the tunnel goes down
Type=simple
# forward *local* port 80 to port 8088 on the remote host
ExecStart=ssh [email protected] -N -R 8088:localhost:80
# send an exit signal to the SSH master process that controls the tunnel
ExecStop=ssh arh -O exit -R 8088:localhost:80
# see: https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStartSec=
TimeoutStartSec=10
TimeoutStopSec=10
Restart=always
RestartSec=10

# TODO: set the user name here; root will be used if unset
# make sure the user has the correct ssh key or otherwise 
# explicitly pass the key with `ssh -i /path/to/key/id_rsa   
User=my-user-name

# TODO: for debugging only; remove once you get the service working
StandardOutput=console

[Install]
WantedBy=multi-user.target

3. Enable and start the service unit:

# make service start on every boot (after network is up)
sudo systemctl enable ssh-tunnel

sudo systemctl start ssh-tunnel

4. (Optional) Check the status of your service:

systemctl status ssh-tunnel

How does this work?

StartLimitIntervalSec configures the interval for counting service starts. If a service is started more than StartLimitBurst times within this interval, it is not permitted to start any more. TimeoutStartSec configures the max time that systemd will wait for the service to start before it is considered to be in failed state.

With the configuration above if the SSH tunnel goes down, for example, because of temporary network glitches, systemd will wait for 10 seconds, and then restart the tunnel.

This approach is reliable, configurable and achieves the same result as the scripts given in the other answers or utilities like autossh, but in a generic way that can be used with any other tool.

The key here is to start SSH in foreground mode (i.e. do NOT send it to background with -fn switches) so that systemd can monitor whether the service is still alive and restart it in case of failure.

See also:

• systemd:TimeoutStartSec
• Self-healing services with systemd

Answer 3

(I add this as an answer since there is not enough room for it un a comment)

Ok, I managed to make the batch run to launch the ssh tunnel (I had to specify my hostname instead of localhost in order it could be triggered) :

#!/bin/bash

LOGFILE="/root/Tunnel/logBatchRedemarrage.log"
NOW="$(date +%d/%m/%Y' - '%H:%M)" # date et heure du log


REMOTEUSER=username
REMOTEHOST=remoteHost

SSH_REMOTEPORT=22
SSH_LOCALPORT=10022

TUNNEL_REMOTEPORT=12081
TUNNEL_SPECIFIC_REMOTE_PORT=22223
TUNNEL_LOCALPORT=8082

createTunnel() {
    /usr/bin/ssh -f -N  -L$SSH_LOCALPORT:$REMOTEHOST:$SSH_REMOTEPORT -L$TUNNEL_LOCALPORT:$REMOTEHOST:$TUNNEL_REMOTEPORT [email protected] -p $TUNNEL_SPECIFIC_REMOTE_PORT
    if [[ $? -eq 0 ]]; then
        echo [$NOW] Tunnel to $REMOTEHOST created successfully >> $LOGFILE
    else
        echo [$NOW] An error occurred creating a tunnel to $REMOTEHOST RC was $? >> $LOGFILE
    fi
    }

## Run the 'ls' command remotely.  If it returns non-zero, then create a new connection
/usr/bin/ssh -p $SSH_LOCALPORT [email protected] ls >/dev/null 2>&1
if [[ $? -ne 0 ]]; then
    echo [$NOW] Creating new tunnel connection >> $LOGFILE
    createTunnel
fi

However, I got some immediate message (below) when the tunnel is running and when cron tries to lauch the batch again... sounds like it cannot listen to it. Also since I need some time to get a proof , I can't say yet it will successfully restart if the tunnel is out.

Here's the response to the second start of the batch.

bind: Address already in use channel_setup_fwd_listener: cannot listen to port: 10022 bind: Address already in use channel_setup_fwd_listener: cannot listen to port: 8082 Could not request local forwarding.