I use an SSH tunnel from work to go around various idotic firewalls (it's ok with my boss :)). The problem is, after a while the ssh connection usually hangs, and the tunnel is broken.
If I could at least monitor the tunnel automatically, I could restart the tunnel when it hangs, but I haven't even figured a way of doing that.
Bonus points for the one who can tell me how to prevent my ssh connection from hanging, of course!
Sounds like you need autossh. This will monitor an ssh tunnel and restart it as needed. We've used it for a couple of years and it seems to work well.
autossh -M 20000 -f -N your_public_server -R 1234:localhost:22 -C
More details on the -M parameter here
autossh -f -nNT -i ~/keypair.pem -R 2000:localhost:22 [email protected] You might notice that I set this up using -nNT which doesn't create a remote terminal so that I can put autossh into the background, and the -i option for SSH to use a .pem file. If you're going to be keeping a connection open all the time, I definitely recommend going through the extra setup.
-M parameter: bugs.debian.org/cgi-bin/bugreport.cgi?bug=351162
-C forces compression of the connection, it's useful on modem lines and slow internet connections but if a fast connection is in place it will actually slow things down. Nowadays it should most likely be not used, unless your connection speed is terribly slow.
Systemd is ideally suited for this.
Create a service file /etc/systemd/system/sshtunnel.service containing:
[Unit]
Description=SSH Tunnel
After=network.target
[Service]
Restart=always
RestartSec=20
User=sshtunnel
ExecStart=/bin/ssh -NT -o ServerAliveInterval=60 -L 5900:localhost:5900 user@otherserver
[Install]
WantedBy=multi-user.target
(Modify the ssh command to suit)
sshtunnel so make sure that user exists firstsystemctl enable sshtunnel to set it to start at boot timesystemctl start sshtunnel to start immediatelyUpdate Jan 2018: some distros (e.g. Fedora 27) may use SELinux policy to prevent the use of SSH from systemd init, in which case a custom policy will need to be created to provide the necessary exemptions.
systemd system. If one uses Restart=on-failure then manually killing the SSH client will not result in a restart-by-systemd as the SSH client with exit with success.
Commented
Mar 22, 2018 at 19:36
ExecStart for example to build the ssh argument list, do basic checks etc then call it from the script like so exec /bin/ssh -N .... Here is my command: exec /bin/ssh -N -oExitOnForwardFailure=Yes -oTCPKeepAlive=no -oServerAliveInterval=5 -oServerAliveCountMax=6 -i "${LOCAL_PRIVATE_KEY}" -L "${TUNNEL_INLET}:${TUNNEL_OUTLET}" "${REMOTE_USER}@${REMOTE_MACHINE}" where TUNNEL_INLET="127.0.0.1:3307" and TUNNEL_OUTLET="127.0.0.1:3306"
Commented
Mar 22, 2018 at 19:42
StartLimitIntervalSec=0 to the [Unit] section. This disables the rate-limiting feature of systemd, which prevents restarting services if they fail too fast. This could happen if the local network device is temporarily down, and ssh would quit immediately with a connection refused error.
channel 0: open failed: connect failed: Connection refused stdio forwarding failed kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535 but 65535 was not mentioned in my command. My tunnel works fine if I set it up myself.
All stateful firewalls forget about a connection after not seeing a packet for that connection for some time (to prevent the state tables from becoming full of connections where both ends died without closing the connection). Most TCP implementations will send a keepalive packet after a long time without hearing from the other side (2 hours is a common value). If, however, there is a stateful firewall which forgets about the connection before the keepalive packets can be sent, a long-lived but idle connection will die.
If that is the case, the solution is to prevent the connection from becoming idle. OpenSSH has an option called ServerAliveInterval which can be used to prevent the connection from being idle for too long (as a bonus, it will detect when the peer died sooner even if the connection is idle).
I've used the following Bash script to keep spawning new ssh tunnels when the previous one dies. Using a script is handy when you don't want or can't install additional packages or use compiler.
while true
do
ssh <ssh_options> [user@]hostname
sleep 15
done
Note that this requires a keyfile to establish the connection automatically but that is the case with autossh, too.
On your own mac or linux machine configure your ssh keep the server ssh alive every 3 minutes. Open a terminal and go your your invisible .ssh in your home:
cd ~/.ssh/
then create a 1 line config file with:
echo "ServerAliveInterval 180" >> config
you should also add:
ServerAliveCountMax xxxx (high number)
the default is 3 so ServerAliveInterval 180 will stop sending after 9 minutes (3 of the 3-minute interval specified by ServerAliveInterval).
ServerAliveInterval 180 give us 6 minutes? intuition makes me try this : 180/60 == 3. So, does ServerAliveInterval work in multiples of 30 secs?
ServerAliveCountMax to a huge number - you are effectively disabling keepalive by doing so. ServerAliveCountMax limits the number of missed replies, not the number of successes, and terminates the connection if the other side is not responding. The understanding that it limits the overall amount of "keepalive" packets sent is based on a misreading of the manual. Thirty seconds of Googling will make this very clear.
For those who don't want to (or) can't use AutoSSH...
I have a NAS that I want to reach from the internet, I can't use port forwarding because my ISP uses CGNAT (my public IP is not really my public IP, I'm behind another router I don't have any control over). Therefore, to reach my NAS, I have a VPS (which I rent from OVH for a very small monthly cost), and that has a fixed public IP address. So to reach my NAS from the internet, I simply need to create an SSH tunnel between my NAS and my VPS, that reliably stays open all the time (for round the clock access). However, I suffered from the SSH tunnel being "closed" due to inactivity (depsite the ssh process staying up). This can easily be overcome by having the client (in my case, the VPS) "ping" the server (in my case, the NAS) using the keep alive option.
To create an SSH Tunnel, I issue the following command (from the NAS):
ssh -NT -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o ExitOnForwardFailure=yes -i /var/services/homes/foouser/.ssh/id_rsa -R 8080:localhost:80 -R 4443:localhost:443 foouser@<VPS>
To explain this command:
-N - Do not execute a remote command; this is useful for just forwarding ports.-T - Disable pseudo-tty allocation.-R 8080:localhost:80 - Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side. In this case, it means forward port 80 of the remote server to port 8080 of the client.-i /path/to/key - Specify the path to ssh key used to establish the ssh session, without this you will have to enter username (if not supplied) and password to establish the ssh session.ServerAliveInterval - the number of seconds that the client will wait before sending a "server alive" message to the server to keep the connection alive.ServerAliveCountMax - the number of "server alive" messages which may be sent without reply from the server. If this threshold is reached ssh will disconnect from the server, terminating the session.ExitOnForwardFailure - if set to "yes", the connection shall be terminated if ssh cannot set up all requested dynamic, tunnel, local, and remote port forwardings, (e.g. if either end is unable to bind and listen on a specified port).foouser@<VPS> - Specifies the user account foouser used to establish the remote port forwarding ssh session with the server <VPS>.It is also worth adding some ssh config options to the server (in my case, on my VPS) as well; by adding the following file if it doesn't already exist:
[foouser@vps ~]$ cat /home/foouser/.ssh/config
Host *
TCPKeepAlive yes
ClientAliveInterval 30
ClientAliveCountMax 9999
Note: you could replace the * (which means apply this config to "all hosts") with a specific host - In my case my NAS (i.e. the host that connects to my VPS) is behind my router; the public IP address of my router frequently changes as it's DHCP assigned (from my ISP) so I stuck with "all hosts".
SystemD Process (Synology NAS)
I also have this command (the one that starts the SSH tunnel as a systemd process, if anyone is interested, here is the script:
foouser@nas:~$ cat /etc/systemd/system/sshtunnel-web.service
[Unit]
Description=SSH Tunnel for WebStation
After=network.target
[Service]
Restart=always
RestartSec=1
User=foouser
ExecStart=/bin/ssh \
-NT \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=10 \
-o ExitOnForwardFailure=yes \
-i /var/services/homes/foouser/.ssh/id_rsa \
-R 8080:localhost:80 \
-R 4443:localhost:443 \
foouser@<VPS>
[Install]
WantedBy=multi-user.target
To start and enable the SSH Tunnel service:
foouser@nas:~$ sudo systemctl daemon-reload
foouser@nas:~$ sudo systemctl start sshtunnel-web.service
foouser@nas:~$ sudo systemctl enable sshtunnel-web.service
This has worked reliably for me for several months. This includes being reliable over several reboots of my home router, the VPS server, and the NAS.
It sure looks to me that you're all misinterpreting ServerAliveCountMax. As I understand the docs, it is the number of server alive messages which can go unanswered without the connection being terminated. So in cases like we're discussing here, setting it to a high value will just ensure that a hung connection will not be detected and terminated!
Simply setting ServerAliveInterval should be sufficient to solve the problem with a firewall forgetting about the connection, and leaving ServerAliveCountMax low will allow the originating end to notice the failure and terminate if the connection fails anyway.
What you want is, 1) for the connection to stay open permanently under normal circumstances, 2) for connection failure to be detected and the originating side to exit on failure, and 3) for the ssh command to be re-issued every time it exits (how you do that is very platform dependent, the "while true" script suggested by Jawa is one way, on OS X I actually set up a launchd item).
Always use ServerAliveInterval SSH option in case the tunnel issues are generated by expired NAT sessions.
Always use a respawning method in case the connectivity goes down entirely, you have at least three options here:
while true do ssh ...; sleep 5; done) do not remove the sleep command, ssh may fail quickly and you'll respawn too many processes/etc/inittab, to have access to a box shipped and installed in another country, behind NAT, without port forwarding to the box, you can configure it to create an ssh tunnel back to you:
tun1:2345:respawn:/usr/bin/ssh -i /path/to/rsaKey -f -N -o "ServerAliveInterval 180" -R 55002:localhost:22 user@publicip 'sleep 365d'
upstart script on Ubuntu, where /etc/inittab is not available:
start on net-device-up IFACE=eth0
stop on runlevel [01S6]
respawn
respawn limit 180 900
exec ssh -i /path/to/rsaKey -N -o "ServerAliveInterval 180" -R 55002:localhost:22 user@publicip
post-stop script
sleep 5
end script
or always use both methods.
I solved this problem with this:
Edit
~/.ssh/config
And add
ServerAliveInterval 15
ServerAliveCountMax 4
According to man page for ssh_config:
ServerAliveCountMax
Sets the number of server alive messages (see below) which may be
sent without ssh(1) receiving any messages back from the server.
If this threshold is reached while server alive messages are
being sent, ssh will disconnect from the server, terminating the
session. It is important to note that the use of server alive
messages is very different from TCPKeepAlive (below). The server
alive messages are sent through the encrypted channel and there‐
fore will not be spoofable. The TCP keepalive option enabled by
TCPKeepAlive is spoofable. The server alive mechanism is valu‐
able when the client or server depend on knowing when a connec‐
tion has become inactive.
The default value is 3. If, for example, ServerAliveInterval
(see below) is set to 15 and ServerAliveCountMax is left at the
default, if the server becomes unresponsive, ssh will disconnect
after approximately 45 seconds. This option applies to protocol
version 2 only.
ServerAliveInterval
Sets a timeout interval in seconds after which if no data has
been received from the server, ssh(1) will send a message through
the encrypted channel to request a response from the server. The
default is 0, indicating that these messages will not be sent to
the server. This option applies to protocol version 2 only.
ExitOnForwardFailure yes is a good adjunct to the other suggestions. If it connects but can't establish the port forwarding it's just as useless to you as if it hadn't connected at all.
I've had a need to maintain an SSH-tunnel long-term. My solution was running from a Linux server, and it's just a small C program that respawns ssh using key-based authentication.
I'm not sure about the hanging, but I've had tunnels die due to timeouts.
I would love to provide the code for the respawner, but I can't seem to find it right now.
while there are tools like autossh that helps to restart ssh session... what i find to be really useful is to run the 'screen' command. It allows you to RESUME your ssh sessions even after you disconnect. Especially useful if your connection is not as reliable as it should be.
...don't forget to mark this is the 'correct' answer if it helps you k! ;-)
A bit of a hack, but I like to use screen to keep this. I currently have a remote forward that has been running for weeks.
Example, starting locally:
screen
ssh -R ......
When the remote forward is applied, and you have a shell on the remote computer:
screen
Ctrl + a + d
You now have an uninterrupted remote forward. The trick is to run screen on both ends
Recently had this issue myself, because these solutions require you to re-enter the password every time if you use a password login I used sshpass in a loop along with a text prompt to avoid having the password in the batch file.
Thought I'd share my solution on this thead in case anyone else has the same issue:
#!/bin/bash
read -s -p "Password: " pass
while true
do
sshpass -p "$pass" ssh user@address -p port
sleep 1
done
ps aux and look at all the command line arguments of all the processes running on your computer. Instead of passwords, use public keys. Then, you won't be prompted for a password (unless you password-protected the key, of course).
Commented
Jun 14 at 21:10
As autossh does not meet our needs (it exists with error if it can't connect to the server at the very first attempt), we've written a pure bash application: https://github.com/aktos-io/link-with-server
It creates a reverse tunnel for the NODE's sshd port (22) on the server by default. If you need to perform any other actions (like forwarding additional ports, sending mails on connection, etc...) you can place your scripts on-connect and on-disconnect folders.
You could very simply use tmux command on your remote server. It allows you to open a "session" on the remote machine where your will run your remote code.
Then you can exit (or lose the connection in your case) the remote machine without worrying given that your code is safely running in your tmux session.
Finally, when you reconnect on the remote server, you only need to tmux attach to come back to the session you had previously open and that's it.
I use simple terminal command after SSH connection
watch -n 60 echo 1
It keeps my SSH connection all time unless problem with internet.
No need extra packages to install like autossh
I have had similar problems with my previous ISP. For me it was the same with any tcp connection, visiting websites or sending mail.
The solution was to configure a VPN connection over UDP(I was using OpenVPN). This connection was more tolerant to whatever caused the disconnections. Then you can run any service through this connection.
There can still be issues with the connection but since the tunnel will be more tolerant any ssh session will feel a short holdup rather than being disconnected.
To do this you will need a VPN service online which you can setup on your own server.
Rather than using autossh which I highly recommend you can go ahead and use sshpass with ssh through a python script that auto-reconnects when the ssh session is disconnected. This might help some workflows where you would prefer to have access to python to do some tasks when that disconnect happens.
sshpassIf you would like to use the ssh password and not an key.pem file to authenticate, you should use sshpass to install sshpass you can simply do: sudo apt-get install sshpass
import os
while True:
os.system("sshpass -p 'mypass' ssh -L 5000:0.0.0.0:5000 [email protected] -p 22")
# perform x task here, like alert the user, you have choices.
print("[+] Reconnecting Again")
watchcommand like:watch -n1 60 echo "wiiiii". Tunnel will not die unless network is broken or you don't use it.