How to detect orphaned Unix socket files?

Question

I have created a reverse SSH tunnel that creates a UNIX socket on a distant server (let's call it proxy_srv), so that I can connect the tunnel owner (named target_srv in the rest of this question) in 2 steps: creating a link from the Unix socket to a TCP port with socat, and connect to that port with a SSH client (SSH client does not seems to accept connecting to a Unix socket, so I used that socat trick as a workaround).

The commands involved are (enough to reproduce, a little more in practice, with identity files & co), in that order:

From target_srv:

me@target_srv:% ssh -CNR /tmp/$(hostname):127.0.0.1:22 me@proxy_srv -o ExitOnForwardFailure=yes

From proxy_srv:

me@proxy_srv:% socat TCP-LISTEN:2222 UNIX-CONNECT:/tmp/target_srv

From any other computer that can access proxy_srv:

ssh someone@proxy_srv

The goal of all this is to have a tunnel built automatically from machines that use GPRS networks to an access point so that I can access them in case of problems, considering the fact I do not have physical access to them (too many kilometers away).

I have 2 major problems:

• when someone deletes the Unix socket file on proxy_srv, the tunnel does not ends, so I can not recreate it (easy to workaround, just use a dedicated user, but still a little concerning to me),

• most importantly, if for some reason something on target_srv was restarted, either a physical reboot or a system crash with things going up automatically after, the file on proxy_srv is not deleted, and the tunnel can't be rebuilt. Of course, I could just delete it, and hope the problem is not a conflict with another computer, but I think there must be a better way to at least work around that, for example detecting socket files that no longer have listeners and automatically delete them regularly (inside a dedicated folder, of course).

Any idea?

Answer 1

I understand there are many target_srv machines and for this reason you don't use -R 2222:127.0.0.1:22 but connect the 2222 port of proxy_srv to the machine of your choice ad hoc.

I think the right tool is VPN. If the proxy_srv was the VPN server and target_srv machines were clients, you would tunnel SSH to the (VPN) IP address of the desired client. If you made your local computer a client as well, you might even reach other clients semi-directly (without SSH tunneling). VPN software would be responsible for establishing, maintaining and renewing any target_srv's presence in the virtual network.

Suppose you cannot use VPN and this socket-based contraption of yours is a must. I don't know how to detect orphaned Unix domain sockets, however there's another way to deal with the situation.

Create a helper script on the proxy_srv, let's say /home/me/bin/ssh-tunnel-helper.sh:

#!/bin/sh

soc1="$1".tmp
soc2="$1"

mv "$soc1" "$soc2" || { rm "$soc1"; exit 2; }
while sleep 40; do
   [ -e "$soc2" ] || exit 1
done

(Don't forget to make it executable).

Then on a target_srv invoke:

socket="/tmp/$(hostname)-tunnel"
while sleep 5; do
   ssh -CR "$socket".tmp:127.0.0.1:22 -o ExitOnForwardFailure=yes me@proxy_srv \
      /home/me/bin/ssh-tunnel-helper.sh "$socket"
done

Now

• when someone deletes the unix socket file on proxy_srv

the ssh-tunnel-helper.sh script detects it and exits. The while loop on the target_srv renews the tunnel.

• if for some reason something on target_srv was restarted, […] the file on proxy_srv is not deleted

The *-tunnel socket is indeed not deleted, but sshd initially creates *-tunnel.tmp which is renamed later. The trick is in Unix you can open a file and move it (or even delete it). Renaming the socket doesn't disturb the existing tunnel, yet allows a new socket to be created in the future.

If the helper script is interrupted it may leave a stale *-tunnel.tmp socket that would prevent future tunnels from being established. I expect such incidents to be rare. Run rm /tmp/*-tunnel.tmp on proxy_srv to recover. Even if you happen to remove another socket that was just about to be renamed at this very moment, its helper script will exit and the tunnel will be renewed in a while.

Note:

• You may want to use autossh instead of ssh on target_srv machines to detect broken connections etc. Even with autossh you still need the while loop because autossh will exit if the helper script exits (e.g. when someone deletes the socket on proxy_srv).