On-demand SSH Socks proxy through systemd user units with socket-activation doesn't restart as wished

Question

To reach an isolated network I use an ssh -D socks proxy.

In order to avoid having to type the details every time I added them to ~/.ssh/config:

$ awk '/Host socks-proxy/' RS= ~/.ssh/config
Host socks-proxy
  Hostname pcit
  BatchMode yes
  RequestTTY no
  Compression yes
  DynamicForward localhost:9118

Then I created a systemd-user service unit definition file:

$ cat ~/.config/systemd/user/SocksProxy.service 
[Unit]
Description=SocksProxy Over Bridge Host

[Service]
ExecStart=/usr/bin/ssh -Nk socks-proxy

[Install]
WantedBy=default.target

I let the daemon reload the new service definitions, enabled the new service, started it, checked its status, and verified, that it is listening:

$ systemctl --user daemon-reload
$ systemctl --user list-unit-files | grep SocksP
SocksProxy.service   disabled

$ systemctl --user enable SocksProxy.service
Created symlink from ~/.config/systemd/user/default.target.wants/SocksProxy.service to ~/.config/systemd/user/SocksProxy.service.

$ systemctl --user start SocksProxy.service 
$ systemctl --user status SocksProxy.service 
● SocksProxy.service - SocksProxy Over Bridge Host
   Loaded: loaded (/home/alex/.config/systemd/user/SocksProxy.service; enabled)
   Active: active (running) since Thu 2017-08-03 10:45:29 CEST; 2s ago
 Main PID: 26490 (ssh)
   CGroup: /user.slice/user-1000.slice/[email protected]/SocksProxy.service
           └─26490 /usr/bin/ssh -Nk socks-proxy

$ netstat -tnlp | grep 118
tcp     0    0 127.0.0.1:9118        0.0.0.0:*             LISTEN     
tcp6    0    0 ::1:9118              :::*                  LISTEN

This works as intended. Then I wanted to avoid having to manually start the service, or running it permanently with autossh, by using systemd socket-activation for on-demand (re-)spawning. That didn't work, I think (my version of) ssh cannot receive socket file-descriptors.

I found the documentation (1,2), and an example for using the systemd-socket-proxyd-tool to create 2 "wrapper" services, a "service" and a "socket":

$ cat ~/.config/systemd/user/SocksProxyHelper.socket 
[Unit]
Description=On Demand Socks proxy into Work

[Socket]
ListenStream=8118
#BindToDevice=lo
#Accept=yes

[Install]
WantedBy=sockets.target

$ cat ~/.config/systemd/user/SocksProxyHelper.service 
[Unit]
Description=On demand Work Socks tunnel
After=network.target SocksProxyHelper.socket
Requires=SocksProxyHelper.socket SocksProxy.service
After=SocksProxy.service

[Service]
#Type=simple
#Accept=false
ExecStart=/lib/systemd/systemd-socket-proxyd 127.0.0.1:9118
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

$ systemctl --user daemon-reload

This seems to work, until ssh dies or gets killed. Then it won't re-spawn at the next connection attempt when it should.

Questions:

1. Can /usr/bin/ssh really not accept systemd-passed sockets? Or only newer versions? Mine is the one from up2date Debian 8.9.
2. Can only units of root use the BindTodevice option?
3. Why is my proxy service not respawning correctly on first new connection after the old tunnel dies?
4. Is this the right way to set-up an "on-demand ssh socks proxy"? If, not, how do you do it?

Answer 1

• Can /usr/bin/ssh really not accept systemd-passed sockets?

I think that's not too surprising, considering:

• OpenSSH is an OpenBSD project
• systemd only supports the Linux kernel
• systemd support would need to be explicitly added to OpenSSH, as an optional/build-time dependency, so it would probably be a hard sell.

---

• Can only units of root use the BindTodevice option?

User systemd instances are generally pretty isolated, and e.g. can not communicate with the main pid-0 instance. Things like depending on system units from user unit files are not possible.

The documentation for BindToDevice mentions:

Note that setting this parameter might result in additional dependencies to be added to the unit (see above).

Due to the above-mentioned restriction, we can imply that the option doesn't work from user systemd instances.

---

• Why is my proxy service not respawning correctly on first new connection after the old tunnel dies?

As I understand, the chain of events is as follows:

• SocksProxyHelper.socket is started.
• A SOCKS client connects to localhost:8118.
• systemd starts SocksProxyHelper.service.
• As a dependency of SocksProxyHelper.service, systemd also starts SocksProxy.service.
• systemd-socket-proxyd accepts the systemd socket, and forwards its data to ssh.
• ssh dies or is killed.
• systemd notices, and places SocksProxy.service into a inactive state, but does nothing.
• SocksProxyHelper.service keeps running and accepting connections, but fails to connect to ssh, as it is no longer running.

The fix is to add BindsTo=SocksProxy.service to SocksProxyHelper.service. Quoting its documentation (emphasis added):

Configures requirement dependencies, very similar in style to Requires=. However, this dependency type is stronger: in addition to the effect of Requires= it declares that if the unit bound to is stopped, this unit will be stopped too. This means a unit bound to another unit that suddenly enters inactive state will be stopped too. Units can suddenly, unexpectedly enter inactive state for different reasons: the main process of a service unit might terminate on its own choice, the backing device of a device unit might be unplugged or the mount point of a mount unit might be unmounted without involvement of the system and service manager.

When used in conjunction with After= on the same unit the behaviour of BindsTo= is even stronger. In this case, the unit bound to strictly has to be in active state for this unit to also be in active state. This not only means a unit bound to another unit that suddenly enters inactive state, but also one that is bound to another unit that gets skipped due to a failed condition check (such as ConditionPathExists=, ConditionPathIsSymbolicLink=, … — see below) will be stopped, should it be running. Hence, in many cases it is best to combine BindsTo= with After=.

---

• Is this the right way to set-up an "on-demand ssh socks proxy"? If, not, how do you do it?

There's probably no "right way". This method has its advantages (everything being "on-demand") and disadvantages (dependency on systemd, the first connection not getting through because ssh hasn't begun listening yet). Perhaps implementing systemd socket activation support in autossh would be a better solution.

Answer 2

For future reference, i'm pasting below the systemd --user configuration files for on-demand ssh-tunnel using systemd-socket-proxyd daemon, with various enhancements and explanation comments:

~/.config/systemd/user/ssh-tunnel-proxy.socket

[Unit]
Description=Socket-activation for SSH-tunnel

[Socket]
ListenStream=1000

[Install]
WantedBy=sockets.target

~/.config/systemd/user/ssh-tunnel-proxy.service

[Unit]
Description=Socket-activation proxy for SSH tunnel

## Stop also when stopped listening for socket-activation.
BindsTo=ssh-tunnel-proxy.socket
After=ssh-tunnel-proxy.socket

## Stop also when ssh-tunnel stops/breaks
#  (otherwise, could not restart).
BindsTo=ssh-tunnel.service
After=ssh-tunnel.service

[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd --exit-idle-time=500s localhost:1001

~/.config/systemd/user/ssh-tunnel.service

[Unit]
Description=Tunnel to SSH server

## Stop-when-idle is controlled by `--exit-idle-time=` in proxy.service
#  (from `man systemd-socket-proxyd`)
StopWhenUnneeded=true

[Service]
Type=simple
## Prefixed with `-` not to mark service as failed on net-fails;
#  will be restarted on-demand by socket-activation.
ExecStart=-/usr/bin/ssh -kaxNT -o ExitOnForwardFailure=yes  hostname_in_ssh_config  -L 1001:localhost:2000
## Delay enough time to allow for ssh-authentication to complete
#  so tunnel has been established before proxy process attaches to it,
#  or else the first SYN request will be lost.
ExecStartPost=/bin/sleep 2

Customizations

In the scripts above, you have to replace the following strings:

• 1000 - (ssh-tunnel-proxy.socket file)
  what (host:)port to listen locally for socket-activation of the tunnel, e.g. to emulate a local MYSql port.
• 1001 - (ssh-tunnel-proxy.service & ssh-tunnel.service files)
  what local (host:)port to use when proxy-forwarding to ssh service-process;
  just choose an unused port.
• hostname_in_ssh_config - (ssh-tunnel.service file)
  the host-group to connect to, referred in your ssh-config (any SOCKS configuration belong there).
• localhost:2000 - (ssh-tunnel.service file)
  the ssh tunnel's remote host:port endpoint, e.g. where the remote MYSql binds to.
• x2 delay timings (ssh-tunnel-proxy.service & ssh-tunnel.service files)
  they are explained in the comments.
• ssh-tunnel-... - (the prefix of all unit-files)
  make it descriptive of your need for the tunnel, e.g. mysql-tunnel-....
• SSH - (in the Description= directives of all unit-files)
  make it descriptive of your need for the tunnel, e.g. MYSql.

Commands for controlling the tunnel

# After any edits.
systemctl --user daemon-reload
# If socket's unit-file has been edited.
systemctl --user restart ssh-tunnel-proxy.socket

# To start listening for on-demand activation of the tunnel
systemctl --user start ssh-tunnel.socket

# To enable on-demand tunnel on Boot
systemctl --user enable ssh-tunnel-proxy.socket

# To gracefully stop tunnel (any cmd will do)
systemctl --user stop ssh-tunnel.service
systemctl --user stop ssh-tunnel-proxy.service

# To gracefully stop & disable tunnel (till next reboot)
systemctl --user stop ssh-tunnel-proxy.socket

# To view the health of the tunnel
systemctl --user status ssh-tunnel-proxy.{socket,service} ssh-tunnel

# To reset tunnel after errors (both cmds may be needed)
systemctl --user reset-failed ssh-tunnel ssh-tunnel-proxy
systemctl --user restart ssh-tunnel-proxy.socket

Answer 3

Due to the reputation system, I'm not able to comment @ankostis solution...

I suspect that systemd has options to avoid the use of a bash sleep-loop

Yes, there is systemd-notify.

Also incorporating @noam suggestion to use use a socket between systemd-socket-proxyd and ssh to not waste local ports, it would look like the following:

~/.config/systemd/user/ssh-tunnel-proxy.service

[Unit]
Description=Socket-activation proxy for SSH tunnel

## Stop also when stopped listening for socket-activation.
BindsTo=ssh-tunnel-proxy.socket
After=ssh-tunnel-proxy.socket

## Stop also when ssh-tunnel stops/breaks
#  (otherwise, could not restart).
BindsTo=ssh-tunnel.service
After=ssh-tunnel.service

[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd --exit-idle-time=500s ${XDG_RUNTIME_DIR}/ssh-tunnel-proxy

~/.config/systemd/user/ssh-tunnel.service

[Unit]
Description=Tunnel to SSH server

## Stop-when-idle is controlled by `--exit-idle-time=` in proxy.service
#  (from `man systemd-socket-proxyd`)
StopWhenUnneeded=true

[Service]
Type=notify
NotifyAccess=all
## Prefixed with `-` not to mark service as failed on net-fails;
#  will be restarted on-demand by socket-activation.
ExecStart=-/usr/bin/ssh -kaxNT -o ExitOnForwardFailure=yes -o ControlMaster=no -o StreamLocalBindUnlink=yes -o PermitLocalCommand=yes -o LocalCommand="systemd-notify --ready" hostname_in_ssh_config -L ${XDG_RUNTIME_DIR}/ssh-tunnel-proxy:localhost:2000

The ssh_config man page says about LocalCommand "Specifies a command to execute on the local machine after successfully connecting to the server".

ControlMaster=no ensures that the ssh session never gets a master in case multiplexing is used, since it's town down automatically and would kill all other multiplexed ssh sessions to the same host.

Finally, it might be a good idea to bind the socket port to localhost.

~/.config/systemd/user/ssh-tunnel-proxy.socket

[Unit]
Description=Socket-activation for SSH-tunnel

[Socket]
ListenStream=127.0.0.1:1000
ListenStream=[::1]:1000

[Install]
WantedBy=sockets.target

Answer 4

Still testing this (as I am writing this answer, I am using it), but I think the missing ingredient is -o ExitOnForwardFailure=yes as an option to your ssh binary.