Why some commands don't load user environment when executed with ssh? (while other do)

Question

On a target machine (OS X El Capitan) I have brew in /usr/local/bin. When I try to get brew's path I get no result:

# ssh [email protected] -p 2222 which brew
#

---

Troubleshooting:

1. brew is in /usr/local/bin/ and executes properly via ssh:

   # ssh [email protected] -p 2222 ls -l /usr/local/bin/brew
   -rwxr-xr-x  1 vagrant  admin  656 Mar 20 10:05 /usr/local/bin/brew
   # ssh [email protected] -p 2222 /usr/local/bin/brew --version
   Homebrew 0.9.5 (git revision 3a41; last commit 2016-03-20)
   

2. There is /usr/local/bin in path:

   # ssh [email protected] -p 2222 echo $PATH
   /usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/opt/local/bin:/opt/local/sbin
   

3. which itself does not seem to have a problem:

   # ssh [email protected] -p 2222 which bash
   /bin/bash
   

4. Also when logged interactively to the machine I get the intended result:

   # ssh [email protected] -p 2222
   Last login: Sun Mar 20 12:02:55 2016 from 10.0.2.2
   osx-pf:~ vagrant$ which brew
   /usr/local/bin/brew
   

5. Finally which brew works when in command substitution expression:

   # ssh [email protected] -p 2222 echo $(which brew)
   /usr/local/bin/brew
   

6. Calling echo before which brew causes it to give the correct result:

   # ssh [email protected] -p 2222 echo test ; which brew
   test
   /usr/local/bin/brew
   

7. Calling shell explicitly to run which brew does not change the situation:

   # ssh [email protected] -p 2222 /bin/sh -c "which\ brew"
   # ssh [email protected] -p 2222 /bin/sh -c "which\ which"
   /usr/bin/which
   

   However with explicit call to /bin/sh the "trick" from (6) no longer works:

   # ssh [email protected] -p 2222 /bin/sh -c "echo\ test\ \;\ which\ brew"
   test
   #
   

---

Why running which brew directly as an ssh command argument fails?

Why echo which external command in /bin/echo sees proper PATH while which does not? And why running two commands in sequence causes the second one to use environment of the first?

Is it possible to assure the environment would be loaded without referring to undocumented (I presume) workarounds like in (6)?

Answer 1

ssh [email protected] -p 2222 echo $PATH

Your local system's PATH will be printed here. You should quote the variable to prevent it from being expanded by the local shell:

ssh [email protected] -p 2222 'echo $PATH'

The same applies to:

ssh [email protected] -p 2222 echo $(which brew)

As for:

ssh [email protected] -p 2222 echo test ; which brew

which brew is run after the SSH command ends, by your local shell. That's why this didn't work:

ssh [email protected] -p 2222 /bin/sh -c "echo\ test\ \;\ which\ brew"

Side note: you don't need to do all that escaping.

---

The problem here might be that you're adding brew to your PATH in .profile (or .bash_profile or some similar file), which is run when a login shell is started. SSH starts a login shell only when it is run interactively, not when it's told to run a command. From man ssh:

 If command is specified, it is executed on the remote host instead of a login shell

You could try:

ssh [email protected] -p 2222 bash -lc brew

Answer 2

When you run commands directly from ssh, you are not "logging in", you are only running a command.

Some shell scripts run when you "log in" - ~/.bashrc, /etc/profile etc.

which uses environment variables - specifically $PATH - which may be set in one of these scripts. Without $PATH, which will not know where to look.

Also, ssh [email protected] -p 2222 echo $(which brew) will not give you what you expect - $(which brew) is interpreted by the shell, before it even gets to ssh - compare: ssh [email protected] -p 2222 echo $(hostname) with ssh [email protected] -p 2222 echo '$(hostname)' (note single quotes around the shell expansion). The first will get the LOCAL hostname, send it to the remote host to be echo'd. The second will be send the $(hostname) (literally) to the remote shell, to be expanded there.

UPDATE: I was slightly wrong about this.

When you run a command over ssh, ssh works out if a shell is needed (e.g. working out variables etc. If it is not needed, the binary program is run immediately. Otherwise bash is run, which may set up environment variables etc.

This is why (in your comment) brew --version fails - you don't mention a path. When you echo...; brew --version, the semicolon ; is a special shell character; ssh handles this for you, by running bash for you.