How do I solve the error "execv: No such file or directory" from rsync?

Question

How do I solve the error "execv: No such file or directory" from rsync?

I'm trying to back up several machines to a linux server using rsync and it's working in most scenarios:

• path-to-path rsync backup of the server itself
• rsync on the server to get files from windows remotes running cygwin-rsyncd

What doesn't work is backing up my Android phone (running SimpleSSHD) using rsync-via-ssh. At least, not in any way that is initiated by the server, which is my goal.

• server: Gentoo Linux, hostname "crucible", ip 192.168.7.94
• phone: LineageOS on HTC One M8, hostname "rondel", ip 192.168.7.33

Here's a breakdown of what works and what doesn't, and how.

• On the server, opening ssh into the phone, and from there, opening ssh back to the server: WORKS

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel
:/ # ls
[phone file listing]

:/ # ssh -i /data/smbrw_key -p 22 -l smbrw 192.168.7.94
Could not create directory '/.ssh'.
The authenticity of host '192.168.7.94 (192.168.7.94)' can't be established.
ECDSA key fingerprint is SHA256:[random stuff here].
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/.ssh/known_hosts).
smbrw@crucible ~ $ ls
[local server file listing]

• On the server, using rsync via ssh to get files from the phone: ERROR, "execv: No such file or directory"

aeheathc@crucible ~ $ rsync -a --progress --delete -e "ssh -i /home/aeheathc/.ssh/id_rsa -p 2222" backuppc@rondel:/storage/emulated/0/ /backup/redundinator/hosts/rondel/paths/_storage_emulated_0
execv: No such file or directory
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [Receiver=3.2.3]

• On the phone, using rsync to push files to the server via its rsyncd: WORKS

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel
:/ # rsync -a --progress --delete --password-file=/storage/emulated/0/rsync_backup_crucible /storage/emulated/0/ rsync://[email protected]/backup/redundinator/hosts/rondel/paths/_storage_emulated_0
sending incremental file list
[normal rsync progress output]

• On the phone, using rsync to push files to the server via ssh: WORKS

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel
:/ # rsync -a --progress --delete -e 'ssh -i /data/smbrw_key -p 22' /storage/emulated/0/ [email protected]:/mnt/backup/redundinator/hosts/rondel/paths/_storage_emulated_0
Could not create directory '/.ssh'.
The authenticity of host '192.168.7.94 (192.168.7.94)' can't be established.
ECDSA key fingerprint is SHA256:[random stuff here].
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/.ssh/known_hosts).
sending incremental file list
[normal rsync progress output]

• On the server, using a command that opens ssh to the phone and runs a command there to rsync files back to the server (connecting to rsyncd OR ssh): ERROR, "execv: No such file or directory"

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel "rsync -a --progress --delete --password-file=/storage/emulated/0/rsync_backup_crucible /storage/emulated/0/ rsync://[email protected]/backup/redundinator/hosts/rondel/paths/_storage_emulated_0"
execv: No such file or directory

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel "rsync -a --progress --delete -e 'ssh -i /data/smbrw_key -p 22' /storage/emulated/0/ [email protected]:/mnt/backup/redundinator/hosts/rondel/paths/_storage_emulated_0"
execv: No such file or directory

• On the server, using a command that opens ssh to the phone and runs a command there to echo text: WORKS

aeheathc@crucible ~ $ ssh -i /home/aeheathc/.ssh/id_rsa -p 2222 -l backuppc rondel "echo blah"
blah

• From what I understand, the error is execv saying it can't find an executable at the path that was passed into it. But what was that path? I'd need to know that to investigate further on my own.
• /var/log/auth.log contains only successful connections, nothing enlightening.
• sudo before any of the commands doesn't seem to help.

Answer 1

What probably happens is that, when you ssh into your phone directly, PATH gets fully populated and includes the path to rsync. However, different initialization files are run (e.g., /etc/profile, ~/.profile, ~/.bashrc) in different scenarios. If you run ssh «destination» -- «command», ssh runs the shell in non-interactive mode. It seems that this means that the profile scripts won’t run, just the rc ones. For bash, that means that ~/.bashrc runs and ~/.bash_profile doesn’t. See how this output is different for me:

ohnobinki@gibby ~ $ ssh localhost -- "echo \${PATH}"
/usr/bin:/bin:/usr/sbin:/sbin
ohnobinki@gibby ~ $ ssh localhost -- "sh -lc \"echo \\\${PATH}\""
/usr/x86_64-pc-linux-gnu/gcc-bin/9.1.0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/games/bin

The -l option asks sh to treat the shell as a login session, including sourcing /etc/profile and other things.

You might be able to either:

• Specify --rsync-path=/path/to/rsync (discover this using which rsync in a normal interactive ssh session) like:

  rsync -a --progress --delete -e "ssh -i /home/aeheathc/.ssh/id_rsa -p 2222" --rsync-path=/usr/local/bin/rsync backuppc@rondel:/storage/emulated/0/ /backup/redundinator/hosts/rondel/paths/_storage_emulated_0
  

  This method is easy to understand. However, it feels a bit gross. It shouldn’t be necessary for the person running the rsync command to know where rsync is installed on the phone—this breaks encapsulation. It is also more fragile because, if the path to rsync on the phone changes, this command will need to be updated too.

• Use a lot more quoting and let the shell run its interactive initialization which should get rsync into path (yes, I typed rsync twice in --rsync-path because sh -c lets you specify argv[0], so I have to supply that to avoid argv[1] being gobbled up):

  rsync -sa --progress --delete -e "ssh -i /home/aeheathc/.ssh/id_rsa -p 2222" --rsync-path="sh -lc \"rsync \\\"\\\${@}\\\"\" rsync" backuppc@rondel:/storage/emulated/0/ /backup/redundinator/hosts/rondel/paths/_storage_emulated_0
  

  This option is a lot more confusing and tricky, but seems nicer to me. This way, we can let the phone know where rsync is located instead of the command running on the server. It seems to respect encapsulation and be less fragile/more generalized.

• Or edit your .bashrc or equivalent file on your phone to add the necessary path containing rsync to PATH instead of only specifying this in the profile scripts.

  This method feels a bit ugly because the conventions for what different rc/profiles should do indicate that .bashrc should not set environment variables but should instead do lightweight things and be launched by a login shell. Unfortunately, it doesn’t seem like the way that ssh launches a non-interactive shell remotely was taken into consideration when these conventions were developed. Just note that .bashrc runs a lot, so you should be careful to not blindly append to PATH—you might end up with a needlessly long PATH which hopefully won’t harm anything but might be wasteful of resources or cause issues in certain extreme situations.