GNU parallel & RSync via intermediate server

Question

I'm trying to spawn multiple rsync instances with gnu parallel to download via an intermediate server. This is the command I use -

cat ~/filelist | parallel --will-cite -j 5 --verbose --progress --line-buffer --keep-order rsync -avrhP -e "ssh -J intermediateUserID@intermediateIP" sourceUserID@sourceIP:{} ~/scratch/quicktest/

But I get the following error -

rsync error: syntax or usage error (code 1) at main.c(1354) [sender=3.1.3] local:0/728/100%/0.0s rsync -avrhP -e ssh -J intermediateUserID@intermediateIP sourceUserID@sourceIP:/source/directory/path /destination/path/ local:1/728/100%/0.0s Unexpected remote arg: sourceUserID@sourceIP:/source/directory/path

Seems like gnu parallel for some reason removes the quotes after -e. How can I make it include the quotes and basically make the proxy jump work with GNU parallel and rsync?

Answer 1

Bash is parsing your command line and feeding ssh -J intermediateUserID@intermediateIP as a single 11th argument to parallel. In turn, parallel is passing that string unquoted as several arguments to rsync. What you likely want is to pass that argument with internal quotes.

For example:

cat ~/filelist \
    | parallel --will-cite -j 5 --verbose --progress --line-buffer --keep-order \
        rsync -avrhP \
        -e "'ssh -J intermediateUserID@intermediateIP'" \
        sourceUserID@sourceIP:{} \
        ~/scratch/quicktest/

For reference, your command line was: cat ~/filelist | parallel --will-cite -j 5 --verbose --progress --line-buffer --keep-order rsync -avrhP -e "ssh -J intermediateUserID@intermediateIP" sourceUserID@sourceIP:{} ~/scratch/quicktest/.

P.S. Your command should work if you're using passphrase-less public key authentication (and have already accepted and/or bypassed knownhost-files). Otherwise, you'll encounter challenges with parallelized processes (i.e., rsync/ssh) wanting input and, with --tty, parallel assumes also -j1 -u. To overcome that challenge, consider using pssh instead of parallel w. rsync or consider integrating sshpass into your existing command-line.