Rsync files between two remotes servers using SSH

Question

Summary

I want to copy files between 2 remote servers using Rsync and SSH. The way I found to do this is creating a SSH tunnel, but I tried with several examples and any of those worked.

Details

• I have 3 machines: 1 local computer and 2 remote servers
• I can SSH to both remote servers using my local RSA key.
• The 2 remote servers cannot talk to each other. Remote #1 does not have access to Remote #2 and viceversa.
• I can Rsync to both remote servers individually

What I have tried

ssh -vvv -A \
-l user_remote1 \
-R localhost:50000:remote2:22 user_remote1@remote1 \
'rsync -e \
"ssh -A -o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
-o GlobalKnownHostsFile=/dev/null \
-p 50000" \
-var /home/user_remote1/test.txt user_remote2@localhost:/home/user_remote2'

Output of that command

OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug2: resolve_canonicalize: hostname remote1 is address
debug2: ssh_connect_direct
debug1: Connecting to remote1 [remote1] port 22.
debug1: Connection established.
debug1: identity file /Users/user_local/.ssh/id_rsa type 0
debug1: identity file /Users/user_local/.ssh/id_rsa-cert type -1
debug1: identity file /Users/user_local/.ssh/id_dsa type -1
debug1: identity file /Users/user_local/.ssh/id_dsa-cert type -1
debug1: identity file /Users/user_local/.ssh/id_ecdsa type -1
debug1: identity file /Users/user_local/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/user_local/.ssh/id_ed25519 type -1
debug1: identity file /Users/user_local/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/user_local/.ssh/id_xmss type -1
debug1: identity file /Users/user_local/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to remote1:22 as 'user_remote1'
debug3: hostkeys_foreach: reading file "/Users/user_local/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/user_local/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from remote1
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:6gPivXbLGMtZsqEIwH4713sH1S/p/pQfHJ8+DoTsqc8
debug3: hostkeys_foreach: reading file "/Users/user_local/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/user_local/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from remote1
debug1: Host 'remote1' is known and matches the ECDSA host key.
debug1: Found key in /Users/user_local/.ssh/known_hosts:4
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o
debug1: Will attempt key: /Users/user_local/.ssh/id_dsa 
debug1: Will attempt key: /Users/user_local/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/user_local/.ssh/id_ed25519 
debug1: Will attempt key: /Users/user_local/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/user_local/.ssh/id_rsa RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o
debug3: sign_and_send_pubkey: RSA SHA256:TRou12Aso7/pS0D7blnRJsw9YHeFmDjfPIgkrgu2l4o
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to remote1 ([remote1]:22).
debug1: Remote connections from localhost:50000 forwarded to local address remote2:22
debug3: send packet: type 80
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/user_remote1/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/user_remote1/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 81
debug1: remote forward success for: listen localhost:50000, connect remote2:22
debug1: All remote forwarding requests processed
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request [email protected] confirm 0
debug3: send packet: type 98
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env TMPDIR
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env OLDPWD
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env USER
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env PATH
debug3: Ignored env PWD
debug3: Ignored env XPC_FLAGS
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env LOGNAME
debug1: Sending env LC_CTYPE = UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env _
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug1: Sending command: rsync -e \\
"ssh -A -o StrictHostKeyChecking=no \\
-o UserKnownHostsFile=/dev/null \\
-o GlobalKnownHostsFile=/dev/null \\
-p 50000" \\
-var /home/user_remote1/test.txt user_remote2@localhost:/home/user_remote2
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 4 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 50000, originator ::1 port 56418
debug2: fd 7 setting O_NONBLOCK
debug2: fd 7 setting TCP_NODELAY
debug1: connect_next: host remote2 ([remote2]:22) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [::1]
debug1: confirm forwarded-tcpip
debug3: channel 1: waiting for connection
debug1: channel 1: connected to remote2 port 22
debug3: send packet: type 91
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype [email protected] rchan 5 win 65536 max 16384
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 2: new [authentication agent connection]
debug1: confirm [email protected]
debug3: send packet: type 91
debug2: channel 0: rcvd ext data 84
Warning: Permanently added '[localhost]:50000' (ECDSA) to the list of known hosts.
debug2: channel 0: written 84 to efd 6
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i0 o1 sock 8 wfd 8 efd -1 [closed])
debug2: channel 2: output drain -> closed
debug1: channel 2: FORCE input drain
debug2: channel 2: ibuf empty
debug2: channel 2: send eof
debug3: send packet: type 96
debug2: channel 2: input drain -> closed
debug2: channel 2: send close
debug3: send packet: type 97
debug3: channel 2: will not send data after close
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: authentication agent connection, nchannels 3
debug3: channel 2: status: The following connections are open:
  #0 client-session (t4 r2 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)
  #1 ::1 (t4 r4 i0/0 o0/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)
  #2 authentication agent connection (t4 r5 i3/0 o3/0 e[closed]/0 fd 8/8/-1 sock 8 cc -1)

debug2: channel 0: rcvd ext data 38
Permission denied, please try again.
debug2: channel 0: written 38 to efd 6
debug2: channel 0: rcvd ext data 38
Permission denied, please try again.
debug2: channel 0: written 38 to efd 6
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd ext data 229
debug3: receive packet: type 96
debug2: channel 1: rcvd eof
debug2: channel 1: output open -> drain
debug2: channel 1: obuf empty
debug2: channel 1: chan_shutdown_write (i0 o1 sock 7 wfd 7 efd -1 [closed])
debug2: channel 1: output drain -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: obuf_empty delayed efd 6/(229)
user_remote2@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
debug2: channel 0: written 229 to efd 6
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 2
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r2 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1)
  #1 ::1 (t4 r4 i0/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)

debug2: channel 1: read<=0 rfd 7 len 0
debug2: channel 1: read failed
debug2: channel 1: chan_shutdown_read (i0 o3 sock 7 wfd 7 efd -1 [closed])
debug2: channel 1: input open -> drain
debug2: channel 1: ibuf empty
debug2: channel 1: send eof
debug3: send packet: type 96
debug2: channel 1: input drain -> closed
debug2: channel 1: send close
debug3: send packet: type 97
debug3: channel 1: will not send data after close
debug3: receive packet: type 97
debug2: channel 1: rcvd close
debug3: channel 1: will not send data after close
debug2: channel 1: is dead
debug2: channel 1: garbage collecting
debug1: channel 1: free: ::1, nchannels 1
debug3: channel 1: status: The following connections are open:
  #1 ::1 (t4 r4 i3/0 o3/0 e[closed]/0 fd 7/7/-1 sock 7 cc -1)

debug3: send packet: type 1
debug3: fd 1 is not O_NONBLOCK
Transferred: sent 6168, received 6820 bytes, in 2.5 seconds
Bytes per second: sent 2488.0, received 2750.9
debug1: Exit status 255

I have searched about several related topics from that output like searching about "rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]", "ssh debug1: Next authentication method: password" or "read_passphrase: can't open /dev/tty: No such device or address", but could not find anything useful.

Is there something wrong with the command or maybe I need a custom configuration in the servers? Any feedback will be useful. Thanks.

Answer 1

You're using ssh -A. I understand you want ssh run by rsync on remote1 to reach your local ssh-agent. You're hoping to get authorized on remote2 with your local key this way. The plan is good.

The verbose output you published tells me ssh-agent is working locally and ssh run by rsync on remote1 can access it. Still you get Permission denied (publickey,…. The (tunneled) SSH connection from remote1 to remote2 fails to authenticate and closes; rsync cannot continue (rsync: connection unexpectedly closed).

This strongly suggests the local agent does not hold the right key. man 1 ssh-agent reads:

The agent initially does not have any private keys. Keys are added using ssh-add(1) or by ssh(1) when AddKeysToAgent is set in ssh_config(5).

One way to add a key to an already started ssh-agent is with ssh-add before you invoke ssh. Since you want to use the same key your local ssh is going to use, AddKeysToAgent should work as well. You don't need to modify ssh_config, you can pass the option directly in the (local) ssh invocation:

ssh -A -o AddKeysToAgent=yes …

(where … indicates the rest of the command you already have).

Note this requires ssh-agent to run and SSH_AUTH_SOCK to be in the environment, so ssh can communicate with the agent. It seems you have already taken care of this.

Users with similar problem who know little about ssh-agent and how to start it properly (or they are not sure if ssh-agent has been started properly by their OS, by themselves or whatever) may prefer the following command:

ssh-agent ssh -A -o AddKeysToAgent=yes …

It starts a separate (new) ssh-agent. The agent will start ssh in the right environment, so the two can communicate. There are scenarios where this form is particularly useful (example). Here the form is not a must, however it's independent of any agent previously started (or not started at all) and should work out of the box, unless there is some problem with the key(s). So if you're not familiar with ssh-agent then the last command is the best choice. ssh-agent started this way will terminate after ssh exits.

In the OP's case there are few minor improvements to make:

• ssh -l user_remote1 user_remote1@remote1 is redundant, it specifies user_remote1 twice. There are few methods to specify the user and the server. Using more than one method in a single invocation is hardly ever useful (see an edge case when it's kinda useful).

• I think nothing on remote2 needs to access the local ssh-agent. -A in the ssh command rsync (on remote1) is going to use is therefore superfluous. The command will work either way, but mind what man 1 ssh states:

  Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

  You need -A when you ssh from local to remote1; an attacker sitting on remote1 is the risk you take. The superfluous -A when connecting to remote2 is only an opportunity for an attacker sitting on remote2.