Can't do an SCP pull despite SSH working fine

Question

I have setup key authentication between two servers and I can ssh in without a key, for example:

ssh backup@hostname

this works fine and logs me in. But when I try an SCP to pull a file, I get no files.

The target files have chmod 777 done to them for troubleshooting purposes but it's like it just doesn't find any files to copy even though they are there.

Here is the (ip's obfuscated) output of

scp -vvv backup@hostname:/var/stuff/backups/*.tgz /data/backups/

from the point it authenticates onward.

debug1: Authentication succeeded (publickey).    
Authenticated to xxx.xxx.xxx.xxx ([xxx.xxx.xxx.xxx]:22).
debug2: fd 4 setting O_NONBLOCK    
debug2: fd 5 setting O_NONBLOCK    
debug2: fd 6 setting O_NONBLOCK    
debug1: channel 0: new [client-session]    
debug3: ssh_session2_open: channel_new: 0    
debug2: channel 0: send open    
debug1: Entering interactive session.    
debug2: callback start   
debug2: fd 3 setting TCP_NODELAY    
debug3: packet_set_tos: set IP_TOS 0x08    
debug2: client_session2_setup: id 0    
debug1: Sending environment.    
debug3: Ignored env XDG_SESSION_ID    
debug3: Ignored env TERM    
debug3: Ignored env SHELL    
debug3: Ignored env SSH_CLIENT    
debug3: Ignored env SSH_TTY    
debug3: Ignored env http_proxy    
debug3: Ignored env USER    
debug3: Ignored env LS_COLORS    
debug3: Ignored env MAIL    
debug3: Ignored env PATH   
debug3: Ignored env QT_QPA_PLATFORMTHEME    
debug3: Ignored env PWD    
debug1: Sending env LANG = en_AU.UTF-8    
debug2: channel 0: request env confirm 0    
debug3: Ignored env SHLVL    
debug3: Ignored env HOME    
debug3: Ignored env LANGUAGE    
debug3: Ignored env LOGNAME    
debug3: Ignored env SSH_CONNECTION    
debug3: Ignored env LESSOPEN    
debug3: Ignored env XDG_RUNTIME_DIR    
debug3: Ignored env LESSCLOSE    
debug3: Ignored env _    
debug3: Ignored env OLDPWD    
debug1: Sending command: scp -v -f /var/stuff/backups/*.tgz    
debug2: channel 0: request exec confirm 1    
debug2: callback done    
debug2: channel 0: open confirm rwindow 0 rmax 32768    
debug2: channel 0: rcvd adjust 131072    
debug2: channel_input_status_confirm: type 99 id 0    
debug2: exec request accepted on channel 0    
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0    
debug2: channel 0: rcvd eof    
debug2: channel 0: output open -> drain    
debug2: channel 0: obuf empty    
debug2: channel 0: close_write    
debug2: channel 0: output drain -> closed    
debug2: channel 0: rcvd close    
debug2: channel 0: close_read    
debug2: channel 0: input open -> closed    
debug3: channel 0: will not send data after close    
debug2: channel 0: almost dead    
debug2: channel 0: gc: notify user    
debug2: channel 0: gc: user detached    
debug2: channel 0: send close    
debug2: channel 0: is dead    
debug2: channel 0: garbage collecting    
debug1: channel 0: free: client-session, nchannels 1   
debug3: channel 0: status: The following connections are open:    
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
debug1: fd 0 clearing O_NONBLOCK   
debug1: fd 1 clearing O_NONBLOCK    
debug1: fd 2 clearing O_NONBLOCK    
Transferred: sent 3736, received 2280 bytes, in 0.0 seconds    
Bytes per second: sent 764498.2, received 466556.7    
debug1: Exit status 1    
backup@ar-secubn03:/var/scripts$ 

Answer 1

Try this:

scp -vvv 'backup@hostname:/var/stuff/backups/*.tgz' /data/backups/

If that doesn't work try this:

scp -vvv 'backup@hostname:/var/stuff/backups/\*.tgz' /data/backups/

Answer 2

I was having a similar issue that I could exercise ssh commands, but not scp commands. I went through Kenster good debug techniques which still didn't get me anywhere, but very handy! I finally found out the issue was related to echo commands in my .bashrc script (after inserting additional ones). Once I commented out all the echo commands in the script I was able to successfully execute the scp command.

Answer 3

debug1: Sending command: scp -v -f /var/stuff/backups/*.tgz    
debug2: channel 0: request exec confirm 1    
debug2: callback done    
debug2: channel 0: open confirm rwindow 0 rmax 32768    
debug2: channel 0: rcvd adjust 131072    
debug2: channel_input_status_confirm: type 99 id 0    
debug2: exec request accepted on channel 0    
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
...
debug1: Exit status 1

These lines indicate that the local scp instance requested the expected scp command to run on the remote system, and the remote server launched a process in response to the request. But the remote process exited with status code 1 almost immediately, and without emitting any output.

If the remote scp instance couldn't read the files to be copied, or if those files didn't exist, you would have gotten an error message to that effect. If the remote scp program couldn't be found or wasn't executable, then the exit code would have been 126 or 127 instead of 1.

I suspect one of these things is happening:

1. Something in the remote account's .bashrc or .bash_profile is causing the shell to exit early. It may be sensitive to the fact the session doesn't have a TTY, for example.
2. The scp program on the remote system is corrupt or malfunctioning.
3. The scp program on the remote system has been replaced with something else which isn't working like scp. Someone might have replaced it with a poorly written shell script, for example, or a copy of '/bin/false'.
4. The SSH software has been configured to block people from running scp. The key which you're using may be set in authorized_keys to run a specific command, or there may be a ForceCommand directive in the remote server's sshd_config file.

The simplest way to check these would be to ask the remote system administrator, or log into the remote system and inspect the scp executable. Running scp -h will print an scp-specific usage message, for example.

If you have to troubleshoot this remotely, the next thing I'd try is something like this:

$ ssh -T backup@hostname cat /etc/group

This should write the remote system's group file to your terminal. This would prove that you can run arbitrary commands on the remote system.

$ ssh -T backup@hostname scp -v -f /etc/group < /dev/zero

This simulates the remote part of an scp session. If the remote scp program is working, you should get output something like this:

C0644 674 group
root:x:0:
daemon:x:1:
[...]
devuser:x:1001:
Sending file modes: C0644 674 group
$ 

If all of that works, the question becomes why scp fails for "/var/stuff/backups/*.tgz" but works for "/etc/group". You could run this to confirm whether it's working or not:

ssh -T backup@hostname 'scp -v -f /var/stuff/backups/*.tgz' < /dev/zero

If that continues to fail, the nuclear option would be to arrange to run the remote session under strace and see exactly what each program is doing.

Answer 4

Your problem is clearly identified in these lines:

debug2: exec request accepted on channel 0    
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0    
debug2: channel 0: rcvd eof    

which shows your channel is shut down as soon as it is opened, though the reason is not stated.

Normally,this occurs because of a faulty configuration in the /etc/passwd file, see for instance this Server Fault post. You may follow the suggestions in this post, or, better still, you may check the logs on the target machine,

 tail /var/log/auth.log

for further details.

Answer 5

I think you do 1st backups/* .tgz to change backups*.tgz make sure don't use any space. might be 99% issue solved.

Answer 6

I had the same bad experience. I slight modification of the .bashrc on the remote system was the problem. I once added 'ulimit' and got the this problem during the next scp (ssh worked). After removing this line all was fine.

Answer 7

I've got a similar problem : impossible to use SCP between two VMs under CENTOS whereas SSH key was exhanged, and ssh works fine...

But, after weeks...it appeared by both CENTOS does not use the same SHELL. One was with .tcshrc, whereas the other was this .bashrc Consequently, the SCP command was always misinterpreted by the host.

I aligned both SHELL on both VMs, and now, it works fine !