Coerce scp(1), sftp(1), or rsync(1) into transferring a named pipe

Question

I intend to transfer full and incremental backups of my btrfs subvolumes to a tape archival service. The service exposes FTP and SSH endpoints. If I were allowed to execute arbitrary commands on the SSH endpoint, then I would do the following to perform an incremental backup:

btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt |
    ssh -p $PORT $USER@$ENDPOINT "cat > $SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg"

I am, however, not allowed to do that:

$ ssh -p $PORT $USER@$ENDPOINT
Last login: Mon Jan  3 01:23:45 2067 from 123.456.789.123

This account is restricted by rssh.
Allowed commands: scp sftp rsync 

If you believe this is in error, please contact your system administrator.

Connection to some.remote.endpoint closed.

So what I thought to do instead was to use the SCP protocol for the transfer. However, my scp binary refuses to transfer a named pipe:

$ scp -P $PORT <(btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt) \
    $USER@$ENDPOINT:$SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg
/dev/fd/63: not a regular file

The irony is that apparently, scp used to do the right thing once. I suppose not everybody thought transferring named pipes could be sane / useful. EDIT (2017-06-03): This was an incorrect observation. As noted by Kenster, the SCP protocol does not permit sending files of unknown size.

UPDATE (2017-06-04): I also tried transferring the data using the SFTP protocol. Apparently, the FTP protocol permits sending files of unknown size, as evidenced by the support for piping data in the ftp (link) and ncftpput (link, section Description, last paragraph) binaries. I found no such support in the SFTP clients I have tried (sftp, lftp), which might be an indication that SFTP (unlike FTP) does not support sending files of unknown size (contrary to what I thought, SFTP is not just FTP tunelled over SSH; it is a different protocol).

UPDATE (2017-06-05): According to the SFTP protocol version 3 internet draft (link), each application-level packet must specify the payload length before the payload (link, Section 3). However, SFTP supports seeks in the written file (link, Section 6.4) with explicit support for writes beyond the current end of file. Therefore, if should be possible to use a small buffer on the client side and send a file of unknown size in small known-sized chunks:

#!/bin/bash
# <Exchange SSH_FXP_INIT requests.>
# <Send an SSH_FXP_OPEN request.>
CHUNK_SIZE=32768
OFFSET=0
IFS=''; while read -r -N $CHUNK_SIZE CHUNK; do
  ACTUAL_SIZE=`cat <<<"$CHUNK" | head -c -1 | wc -c`
  # <Send an SSH_FXP_WRITE request with payload $CHUNK of size
  # $ACTUAL_SIZE at offset $OFFSET.>
  OFFSET=$(($OFFSET+$ACTUAL_SIZE))
done < <(command)
# <Send an SSH_FXP_CLOSE request.>

However, doing the communication manually over the shell would be quite painful. I am looking for an SFTP client that exposes this kind of functionality.

References

• Can you scp, sftp, or rsync, a pipe? — Unix & Linux Stack Exchange
• ssh - how to pipe data to sftp connection? — Server Fault
• Using FTP with pipes | KrazyWorks
• ncftpput(1) manual page
• SSH File Transfer Protocol

Answer 1

lftp 4.6.1 and newer should be able to do this: https://github.com/lavv17/lftp/issues/104

Unfortunately, the command suggested in the linked issue does not work, but a slightly-modified one does:

lftp -p $port sftp://$user:$pass@$host -e "put /dev/stdin -o $filename"

Due to a bug, you must provide something in the password field if you use a SSH agent. But any random string will do.

This command reads from /dev/stdin. If you must use a different named pipe, I imagine that should work too - if not, you can always cat named-pipe | lftp ....

I've tested lftp with a >100GB upload without any issues.

---

rclone also supports SFTP, and should be able to upload from a piped input with the rcat command, due to be released in 1.38 (next version).

Answer 2

SCP isn't well-suited for your purpose. The SCP protocol doesn't support sending an unknown-sized stream of bytes to the remote system to be saved as a file. The SCP protocol message for sending a file requires the size of the file to be sent first, followed by the bytes that make up the file. With a stream of bytes read from a pipe, you typically wouldn't know how many bytes the pipe is going to produce so there's no way to send an SCP protocol message including the correct size.

(The only online descriptions of the SCP protocol that I could find are here and here. Focus on the description of the "C" message.)

The SFTP protocol can be used for this sort of thing. As far as I know, the normal sftp command-line utility doesn't support reading a pipe and storing it as a remote file. But there are SSH/SFTP libraries for most modern programming languages (perl, python, ruby, C#, Java, C, etc). If you know how to use one of these languages, it should be straightforward to write a utility that does what you need.

If you're stuck with shell scripting, it's possible to spoof enough of the SCP protocol to transfer a file. Here's an example:

#!/bin/bash
cmd='cat /etc/group'

size=$($cmd | wc -c)    
{
        echo C0644 $size some-file
        $cmd
        echo -n -e '\000'
} | ssh user@host scp -v -p -t /some/directory

This will create some-file in /some/directory on the remote system with permissions 644. The file contents will be whatever $cmd writes to its standard output. Note that you are running the command twice, with whatever resource consumption and side effects that implies. And the command must output the same number of bytes each time.

Answer 3

If you can write perl, you could try using Net::SFTP::Foreign which is able to transfer files from an open file descriptor via SFTP:

#!/usr/bin/perl
# untested!
use Net::SFTP::Foreign;
my $user = ...;
my $host = ...;
my $remote_path = ...;
my $btrfs_cmd = 'btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME';
my $sftp = Net::SFTP::Foreign->new("$user\@$host", ...);
open my($pipe), "$btrfs_cmd | compress | encrypt |"
    or die $!;
$sftp->put($fh, $remote_path);