11

Is it possible to rsync a real filesystem (remote) with a tar archive (local)? If so, how?

Problem is I need to correctly backup user/group/permission settings and, while I have root access on remote I would like to avoid running as root on the local machine.

My first (and foremost) usage case would be to update a remote embedded target (ARM9) from a .tar produced by Buildroot. I do not have the "real thing" on disk (I can produce a copy while being root) and I would like to avoid transferring the whole rootfs to update a few files.

2
  • No, you can not sync files directly into/from a tar archive with rsync. How about just extracting the tar file into a directory hierarchy, then running rsync against that, and finally putting everything back into a tar archive again? That would save you from transferring files, but does require some fiddling with TAR files, of course.
    – user260419
    Commented Jan 12, 2016 at 16:26
  • @SamiLaine: as said my problem with that approach is I need to preserve user/group/permission settings and that is only possible for root. I am currently looking into using fakeroot, which should be exactly what I need. I will post my solution as "Answer", if it actually works.
    – ZioByte
    Commented Jan 12, 2016 at 16:32

2 Answers 2

4

I haven't actually tried this, but it should work.

Using 'archivemount' (source from:)

http://www.cybernoia.de/software/archivemount/

and a 'libarchive' included in many distros (suse, redhat, etc)...

Or a pre-built one from:

https://rpmfind.net/linux/rpm2html/search.php?query=archivemount

You can mount a tar-archive using the fusermount facility in linux.

From there, you should be able to use rsync directly to the final system.

I wrote a simple passthrough batchfile to test rsync's passthrough:

#!/bin/bash
# ussh -- use root@ssh to target system
exec ssh  root@"$@"

then, as a test, used rsync to pass dir 'test1' to 'ishtar', calling it /tmp/test2 on the target:

RSYNC_RSH=$PWD/Ussh rsync -uva /tmp/test1/ ishtar:/tmp/test2

It will ask you for the password of the target sys's root logon, or you could setup the target system to accept a root login via a certificate so no password would be needed.

This would seem to be the most efficient way to do what you want (you might need to modify the rsync options to not copy dir times and things like that), but is this the type of thing you were looking for?

-Astara

7
  • I will check as soon as I find a bit of time, as it looks much more efficient than current method; just one perplexity: If I actually "mount" the tar archive and "just use" it as normal user won't I be denied access to all files "readable from root only"? There are many of those in a rootfs, even excluding log files (empty by definition).
    – ZioByte
    Commented May 7, 2017 at 9:59
  • The access issue you mention shouldn't be an issue. If there are root-read-only files in the image, then how would 'Buildroot' (created by a user, right?) read the files to put in the tar-image in the first place?
    – Astara
    Commented May 8, 2017 at 22:55
  • 1
    I'm afraid You don't understand the complexity ;) Buildroot produces a full directory mimicking the root filesystem but, obviously, with the ownerships of the current user. When everything is ready it does a pass under fakeroot (see: wiki.debian.org/FakeRoot) to prepare the tar archive with proper ownerships and permissions. This is exactly (in reverse) what I am currently doing to ship everything to target, where I can recreate the uid/gid/permissions because there I am root. Tar mounting via loopback is a nice idea, but I would have to become root to run rsync :(
    – ZioByte
    Commented May 9, 2017 at 0:00
  • Which is true: Buildroot creates a tar-image from a real-root, or Buildroot creates a tar image from a tree that is entirely owned by the user? Does Buildroot use 'fakeroot' to simulate ownership of the files in the user-tree, so Buildroot "sees" files owned by 'root' (or other users) when it is creating the tar image? IF that's the case, then why not rsync from the original user tree that was used to create the 'root' (where all the files are owned by the user), but use fakeroot under 'rsync', so rsync will see the files with the "correct" file perms+ownerships?
    – Astara
    Commented May 10, 2017 at 0:34
  • @ZioByte: If you use fakeroot to create a tar with root-perms, you've created a type of "protected-image" that can't be extracted (keeping perms) w/o root privs and, in the same way, won't be fully accessible if you mount it as an archive. You need to access the files before you've "protected" the archive, then run fakeroot under rsync, so rsync thinks it is transferring root-owned files -- that the the remote rsync (which is running under root) will extract w/the proper permissions.
    – Astara
    Commented May 10, 2017 at 0:39
2

The right answer seems to be to unpack the tar archive using fakeroot (to avoid becoming root) and then use rsync. Possibly repack the archive, if needed.

Unfortunately things are not this easy because bad interaction between ssh and fakeroot. I will detail what I did to help whoever will search.

Theory is straightforward:

  1. create a temp directory
  2. unpack tar archive into it
  3. rysnc as needed
  4. if something changed locally repack int a new tar archive
  5. cleanup

In order to preserve all user/group/permissions steps 2..4 must be done under fakeroot.

Catch is rsync uses ssh for communication (and I want it to!) and thus, being "fakerooted" it tries to open root credentials (in /root/.ssh/), failing badly. The following set of options work for me.

#!/bin/bash

target=myHost
here=$(pwd)

# 1. create a temp directory
cd /tmp
mkdir TMPfs
cd TMPfs

fakeroot bash <<- EOF
    # 2. unpack tar archive into it
    tar xf $here/archive.tar
    # 3. rysnc as needed (ssh options are *the* relevant thing)
    rsync -av -e "ssh -i $HOME/.ssh/id_rsa -oUserKnownHostsFile=$HOME/.ssh/known_hosts" . root@$target:/
    # 4. if something changed locally repack int a new tar archive (not needed here)
EOF

# 5. cleanup
rm -rf *
cd ..
rmdir TMPfs

I still get the error "Could not create directory '/root/.ssh'." but it appears to be benign (no files are created/used there).

5
  • What do you mean "unpack tar using fake root"? Are permissions a problem? I.e. I guess I assumed that you were transferring a tar archive of your files via rsync to your directory on another system. If you are talking about preserving permissions, does the tar you use even copy / preserve meta data other than UID/GID's? rsync can transfer ACL's and other extended attributes (metadata), but most current/standard tar programs won't store that info, So presumption was you didn't need it. Why are you packing into a tar archive? --- if something changes locally, why not rsync directly?
    – Astara
    Commented May 1, 2017 at 3:17
  • Otherwise, you could transfer the tar-archive and unpack it over the target and use the --keep-newer-files switch, which says don't replace existing files that are newer than the version in the tar archive. But if you just have a few files, not sure why you won't just rsync directly. If you own files on both ends, don't see why you would need fakeroot. Conversely, if can become root on both systems, just use 'rsync' as root. If you use ssh, just create a key on source system(as root), and store that in remote-root's .ssh dir. Then you can login w/no password (if ssh config'ed that way).
    – Astara
    Commented May 1, 2017 at 3:23
  • @Astara: The tar is the complete root-filesystem as packed by Buildroot; it is complete with GID/UID/permissions as needed and is packed into a GNUtar archive by the build system itself. I don't really want to meddle with it. Since it may be quite sizeable and target may be very remote I am using rsync to update target with minimal transfer time. I will need to become root on target in order to overwrite system files, but that may not be the case on the development system.
    – ZioByte
    Commented May 5, 2017 at 17:36
  • BTW, rsync uses 'rsh' by default which has lower security but also has lower security requirements.
    – Astara
    Commented May 13, 2019 at 21:20
  • You can probably avoid the error/warning about /root/.ssh by specifying the config file with the -F option. That should prevent ssh from trying to access/create the per user config file. As a bonus(?) you can then move your other ssh options into that file.
    – Eph
    Commented Jul 27, 2023 at 14:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .