3

I would like to one way synchronise big directory between two machines without direct connection. Either using some cloud storage (without computing power) connected to both machines or an intermediate storage device (like hdd, flash drive, not another machine) that can be connected to machines alternately.

What is important to note is that there is not enough space on intermediate storage for the whole folder, though enough for the biggest file to transfer.

I imagine the flow to be:

  1. On destination machine: Create snapshot of destination folder and put it on shared storage.
  2. On source machine: Compare structure of source folder with the snapshot from shared storage and put all missing files with meta information to the shared storage (limited by available space on shared storage).
  3. On destination machine: Copy files from the shared storage and tell if flow should be repeated.

Is there an already existing solution to do something like this from macOS/Linux shell?

Improve this question
8
  • @JakeGould I've edited the question to hopefully be more clear, the point is that the storage (cloud or device) is not a machine, so can be only used to store data, not execute commands, also cloud device can be connected to both, but media only alternatively to one of the machines
    – tig
    Commented Jun 26, 2019 at 15:56
  • @KamilMaciorowski Exactly. For my task I need only one way synchronisation, but if there exists a tool, it probably can do both one way and two way sync
    – tig
    Commented Jun 27, 2019 at 9:17
  • In theory btrfs-send and btrfs-receive can clone a btrfs subvolume to another btrfs filesystem. There is an incremental mode that relies on identical snapshots already present in both filesystems. Data you need to transfer is a bytestream, so you can save it on a flashdrive (possibly after compressing, whatever). This approach won't merge changes but with careful planning (or firm scripting) I think you can propagate changes back and forth. I have never tried this, I don't know how the approach performs, therefore it's not an answer. Start from man 8 btrfs-send, man 8 btrfs-receive.
    – Kamil Maciorowski
    Commented Jun 27, 2019 at 9:40
  • The biggest problem I see is how one side should figure out which files to sync to the other side when there's no direct connection to the other side. Is there some kind of local history on each side (e.g., could you make each side a git repository)? Or is the only use case "file present or not present", and "file is changed" never happens? Then you could just package off the changes and send them to the other side. Otherwise it's going to be difficult.
    – dirkt
    Commented Jun 28, 2019 at 7:23
  • @dirkt Your question is relevant only for two way sync, which is much more complicated topic. My question is about one way sync, so keeping folder on machine B identical to folder on machine A
    – tig
    Commented Jul 6, 2019 at 11:34

1 Answer 1

Reset to default
2

Just use Rsync on the intermediate device.

Rsync is perfectly suited for a task like this: Make the source the source machine and the destination the destination machine and there you go.

The command would be something like this:

rsync -avz user@source_machine.local:/source/directory/ user@destination_machine.local:/destination/directory/

This should work without any concerns about storage overhead on the intermediate machine since—when using Rsync—it all becomes a networking task that just utilizes CPU and networking resources on that intermediate machine.

But both of these options require that the source and destination machines have some kind of OS running on them. A lone piece of cloud-based “dumb” storage or a simple external storage device like a USB flash drive cannot be “bridged” without some OS between them.

You ultimately need an OS on both sides of the equation. And perhaps an intermediate device can mount the cloud device and then mount the external storage device like a USB flash drive and then you can just run Rsync and you are in business.

Improve this answer
1
  • 1
    Unfortunately not a solution, as the intermediate device is not a machine itself, so is only storage
    – tig
    Commented Jun 27, 2019 at 13:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .