136

I need to be able to copy about 2TB of data from an external drive (single USB external disk) to a another external drive (Drobo attached via Firewire).

Finder is not an option. If it hits any problem, it stops the process and I have to figure it out why it failed and start over. It could take me months to get through it.

I've read about CP and RSYNC but I'm not too familiar and I'm afraid I might get some of the switches wrong and miss files, or screw up the file dates, or worst...

Can anyone suggest how I might pull this off with minimum risk and cost?

Improve this question
3
  • 1
    Starting with MacOS 10.13.4+, you actually can use Finder to copy huge files (and all the associated links). I just tried it myself to create another Time Machine backup disk (from an existing one) and it works quickly and beautifully. Apparently, before 10.13.4, this was a bug but that's been fixed. More info in this answer: apple.stackexchange.com/a/323691/261070.
    – youngrrrr
    Commented Aug 31, 2018 at 2:48
  • The answer to this depends on how you define "fastest". Do you want the fastest transfer speed or the shortest time between when you (a human) start working on this and finish it. The time it takes you will depend on things like whether the Terminal scares you or not. The ideal answer for you will also depend on whether this is a one-time operation or a commonly repeated one, as well as whether you're willing to spend money or not.
    – iconoclast
    Commented Jun 20, 2019 at 21:36
  • @youngrrrr -- that post says Apple has fixed Finder copy to handle hard links. There is still the problem that if it fails for any reason (power outage, whatever), you cannot pick up where you left off -- Finder won't just transfer the diffs. So this can indeed take many times as long as it should, as you repeatedly start from scratch.
    – TextGeek
    Commented Jul 13, 2022 at 13:54

6 Answers 6

Reset to default
225

remote sync, rsync, is a reliable choice for copying large amounts of data. You can prepare the command and perform a dry-run before committing to the copy; add --dry-run to simulate the copy.

Your final command will be fairly simple:

sudo rsync -vaE --progress /Volumes/SourceName /Volumes/DestinationName

The flags are:

  • v increases verbosity.
  • a applies archive settings to mirror the source files exactly, including symbolic links and permissions.
  • E copies extended attributes and resource forks (OS X only).
  • --progress shows progress during the copy.

sudo, is used to ensure rsync has appropriate rights to access and read all files on your drive regardless of owner. This also allows rsync to write the files to the new drive recreating the original owner information.

rsync is likely the best choice because it can be rerun in case of problems, offers detailed logging, and is as fast as can be while remaining safe.

There are numerous guides for getting the most from rsync, rsync command examples provides relevant examples. Take care with the trailing slashes; these can make a world of difference if your copy starts with a folder.

Alternative tools include ditto and cp. Both are reasonable choices but offer differing syntax.

Improve this answer
2
  • 2
    This answer is complete and well done, makes me feel peace inside
    – Darwin Santos Arismendy
    Commented Dec 21, 2022 at 22:02
  • I use "rsync -alhcuXD --info=progress2,stats", it requires an updated rsync (use brew) but well worth it. there. also, there is no "E", only an "X", but that may be my updated rsync version.
    – Fred Leonard
    Commented Mar 6 at 15:41
10

I answered a similar question here a while back. My answer is copied below.

I use cp -Rfv sourcefile destinationfile with success on a pretty regular basis.

cp = copy

R = maintains file hierarchies

f = if an existing destination file cannot be opened, remove it and try again

v = verbose mode, displays files transferred as it progresses

sourcefile = data you want to copy

destinationfile = directory/drive you want to copy to

Improve this answer
2
  • 15
    It's worth noting however that cp will recopy files even if they haven't changed whereas rsync will not waste time copying files that already exist on the target. Thus you can restart from where you left off and/or do incremental copying.
    – David
    Commented Jul 3, 2015 at 22:53
  • 4
    Can add -n to not copy files that already exist, and -p to preserve all permissions including extended attributes and resource forks where possible.
    – SpinUp __ A Davis
    Commented Feb 27, 2019 at 3:15
8

I used bittorrent sync to sync 8TB between my Drobo and QNAP.

Improve this answer
3
  • 2
    bittorrent sync is a way to do this as well. The downvotes are (in my opinion) not realy needed.
    – CousinCocaine
    Commented Oct 25, 2015 at 13:19
  • 8
    @CousinCocaine yeah, but the question was about how to do something, not what app to use.
    – Dillon
    Commented Jan 1, 2017 at 4:18
  • 6
    @Dillon: knowing what app to use is sometimes 99% of what you need to know to know how to do it.
    – iconoclast
    Commented Jun 18, 2019 at 23:51
6

If you don't want to go the command line route, I use the FreeFileSync app routinely to sync 2 TB of data from an external array to a network location without issue. You can control how it handles errors and get a log when its complete.

Improve this answer
1
  • I am on Mac and used rsync in the past, and while it works, it's feedback is either nonexistent - or way too verbose. I tried a couple of other Mac apps, but none of them were really reliable - so I was very happy to see FreeFileSync, an OpenSourceApp, do a fantastic job - great progress, error handling - and it was superbly fast too. Five stars!
    – patschiboy
    Commented Jun 18, 2020 at 10:15
4

The "fastest" way would be to physically move both drives to be internal to a single computer, do the copy (or rsync), and then move them back.

I'd still use rsync, because if interrupted for any reason (cat steps on the power switch?), rsync will not recopy the files that were already successful. It also won't copy any files that are the same and in the same place.

Improve this answer
7
  • Note that this is faster because of the size of the data set. If it were a lot smaller, the speed advantage would be canceled by the time moving the drives.
    – WGroleau
    Commented Mar 23, 2018 at 15:35
  • No, faster for a large amount of data. For a small amount of data, the copy is still faster, but the time it takes to move the drives cancels that out. However, it's potentially faster under NON-perfect conditions, because if interrupted, rsync is able to avoid copying again the ones that were done before the interruption.
    – WGroleau
    Commented Sep 7, 2018 at 18:12
  • 1
    If the transfer speed is faster but an interruption makes you start over, the entire process is likely to not be faster, and the entire process is what matters. "perfect conditions" = nothing interrupts the process. In my experience it's very common for something to happen to cause a problem when you're dealing with a massive transfer. That's why I refer to an uninterrupted transfer as "perfect".
    – iconoclast
    Commented Sep 13, 2018 at 21:12
  • If you are using rsync as I said, an interruption does not make you start over.
    – WGroleau
    Commented Jun 16, 2019 at 19:17
  • You're not understanding what I mean by "start over" in this context. I'm well aware of the fact that rsync can pick up where it left off in syncing data. But if you issue a command, and it fails to complete, you are starting over with the human intervention, which makes the whole process take longer. Either you waste your time by babysitting the whole thing, or you there's a big delay between the failure and when you discover it. What matters to a human is the time of the entire process, and the time of the human intervention.
    – iconoclast
    Commented Jun 19, 2019 at 0:08
3

If you would like to copy drive A to drive B exactly you can do this with Disk Utility.

We will be using Disk Utility's restore function. Some background on the different between Restoring vs copy and pasting:

The Restore function in Disk Utility makes use of a block copy function that can speed up the copy process. It also makes an almost exact copy of the source device. When we say "almost exact," we don't mean to imply that useful data may get left behind, because that's not the case. What it means is that a block copy copies everything in a data block from one device to the other. The results are almost an exact copy of the original. A file copy, on the other hand, copies data file by file, and while the file data remains the same, the location of the file on the source and destination devices will likely be very different.

Using a block copy is faster, but it does have some limits that affect when it can be used, the most important being that copying block by block requires that both the source and destination devices be first unmounted from your Mac. This ensures that block data doesn't change during the copy process. Don’t worry, though; you don’t have to do the unmounting. Disk Utility’s Restore function takes care of that for you. But it does mean that neither the source nor the destination can be in use when you use the Restore capabilities.

Source: https://www.lifewire.com/use-disk-utility-to-clone-macs-drive-4042367

Instructions

for copying all files from Device A to Device B

WARNING: When you restore one volume to another, all the files on the destination volume are erased. Before you restore a volume, copy any files on the destination volume that you want to save to a different volume.

  1. Choose View > Show All Devices.
  2. Select the volume in the sidebar you want to restore, then click the Restore button enter image description here or choose Edit > Restore.

This is the volume that is erased and becomes the exact copy. [Device B]

  1. Click the Restore pop-up menu, then choose the volume you want to copy [Device A].
  2. Click Restore, then click Done.

Source: https://support.apple.com/guide/disk-utility/restore-a-disk-dskutl14062/mac

Improve this answer
8
  • Block copy also wstees time copying free space and will fail if the destination is one block smaller than the source. (Might fail in a way that isn't detected immediately.) And if the destination is bigger, the extra space becomes unusable.
    – WGroleau
    Commented Sep 13, 2018 at 22:09
  • @WGroleau can you expand on "the extra space becomes unusable"
    – user7886229
    Commented Sep 13, 2018 at 23:53
  • block copy includes the block that says which block is the last. If the destination is big enough for the data but smaller than the source, the operating system thinks there are blocks available that don't exist. If the destination is bigger, the O.S. thinks the extra blocks don't exist.
    – WGroleau
    Commented Sep 13, 2018 at 23:57
  • @WGroleau just so I understand you correctly if I have a 500gb drive with a single 1gb file and block copy to a 1000gb drive what will happen? And can you site link?
    – user7886229
    Commented Sep 14, 2018 at 0:03
  • Look at the man page for ‘dd’. It will copy 500GB, but most of the 499 GB will be empty. The part that’s not is only visible to the operating system, and tells it that you have a 500 GB drive. Also look up “partition map” and “volume map”
    – WGroleau
    Commented Sep 14, 2018 at 16:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .