Is there a faster way to copy time machine files from one disk to another?

Question

I'm trying to move my time machine backup files all under Backups.backupdb to another drive. I initiated a file copy overnight (b/c I saw that it took OSX forever to prepare for the copy...it basically was counting the files for hours). In the morning I saw that only certain backups(folders with dates) got copied over. I then tried to copy over the ones that didn't get copied...but the OS wouldn't allow me to do that. I got and error that "The operation can’t be completed because backup items can’t be modified." So my plan is to delete the incomplete copy on the new drive and then try to copy over the Backups.backupdb folder again.

Pretty frustrating. Is there a faster way to copy these files via a terminal command so that it doesn't perform all of that file counting prep?

I probably can tar up the entire folder and then do a copy, but will that interfere with any of the file permissions, etc.? The one thing with this approach is that I don't have any more space on my source volume for the tar.

UPDATE

I've tried some of the methods that people have suggested below, specifically using Disk Utility's restore function and it's giving me some error messages and unexpected results (at least to me). I've tried to do the restore two ways:

• With "Erase Destination" checked : Each time (I've tried twice), when the restore has finished I see a message "Could not restore - Invalid Operation" and "Could not restore - Invalid Argument". However, my destination disk does get a copy of my TM files. The weird thing is that my destination disk is EXACTLY like my source disk...even the size. My destination disk is actually 1 TB but after the restore, it shows as 200 GB when I get info from the finder. But in Disk Utility, it shows a 1 TB partition!

I then tried to verify/repair the disk and got:

    Invalid B-tree node size
    Checking Journaled HFS Plus volume.
    Invalid B-tree node size
    Volume repair complete.
    Updating boot support partitions for the volume as required.
    Error: Disk Utility can’t repair this disk. Back up as many of your files as possible, reformat the disk, and restore your backed-up files.

Don't know if I'm even suppose to verify/repair a TM disk...

• With "Erase Destination" UNchecked : The restore never starts and I get:

    Could not restore - Operation not permitted

Answer 1

A normal copy (or copy via rsync or ditto) will not replicate a Time Machine fully as it will convert two directories linked together (as occurs in successive TM backups with no change between) into two separate directories.

The best way is to copy the whole the disk using Disk Utility or the block copy part of Carbon Copy Cloner and probably similar on SuperDuper.

Answer 2

Migrating a full 3TB Time Machine encrypted drive to a new 8TB one on macOS 10.14, I ran into all sorts of issues. Trying to do a restore in Disk Utility errored out with “unable to validate source” or “Operation not permitted.” Trying some other suggestions in this post and others, I was able to get exciting new error messages like “Catalog file on image/volume is too badly fragmented,” but no copy.

What worked in the end, at the terminal:

1. Erase the new disk with Disk Utility, matching the format of the source drive: MacOS Extended (Journaled, Encrypted)
2. Use diskutil cs list in the terminal to get the exact byte size of the Logical Volume on the old drive, and the GUID of the new Logical Volume, as well as the disk numbers for both, e.g., disk4.

3. Use the exact byte size from step2 as the size of the new volume. In my case with a 3TB drive it was 2,999,772,905,472 bytes:

   sudo diskutil cs resizeVolume $new_lv_guid 2999772905472
   

4. Using the pv command from homebrew, do a low-level block copy of the disks. This is a lot like using dd, except you get a progress meter with ETA.

   You need to get the disk numbers from the diskutil cs list output. Be careful. It’s very easy to accidentally overwrite your full backup drive with the new blank one here.

   sudo sh -c "$(which pv) --buffer-size 50M -s 2999772905472 < /dev/rdisk${source} > /dev/rdisk${target}"
   

   If you get a permission denied / operation not permitted error here, go into Security & Privacy Preferences and add Full-Disk Access for Terminal.app.

   For me this took about 10 hours—I let it run overnight—but, with pv, at least you get a progress meter with an ETA.

5. Now, expand the volume to take up all remaining space on the drive:

   sudo diskutil cs resizeVolume $new_lv_guid 0
   

   This took ~ 3 hours for me, with about 5 years of backups. Most of that time was spent by macOS fscking.

Now you can enjoy your new, more spacious Time Machine drive. You can repurpose the old one, or stow it somewhere safe in case something happens to the new drive.

---

The resizing steps seem to be important; skipping them resulted in a 10-hour file copy that yielded an 8TB volume containing a 3TB filesystem that I couldn’t figure out how to resize.

---

UPDATE One potential downside to this approach is that because it’s a bit-for-bit copy, the identifiers are the same between the old disk and the new disk. If I connect the old full disk, Time Machine thinks it’s the new disk, tries to back up, and starts deleting old backups to make room for new ones. It seems like a fine approach for moving data to a larger disk where the older smaller disk will then be wiped.

Answer 3

Why not just use terminal:

cp -RnpP Backups.backupdb

• -R recursive
• -n do not overwrite (if existing copy remnants remain from previous attempt)
• -p preserve ACL's, permissions, creation/mod dates, etc.
• -P preserve hard links, do not follow any hard or symlinks.

Answer 4

Following the +1 comment about Super Duper!, here are the specs (they did not fit in a comment):

12.250.329 files evaluated, 10.408.594 files copied. Effective copy speed 8,68 MB/s.

for cloning a magnetic 2TB Backup drive with several years of backups this year (2019).

This took 63h in total (the info screen would reset its clock every 24 hours, so it showed 15:04:43 in the end) as opposed to a Finder copy which I cancelled after roughly 4 days and a quarter of the files.

Obviously the magnetic disk was not the reason for this taking so long. The reason Finder copies stall on long running backup disks is the sheer number of cascading symbolic links on unchanged files, especially for many small files like Git indices.

Answer 5

I had the same problem today. Tried the old drag & drop of my TM backup from my old external drive to a new one. Almost three hours later the file count was well over 1,000,000 and it was still ‘preparing’ to move them.

Here’s what I did, and it took less than a hour!

1. Restart Mac in Recovery Mode & choose Disk Utility

2. Select the new drive in Disk Utility’s sidebar

3. Click the Restore button (or go to Edit > Restore)

4. In the Restore drop-down choose the drive you want to clone/move

5. Click Restore

6. Once cloned, wait for Disk Utility to say it’s complete & click Done.

Hope this helps :-)

Answer 6

This answer will not get it done faster, but I've found is a way to copy the data properly while preserving the de-duplication (hard links) and permissions. As an added bonus I use this to make a compressed dmg of the final product for archiving.

1. Using Disk Utilities, make a disk image that is larger then your Backups.backupdb directory. I would also suggest that you use sparse bundle disk image for Image Format and Hard disk for Partitions. After this image is mounted, Get info on it and unselect Ignore ownership on this volume.

2. Now turn off Time Machine, and using the finder copy the Backups.backupdb folder to the mounted image. The finder will ask you for super user permissions to copy the data. Get a drink or do something else for a while.

3. When the copy is finished, look to be sure everything is okay and unmount the image. From Disk Utility, select Convert and turn the sparse bundle image into a compressed image. Again, this may take a while.

You should end up with two copies of your Time Machine backup, you can delete the sparse bundle version, and put the dmg in a safe place as an archive in time.

One thing I haven't tried with this is to do a system restore from the dmg, but I suspect that it should work, my goal was more for archiving the time machine incremental changes and keeping the hard link structure.

I have also tried rsync and cp, but they didn't seem to keep the hard link structure which would end up making x times the size, x being the amount of dates you had in the past. This method worked well, but again may not get speed of a block copy solution.

Answer 7

rsync is a great utility for stuff like this. I generally use it for stuff like this. In this case I might use the -aP flags. I think part of -a ("archive") is also to preserve permissions, ACLs and the like, but I'm not sure.

IIRC, there's also a --delete option that allows you to delete the source file once it has been successfully copied to the destination. I'd be wary of using that though - usually I do a complete mirror without the --delete option, then I'll re-run the command with the -c and --delete options. -c is checksum, so it checks all the files you've downloaded against all those on the source via checksum, then deletes the source if there's a match, otherwise it re-copies or resumes copying as the case may be.

EDIT: please use the -H flag in this case as per comments, in order to preserve hard links.

Answer 8

Recently I had to copy almost full TM 2Gb disk to 1Tb. So, I've written a small script to make it possible. Unfortunately target folders take more space than source folders. In case someone have ideas how it can be solved I'd like to know.

Here's line to gist with the script – https://gist.github.com/tyzhnenko/d17b3cdc7ec6edf4164d788b552c1513

Answer 9

Apple has an official tutorial for this: “Time Machine: How to transfer backups from a current backup drive to a new backup drive”.

The high-level steps from that page:

1. Check the format of your new backup drive
2. Set permissions on your new backup drive
3. Temporarily turn Time Machine off
4. Copy your backup data from your original drive to your new drive
5. Set Time Machine to use your new drive

This is how the page recommends doing the copying step:

Copy your backup data from your original drive to your new drive

1. Open a new Finder window. In the Finder sidebar, click the icon of the original backup drive.
2. Open a new Finder window. In the Finder sidebar, click the icon of the new backup drive.
3. Drag the folder "Backups.backupdb" from the original backup drive to the top level of the new backup drive.
4. Enter an administrator name and password, then click OK to start the copying process.

Copying your backup data might take some time to complete, depending on the size of your backup.

Answer 10

With hard drives, when you move multiple files from one drive, the reader moves back and forth making a scary clicking noise, and it slows the transfer rate significantly, for example- one file with usb 2.0 moves at 30 mbps on my computer from 2 external harddrives, but 2 files move at 11 mbps. and 3 files move at 6 mbps. etc etc. zip files will move faster than files.