0

I have folders with large amounts of data (TB) in need of backups, and I have a collection of hard drives to store the backups in. The problem is, the computers here are not authorized to connect to unsecured USB peripherals. We do have one secured USB portable hard drive that can be used, but it only has a 750GB capacity. I did a manual backup a few months ago, and it was a pain to copy the several TB of data in 750GB chunks.

Since I already have an old copy of the data on the hard drives, I was wondering if there is a way to copy only the changed/new data so that I don't have to copy everything again?

UPDATE:

Since this was not obvious to a commenter, here's a step-by-step of how the data transfer takes place:

  1. Download new data from computer A to 750GB portable HDD
  2. Connect both the 750GB portable HDD and the backup HDDs to computer B
  3. Transfer the data from the 750GB portable HDD to the HDDs

(These steps are for the case where we do a complete backup from scratch, but I need to modify this somehow to avoid copying redundent data that has not changed)

7
  • There are numerous ways to do this. What methods have you tried?
    – Ramhound
    Commented Oct 2, 2014 at 23:30
  • Just to clarify, the hard drives with the backups CANNOT be connected to the computer with the live data. If you have any methods in mind, feel free to recommend.
    – Setsu
    Commented Oct 2, 2014 at 23:49
  • update your question
    – Ramhound
    Commented Oct 3, 2014 at 0:56
  • Can you not do it over a network? Make a share that you have access to, and put all the files in there.
    – Adam
    Commented Oct 3, 2014 at 16:55
  • No, the IT here is very strict about what connects to their network. In short, computer A and B cannot talk to each other directly.
    – Setsu
    Commented Oct 3, 2014 at 16:58

1 Answer 1

1

So after poking around the documentation for robocopy I figured out a way to accomplish this. It is not bullet-proof, though (see caveats).

What this will accomplish

Computer A has the source directory and computer B has the destination directory. The two computers cannot be connected to each other, but must use a portable storage device to transfer files. This method will allow the source directory to be backed up at the destination directory without copying the entire contents each time a backup is performed. The backup will be an exact copy of the source (see caveats section for exceptions).

For demonstration the following paths are used:

  • C:\SOURCE (on computer A)
  • D:\DESTINATION (on computer B)
  • X:\TEMP (on portable storage device, such as USB hard drive)

Step 1: Create directory structure

We need a way for computer A to be able to figure out what the contents of the backup are, so we use robocopy to create the directory structure on the portable storage device. This will create identical folders and files, except that the size will be 0.

robocopy /e /create D:\DESTINATION X:\TEMP
(note: make sure X:\TEMP is empty or non-existent before doing this.
 RC won't create 0-size files if they already exist)

Step 2: Copy new files

Now connect the portable storage to computer A, and use robocopy to copy new files. Note that robocopy treats the 0-sized files as "changed" files if the fully-sized files exist in the source, so we exclude them to avoid copying redundant files that have not changed.

robocopy /mir /xc C:\SOURCE X:\TEMP
(note: the documentation on Microsoft technet says the switch is /xct,
 but the copy of RC on my Windows 7 machine says it's /xc)

Step 3: Update the backup

Now that we have the new files copied, we can connect the portable storage back to computer B and use robocopy to finish the job. Note that we use the /XC switch again to avoid overwriting our backup files with 0-sized files.

robocopy /mir /xc X:\TEMP D:\DESTINATION

Caveats

This method relies on the fact that robocopy creates 0-sized files that are otherwise identical to the original files. They are identified as "changed" because they have the same timestamp as the original files but have different file sizes. Under normal circumstances when a file is modified its last modification date will get updated and robocopy will identify them as "newer". If you have files that have their contents modified but not their metadata, the timestamp will stay the same and the result is the above method will NOT backup those files.

Additionally, the portable storage media used can affect the files as well. If you use a FAT32 system, then things like NTFS security info and encryption won't transfer since they aren't supported on FAT32. Also, files larger than 4GB will fail to transfer. So it's best to use NTFS all the way.

1
  • I wish I found this out BEFORE I went and coded my own thing that analyzes folders and pinvokes CopyFileEx ...
    – Setsu
    Commented Oct 16, 2014 at 21:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .