20

Short: Is it possible to (robo)copy a directory tree using hard links instead of physical copy?

I want to make a daily backup of my data files, every day in it's own directory. Now most files don't change every day, so my idea to speed up backup and use less disk space was to copy the backup of (day-1) to (day) using hard links, then use robocopy to replace the files which have changed since yesterday. This way I would only need to copy the full backup once to the USB drive, afterwards I could just copy the changes and still have directories containing all the files, sharing files with other backups using hard links. This way I got two full backups but it uses only file space for one+differential, and the backup files are accessible without need for special software, which is a big plus for me.

Is this possible?
Is there a program to create hard link copies of all files in a directory tree?
Am I missing some pitfall I'm not aware of?

Improve this question
1

4 Answers 4

Reset to default
10

There is a way to make backups like you say, but I don't believe with Robocopy alone.

There are several programs that will hard link or dedupe backups, and be space efficient.

Here are a few programs I've used or ran across:

  • Dupemerge - free - While not a "backup program" it is a bulk hard linker.
  • hardlink backup - free for personal use - uses hard links
  • Duplicati - free - (dedupes I believe like rsync does)
  • rdiff-backup - free - dedupes like rsync

I believe what may be a good fit for your questions is Dupemerge. You could have a batch file execute Robocopy to copy the contents to a directory (ex. d:\backups\YYYY-MM-DD). When that command is done, the batch file could call Dupemerge on d:\backups. That would hard link the files within and across all the subdirectories of d:\backups.

The downside of this is you would need enough space on the external to hold the newly created unhard linked backup. After Dupemerge is run, the newly created backup will be hard linked and the space will be freed.

This would make restoring easy because each dated directory would appear to have a full backup.

As a side note, if you are going to use hard links, I like Link Shell Extension. It helps let you know what isn't hard linked, what is and what it is linked to.

I’ve run across other hard link or deduping programs, but I liked these the best.

Improve this answer
6
  • 5
    +1 For having a decent roundup and mentioning LSE. However, you forgot the single tool that gives the OP almost exactly what he wants: ln.exe's Delorean Copy ln.exe is written by the same person who write both Dupemerge and LSE. All that's missing is VSS support and... oh look! Here we go! ;-)
    – afrazier
    Commented Dec 15, 2011 at 3:50
  • @afrazier good call with Delorean Copy. Looks like Delorean Copy has some good features, also that website is good for NTFS link information, I should read it more. I'm surprised about the 1023 link limit on Delorean Copy, however, probably not a problem for most. Dupemerge will link to another file at 1022 links. I guess the way the Delorian Copy works, linking to a non-source file wouldn't be desirable? I will have to read up more on it. Great suggestion.
    – Scott McClenning
    Commented Dec 15, 2011 at 7:32
  • 2
    The 1023 link limit is an NTFS limitation -- The directory entry's link counter attribute is only 10 bits.
    – afrazier
    Commented Dec 22, 2011 at 17:28
  • Great summary of options. Unfortunately your recommendation doesn't cater to the original poster's request of only copying changed files on each subsequent backup. Sounds like ln.exe is a better bet.
    – Simon E.
    Commented Sep 12, 2019 at 10:05
  • There's also the Link Shell Extension by the same makers of ln.exe which provide a simple way to make these kinds of backups using the mouse in Windows Explorer. It also creates timestamped directories for you during the backup process.
    – Simon E.
    Commented Sep 12, 2019 at 10:27
1

BackupAssist will let you exactly what you require. Its use of hardlinks is named "Single Instance Store", but it's actually hard links under the hood. It can be used, among other things, to backup to USB drives the way you asked for, or to remote locations using rsync.

Improve this answer
1

We have written our own tool for that kind of work. Its called ntfs-hardlink-backup: https://github.com/International-Nepal-Fellowship/Windows-Tools/tree/master/ntfs-hardlink-backup

For creating the hardlinks its uses the ln tool of Hermann Schinagl: http://schinagl.priv.at/nt/ln/ln.html

So you could also use ln directly. Its pretty powerful

Improve this answer
4
  • Please quote the essential parts of the answer from the reference link(s), as the answer can become invalid if the linked page(s) change.
    – DavidPostill ♦
    Commented Jan 28, 2015 at 8:25
  • 1
    schinagl.priv.at/nt/ln/license.txt
    – bummi
    Commented Jan 28, 2015 at 11:35
  • How did you increase the maximum number of hard links NTFS allows to a single file?
    – Michael
    Commented Sep 19, 2022 at 15:48
  • How did you increase the maximum number of hard links NTFS allows to a single file?
    – Michael
    Commented Sep 19, 2022 at 15:48
-4

There is a flaw in your logic in the first place.

If you create hard links how can that be called a backup? once your file gets deleted/corrupted, your so-called backup goes that way too. All the requirements that you specified are usually covered by today's backup software.

P.S.: NTFS does't support hard links AFAK.

Improve this answer
6
  • 3
    Sorry, my description was not clear enough: I create a full copy using robocopy on a external USB drive. For the next day, I create a hard link copy, then replace all the files which have changed since yesterday. Now I have a two full copies on my USB drive, but at the space cost of one full + one differential.
    – Sam
    Commented Jun 14, 2011 at 8:41
  • 4
    Oh, and of course NTFS does support hard links.
    – Sam
    Commented Jun 14, 2011 at 8:42
  • 1
    The technique you describe is exactly what rsnapshot (seems to be down ATM) implements in Unixoid OS. It is definitely possible and works very well. Unfortunately for you rsnapshot doesn't run on Windows (that's why I write this as a comment only). But the FAQ mentions something about remote syncing Windows clients. If you have Unixoid machines around maybe this would work for you. I don't know any Windows equivalents of rsnapshot.
    – Daniel Böhmer
    Commented Jun 14, 2011 at 8:59
  • I don't see any gain by creating hard links to unchanged files. Backup softwares take backup based on Archive bit on/off. You should just copy those files which are modified since last backup. Also, I was under the impression that ONLY Unix File Systems support hard links. Thanks to bring it to my notice!
    – anilmwr
    Commented Jun 14, 2011 at 9:11
  • @Halo Thought it should be used already at a lot of places. Pity I got no Unix-Computers around, so I'll have to wait for someone who knows about this for windows. Thanks for the heads-up!
    – Sam
    Commented Jun 14, 2011 at 9:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .