Creating a hard linked differential backup (trying with rsync)

Question

I am trying to create a script that creates both a snapshot backup and a differential backup, but with hard links instead of duplicated files. In other words:

• All unchanged files between the current snapshot backup and the previous snapshot are hard linked instead of being duplicated;
• All files on the differential backup are hard links to the files on the current snapshot.

Let me try to clarify. Be:

• $SRC: Source tree, original files to be backed up
• $PREV: Previous snapshot backup
• $SNAPSHOT: Current snapshot backup
• $DIFF: Differential backup (only changed files)
• $RF: rsync flags, like -aPhvv

I am able to create the hard linked snapshot with the following command:

rsync $RF --link-dest=$PREV $SRC $SNAPSHOT

This works perfectly fine.

Now I want to create another tree, $DIFF, with only the differences between $SNAPSHOT and $PREV.

I can do exactly that with the following command:

rsync $RF --compare-dest=$PREV $SNAPSHOT $DIFF

This also works fine and is very close to what I want to achieve. The only problem is that all the files that changed between $PREV and $SNAPSHOT are now duplicated on $DIFF, I would like them to be hard links to $SNAPTSHOT instead of plain copies.

From rsync perspective, what I want to do is this: after the snapshot is created, it becomes my new source tree, $SNAPSHOT. The destination is $DIFF. The base of comparison with the source is $PREV, so the argument --compare-dest=$PREV should take care of it. This, like the previous command, would copy only the files changed between previous snapshot and the current snapshot. Now, for the files to be copied, I want them to be compared to the files in $SNAPSHOT first and then, if existing there, linked against them instead of copied. This is what --link-dest=$SNAPSHOT does, and it also works. Both options work well individually, compare-dest creates a differential copy and link-dest creates a linked tree, but I want to mix both, in the way described above, which rsync does not permit.

If rsync permitted, this what I wanted to do:

# Create the snapshot backup:
myhost:~$ rsync $RF --link-dest=$PREV $SRC $SNAPSHOT
     // successful. output suppressed...

# Try to create the hard linked differential backup:
myhost:~$ rsync $RF --compare-dest=$PREV --link-dest=$SNAPSHOT $SNAPSHOT $DIFF
rsync: You may not mix --compare-dest, --copy-dest, and --link-dest.
rsync error: syntax or usage error (code 1) at main.c(1567) [client=3.1.2]

Any ideas on how to create the $DIFF tree in away that the files on this tree are all hardlinks to $SNAPSHOT ? Or other suggestions on how to achieve my goals with other linux commands?

Answer 1

You can do this by creating $DIFF and $SNAPSHOT in the opposite order. First run

rsync $RF --compare-dest=$PREV $SRC $DIFF

Then create a hardlinked copy of the previous backup:

cp -al $PREV $SNAPSHOT

Finally, hardlink copy the changes onto there:

cp -al $DIFF $SNAPSHOT

Note: I haven't actually tested this, so can't guarantee it works.

Answer 2

Note that --link-dest option can be specified multiple times (since rsync version 2.6.4). It allows to create $SNAPSHOT in the following way:

rsync $RF --compare-dest=$PREV $SRC $DIFF

rsync $RF --link-dest=$PREV --link-dest=$DIFF $SRC $SNAPSHOT

NB: if the files content is the same but some preserved attributes are different (timestamp, permissions, ownership, etc) — the destination file will not be skipped or linked, but copied.

Regarding the timestamp attribute — the file (having the same content) will be copied when the timestamp is different even if the timestamp is not preserved but just used for files comparison (at least in rsync 3.1.1). Workaround: use --size-only or --checksum options.