How can I compare lists of files for manual incremental backup purposes?

Question

I have backups of my hard drives in a physically different location. Periodically I will bring my hard drives to this location to update the backups. I do this because the data on my drives is constantly changing so it is not as easy as dumping 'new' or 'changed' data into a folder and only moving the contents of that folder via an intermediary portable hard-drive.

That said, is it possible to capture an 'image' of my backups (file paths, sizes, modified date) so I can compare the backups against the state of my hard drives and only add the changed / new files etc to an intermediary portable hard-drive?

Failing this, is there software with this capability?

Answer 1

rsync is intended for this purpose. It will work remotely as well copying only changed files. Once you have done the initial copy all copies will be incremental. It can also be used to retain prior versions.

There are a number of backup packages that are designed to use rsync functionality to create backups with the capability to retrieve older versions.

Answer 2

You can use the rsync style solutions (rsync, rsnapshot, rdiff-backup), or a program like Duplicity or borg-backup.

The advantage of doing it with tools like Duplicity or borg-backup is that you could store dozens / hundreds of snapshots on the backup drive, which would let you recover to multiple points in time instead of just to your last backup. (Both Borg and Duplicity are very efficient, without creating lots and lots of inodes/files. And Borg snapshots can be mounted as a file system.)

For my backups, I use Borg to write to the backup directory via SSH, then I use rsync to copy the borg repositories off to LUKS-encrypted USB drives that get rotated off-site. I keep at least a year's worth of snapshots in the borg repo, plus any manual snapshots that I have created.