Incremental Snapshot Style backups with rsync on NTFS

Question

I've taken over development of an embedded Linux product which needs a reliable backup solution. Backups are to be performed without compression to an attached USB storage device formatted to NTFS.

The current implementation is a simple rsync --archive --verbose --delete which works, but given the --delete parameter any accidentally deleted files will also be removed from the backup which is no good.

By not using --delete on the other hand we may end up with duplicates if files get renamed.

Id like to switch this over to using hard-links to create snapshot-style backups as done by Mike Rubel, or http://webgnuru.com/linux/rsync_incremental.php, which basically comes down to:

rsync -avh --delete --link-dest=<previous-backup> <source> <backup-location>

I haven't seen any explicit mentions of file systems for this. I know that I can do this on NTFS but are there any hickups? will backups created this way be readable as expected on OSX and windows?

Answer 1

I haven't seen any explicit mentions of file systems for this. I know that I can do this on NTFS but are there any hickups? will backups created this way be readable as expected on OSX and windows?

While this is old, I would just like to mention one additional hickup I came across in the past years, beyond things like losing permissions. While NTFS is able to store multiple hard-links for files, their number is limited to 1023 and I had former discussions where it was reported that performance of at least older Windows/NTFS in deleting files with a lot of links was very bad. It was a discussion in a German forum about a tool called HardlinkBackup, which essentially works the same like rsync regarding hard-links. For some users it took hours to days to remove 25'000 files with 10 links from some backup history. Same people didn't report any problems with a comparable setup using some older Linux and ext* file system. I myself didn't recognize such a problem during my own tests, but things started to slow down in my tests with 50 links as well.

So while I guess things have improved a lot since Windows 7, especially because Windows relies on hard-links for updates and such itself much more these days, it might be a good idea to actually test it with some larger backup history.

Answer 2

Rsync creates mirrors, and mirrors are not backups; not unless you have at least 2, preferably 3+, of these USB storage devices and rotate them (in which case the --delete issues you mention are not important).

You should consider using a more appropriate tool such as duplicity, dejadup (essentially a front end for duplicity), rdiff-backup, or some other tool that keeps the history of your files.

NTFS will probably be fine with all of them, provided that exact duplication of file attributes like user, group, etc. are not especially significant. I believe duplicity, in particular, will not care because it keeps all your data inside archives (I use it with Amazon S3 cloud storage).