rsync seems to overwrite already existing file on ExFat

Question

I have a curious behavior from rsync between two hard drives containing the exact same four partitions. However on one partition rsync seems to always copy again the files, I don't understand why.

rsync -r --update --progress "/media/part3/" "/media/part3_backup" --exclude '.Spotlight-V100'  --exclude '.fseventsd'  --exclude '.Trashes'  --exclude '._.Trashes'  --exclude '.TemporaryItems'  --exclude '._.TemporaryItems'  --delete

Any explanation is welcome.

Answer 1

@firepol also has a great answer here. My answer is meant to accompany that answer and enhance it a bit:

Best rsync settings for copying and mirroring to or from FAT and exFAT filesystems

Quick summary

I've just put a couple of painful days into this, and here's what I've come up with:

For exFAT and FAT filesystems, use:

rsync -hvrltD --modify-window=1

The -h makes numbers "human-readable" as base-1000 numbers. The --modify-window=1 part is really important for FAT and exFAT filesystems to sync properly and not recopy all of the data every time! See more on this below, and in man rsync.

A full rsync command might therefore look like this:

SOURCE_PATH="path/to/source/dir"
DESTINATION_PATH="path/to/destination/dir"

# Option 1. **Copy** from source to destination.
#
# dry run
time sudo rsync -hvrltD --modify-window=1 --stats --info=progress2 \
    --dry-run "$SOURCE_PATH/" "$DESTINATION_PATH/"
# actual copy
time sudo rsync -hvrltD --modify-window=1 --stats --info=progress2 \
    "$SOURCE_PATH/" "$DESTINATION_PATH/"

# Option 2. **Mirror** from source to destination. 
# - CAUTION: this deletes files and folders that are on the destination but not
#   the source, thereby making destination dir *exactly match* the source dir.
#
# dry run
time sudo rsync -hvrltD --modify-window=1 --stats --info=progress2 \
    --delete --delete-excluded --dry-run "$SOURCE_PATH/" "$DESTINATION_PATH/"
# actual copy
time sudo rsync -hvrltD --modify-window=1 --stats --info=progress2 \
    --delete --delete-excluded "$SOURCE_PATH/" "$DESTINATION_PATH/"

For non FAT and exFAT filesystems, such as NTFS, ext4, etc., I replace -hvrltD --modify-window=1 in all of the commands above with -hvra instead. -a adds several things that are not compatible with FAT and exFAT filesystems, such as permissions and ownership (-pgo), and --modify-window=1 is only required for FAT and exFAT filesystems, to prevent recopying data to the destination when it is already on the destination.

Additional details

Normally, for filesystems which are not FAT or exFAT, I use -a with rsync, as I explain in my answer here: how to use FreeFileSync and rsync to copy data.

-a, or "archive" mode, includes -rlptgoD, as I explain in another answer here: rsync and symbolic links (the -r isn't always enforced by -a though, beware).

The meanings of -rlptgoD, from man rsync, are:

   --recursive, -r          recurse into directories
   --links, -l              copy symlinks as symlinks
   --perms, -p              preserve permissions
   --times, -t              preserve modification times
   --group, -g              preserve group
   --owner, -o              preserve owner (super-user only)
   -D                       same as --devices --specials
   --devices                preserve device files (super-user only)
   --specials               preserve special files ["such as named sockets and fifos"]

If using rsync on exFAT or FAT filesystems, however, these filesystems do not support POSIX owners, groups, and permissions, so remove -pgo, and use -rltD instead of -a! I like to also add -v for "verbose", and -h for "human readable numbers", resulting in -hvrltD.

At one point I read that FAT and exFAT filesystems do not support symlinks (soft links), which is what -l does, but when testing on a real exFAT filesystem, removing -l resulted in a ton of "skipping non-regular file" errors! So, putting -l back in solved these errors and made rsync run smoother.

As for the --modify-window=1 part, this is really important for FAT and exFAT filesystems! See man rsync here (emphasis added):

--modify-window=NUM, -@

When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. The default is 0, which matches just integer seconds. If you specify a negative value (and the receiver is at least version 3.1.3) then nanoseconds will also be taken into account. Specifying 1 is useful for copies to/from MS Windows FAT filesystems, because FAT represents times with a 2-second resolution (allowing times to differ from the original by up to 1 second).

So, if you leave --modify-window=1 off, rsync will never be able to tell when a timestamp matches, and it will just keep copying, and recopying, and recopying again, all of the data every time you run rsync, rather than seeing that the file size and timestamps match on the destination when the data has already been copied over once! In other words, without --modify-window=1, rsync will copy all of the data again each time instead of ignoring data which is already on the destination. That's not good. Be sure to add this parameter in to fix it. I saw this repeatedly in my testing on exFAT filesystems over the last two days, and lost so much time over it.

See more here:

1. rsync seems to overwrite already existing file on ExFat
2. https://superuser.com/a/384849/425838
3. https://learn.microsoft.com/en-us/windows/win32/fileio/filesystem-functionality-comparison

See also

1. [my answer] Super User: Is it best to reformat the hard drive to exFAT using 512kb chunk, or smaller or bigger chunks?
2. My answer on how to determine the cluster size of any filesystem you have: Server Fault: How to find the cluster size of any filesystem, whether NTFS, Apple APFS, ext4, ext3, FAT, exFAT, etc.

Answer 2

I had the same problem using rsync -avx from ext4 to exfat. It was copying files over all the time. -avx was not preserving the times and was giving me an Operation not permitted (1) error.

This worked for me for an exFAT filesystem:

rsync -rtv [source] [destination]

Explanation:

• r: recursive
• t: preserve times
• v: verbose

You have to make sure that the times are preserved.