rsync: identical files (contents, size, timestamp) between source and destination are not being seen as identical

Question

NOTE: Sadly, it turns out that this has nothing to do with file timestamps.

Per the info given in that other post, I tried --modify-window=30 and even --modify-window=300, but that did not change the incorrect behavior that I'm reporting. And this has nothing to do with timezone differences, either, because rsync correctly identifies most files that are identical between source and destination.

---

I'm using rsync to back up a large amount of data from the file system on a linux machine (Ubuntu 20.04 using the ext4 filesystem) to a file system on a mac (Big Sur 11.7 using the APFS Data Volume filesystem). Because of differences in the OS, some identical files are shown to be different when queried by rsync on linux and on the mac. This discrepancy causes quite a few files which are identical between the source and destination to always be transferred during each rsync run.

... but not all files. Most identical files correctly do not get transferred. But out of the few thousand files on my linux machine, a few hundred files that are identical between source and destination are always consistently treated as non-identical during rsync.

This is the command I'm running ...

rsync -avz -i -e 'ssh -p 6000' --ignore-errors "--exclude-from=excludes" --delete-excluded --delete-during / [email protected]:/var/backup

I have also tried it with other options, such as --no-o, --no-g, and --no-D, but I still get the same results.

And these are the contents of the excludes file ...

excludes
swapfile                                                                                                                                              
mnt                                                                                                                                                   
media                                                                                                                                                 
tmp                                                                                                                                                   
opt/tmp                                                                                                                                               
var/tmp                                                                                                                                               
var/log                                                                                                                                               
run/user                                                                                                                                              
proc                                                                                                                                                  
dev                                                                                                                                                   
sys                                                                                                                                                                                                                                                                                  
**/lost+found*/                                                                                                                                       
**/*Trash*/                                                                                                                                           
**/*trash*/                                                                                                                                                                                                                                                                                                                                                                                                                  
**/.cache/mozilla/**                                                                                                                                  
**/.cache/waterfox/**                                                                                                                                 
**/brave/resources/**                                                                                                                                 
**/usr/src/linux*/**                                                                                                                                                                                                                                                                 
**/.mbsync/**                                                                                                                                         
**/snap/**                                                                                                                                            
**/snapd/**                                                                                                                                           
**/flatpak/**                                                                                                                                         
*.qcow2                                                                                                                                               
*.iso                                                                                                                                                 
                                                                                                                                                          

Also ...

Per the suggestion that was given to me, I re-installed rsync on the Mac via homebrew and retried the sync after having added --rsync-path=/usr/local/bin/rsync to the rsync command line (that's the location where homebrew installs it). Sadly, using that homebrew version of rsync did not fix the problem.

And furthermore ...

I tried this both with and without the --checksum option, and with a few different --block-size options that go along with --checksum. I also tried this with and without --sparse. None of these things fixed the problem, either.

And to repeat some things that I wrote before but which were discussed in comments of mine that other people deleted ...

I can run the rsync command over and over with absolutely NO changes having taken place either for the source or the destination, and yet, the exact, same subset of files and directories are always and repeatedly shown as being different between source and destination, even though all of them are totally identical in terms of contents, size, and timestamps, and even though all of them are totally unchanging.

And by identical size, I mean that the sizes are precisely equal when measured in bytes.

I strongly suspect that this incorrect rsync behavior is due to the fact that the source is on a linux ext4 filesystem and the destination is on a MacOS APFS filesystem. I suspect that there is something about this difference in filesystems which confuses one or more of the file comparison algorithms that rsync makes use of.

Hmm ... perhaps could it be that the APFS filesystem confuses "access time" with "modification time"?

Answer 1

The situation here is triggered by the destination APFS filesystem being case aware but case insensitive. That is, you can create directories with mixed case and the filesystem will record that mixed case (for example Hippo vs hippo), but case is ignored when using the filesystem (so mkdir Hippo; cd hippo will succeed). This is similar to NTFS but markedly different to the standard default settings for ext4.

There are two ways to create APFS filesystems. The default "APFS" is as described here. The alternative is to create an "APFS (Case sensitive)" filesystem, which acts as case aware and case sensitive.

Consider this worked MVE.

Run this first part on a UNIX/Linux system with ext4 filesystem. First we prepare the scenario:

mkdir mve && cd mve         

mkdir a b A; touch a/f1       # Create initial structure
( cd a; ln -s f1 s1 )         # Link to self, a/s1
( cd A; ln -s ../a/f1 s1 )    # Link to A/s1

tree                          # Tree structure showing symlinks

.
├── a
│   ├── f1
│   └── s1 -> f1
├── A
│   └── s1 -> ../a/f1
└── b

3 directories, 3 files

Now copy the filetree across to a Mac with APFS:

rsync -iav . bigsurmac:mve    # Copy the tree to a Mac with APFS

building file list ... done
created directory mve
cd+++++++++ ./
cd+++++++++ A/
cL+++++++++ A/s1 -> ../a/f1
.d..t...... a/
<f+++++++++ a/f1
cL..T...... a/s1 -> f1
cd+++++++++ b/

sent 225 bytes  received 108 bytes  222.00 bytes/sec

Now let's look on the destination system to see what we have received. Run this on the Mac with APFS:

ls -lgR mve                   # Directory created by rsync from UNIX/Linux

total 0
drwxr-xr-x  4 staff  128 Nov  9 08:25 A
drwxr-xr-x  2 staff   64 Nov  9 08:25 b

mve/A:
total 0
-rw-r--r--  1 staff  0 Nov  9 08:25 f1
lrwxrwxrwx  1 staff  2 Nov  9 08:25 s1 -> f1

Notice that there is no a directory - it is considered the same as A. Therefore, when you have a symbolic link in 'A' that points to a file in a and you also have a symbolic link in a named the same as the one in A you will get a conflict. (You will also get conflicts for same-named files in A and a that are different sizes or timestamps.)

To demonstrate your issue, return to the UNIX/Linux system with the ext4 filesystem and copy the original files again, noting that no changes have been made either to source or destination:

rsync -iav . bigsurmac:mve    # Uh oh, some files need recopying

building file list ... done
cL..T...... A/s1 -> ../a/f1
.d..t...... a/
cL..T...... a/s1 -> f1

sent 165 bytes  received 38 bytes  135.33 bytes/sec
total size is 9  speedup is 0.04

You can see that although no changes have been made, some items need to be recopied. This is a consequence of the case-insensitive filesystem on the destination, meaning that you cannot have directories a and A at the same time, so some files in one of those directories will always appear to be missing. Furthermore, if you have a and A each with a file of the same name but different contents, one of them will always be wrong with respect to its source (two sources, one destination).

Because the issue lies with the filesystem differences, this will affect any copy utility and not just rsync. You cannot resolve the problem with scp, for example.

The result is that you cannot reliably copy a case-sensitive filesystem tree such as can be found on ext4 filesystems to an APFS filesystem.