I've encountered a weird issue with rsync (without -c, therefore involving rsync's "quick check" algorithm) copying already synchronized files over and over again to an exFAT formatted drive.
The involved source files are stored on an HFS volume and were created by an automated image extraction process from .docx word documents and have a modification time of Jan 1, 1980 00:00:00 which also happens to be the minimum time supported by the exFAT timestamp format.
After a rsync -a those destination files on exFAT have a modification time of Jan 1, 1980 02:00:00.
It looks like the minimum time supported by my OS X exFAT driver is off by two hours.
And this is the reason that causes rsync's quick check algorithm to believe the files are different.
The following test seems to confirm this:
# create a 10Mb exFAT disk image:
hdiutil create -size 10m -fs ExFAT -volname EXFATTEST exfattest.dmg
# attach the diskimage
hdiutil attach exfattest.dmg
# create a test file with a creation time of Jan 1, 1980 00:00:00
touch -t "198001010000.00" /Volumes/EXFATTEST/test1.txt
ls -lT /Volumes/EXFATTEST/test1.txt
-rwxrwxrwx 1 user staff 0 Jan 1 02:00:00 1980 /Volumes/EXFATTEST/test1.txt
# ^^ off by two hours
# try a file with 3am
touch -t "198001010300.00" /Volumes/EXFATTEST/test2.txt
ls -lT /Volumes/EXFATTEST/test2.txt
-rwxrwxrwx 1 user staff 0 Jan 1 03:00:00 1980 /Volumes/EXFATTEST/test2.txt
# ^^ works as expected
# unmount the disk image
hdiutil detach /Volumes/EXFATTEST/
An equivalent test on Linux does not show this error:
truncate -s 10M exfattest.img
mkfs.exfat exfattest.img
mkdir /mnt/EXFATTEST/
mount -o loop exfattest.img /mnt/EXFATTEST/
touch -t "198001010000.00" /mnt/EXFATTEST/test.txt
ls --full-time /mnt/EXFATTEST/test.txt
rwxr-xr-x 1 root root 0 1980-01-01 00:00:00.000000000 +0100 /mnt/EXFATTEST/test.txt
# ^^ correct modification time
umount /mnt/EXFATTEST/
My simple solution is to update the source files' creation time using touch.
However, I'd like to know if this is really a bug and if someone can reproduce this error on newer OS X releases since Apple doesn't allow my still perfectly fine running early 2008 iMac to be be upgraded ;)
I'm still on El Capitan:
sw_vers
ProductName: Mac OS X
ProductVersion: 10.11.6
BuildVersion: 15G22010
Update:
After inspecting the exFAT table it seems that my El Capitan exFAT driver always uses 0xfc for the LastModifiedTimezoneOffset data field.
This signed byte value specifies the offset from UTC in 15 min increments. 0xfc is -4 in decimal and would be interpreted as a negative UTC offset of one hour.
I don't know why the driver is doing this but if it uses a static timezone offset of -1 it would also always have to subtract 1 hour from the corresponding UTC time stamp when storing the timestamp and adding 1 hour to the stored value when reading the timestamp back from disk.
This basically works, unless the UTC time stamp is younger than Jan 1, 1980 01:00, since the minimum supported time stamp of a file on exFAT is Jan 1, 1980 00:00.
So I consider this a bug, since El Capitan's exFAT driver is not able to set the minimum possible file modification time stamp supported on exFAT.
The following test seems to confirm this. It uses the TZ environment variable to force the usage of the UTC time zone independently of the system's time zone:
# set the minimum possible exFAT file modifification timestamp in UTC
TZ=UTC touch -t '198001010000.00' /Volumes/EXFATTEST/test.txt
# read back the stored timestamp in UTC
TZ=UTC ls -lT /Volumes/EXFATTEST/test.txt
-rwxrwxrwx 1 gollum staff 0 Jan 1 01:00:00 1980 /Volumes/EXFATTEST/test.txt
# ^^^^^^^^ ERROR: off by one hour
Same error when creating a file with this date on HFS and copying it to the exFAT volume:
# set Jan 1, 1980 00:00:00 on HFS (e.g. file in home directory)
TZ=UTC touch -t '198001010000.00' $HOME/test2.txt
TZ=UTC ls -lT $HOME/test2.txt
-rw-r--r-- 1 gollum staff 0 Jan 1 00:00:00 1980 /Users/gollum/test2.txt
# ^^^^^^^^ CORRECT!
# copy the file to the exFAT volume and check the destination date
cp -a $HOME/test2.txt /Volumes/EXFATTEST/test2.txt
TZ=UTC ls -lT /Volumes/EXFATTEST/test2.txt
-rwxrwxrwx 1 gollum staff 0 Jan 1 01:00:00 1980 /Volumes/EXFATTEST/test2.txt
# ^^^^^^^^ ERROR: off by one hour
Another interesting test I've done is taking the exFAT disk image created on linux (which contains the test file with the correct time stamp) and mounting it on OS X:
TZ=UTC ls -lT
total 0
-rwxrwxrwx 1 gollum staff 0 Jan 1 00:00:00 1980 test.txt
# ^^^^^^^^ correct!
However, when duplicating this file the date is off by one hour again. So at least the OS X exFAT driver is able to read the time stamps correctly, even if the files are stored using a non 0xfctimezone offset. Yay!
Now the other way around: Mounting the OS X dmg on Linux reveals a second error of the OS X exFAT driver:
mount -o loop exfattest.dmg /mnt/EXFATTEST/
FUSE exfat 1.0.1
ERROR: bad date 1980-01-00
# ^^^ OS X stored the DAY as zero ::facepalm::
Another perfect example of missing corner case tests in development.
This is basically the answer to my question. However, I will accept an answer that proves my findings wrong or confirms them and which also answers my question if this bug also happens on newer than El Capitan OS X releases.
Update 2:
I've come across a blog post by Adam Harrison (a digital forensics investigator from UK): exFAT Timestamp Behavior Associated with Different Operating Systems
Quoting:
Ubuntu 16.04 and Ubuntu 18.04
All timezones are recorded in UTC. The Timezone fields are consistently set to 00, indicating that they are not in use.OSX 10.13.3
Timezone Fields – Consistently set to “FC” which is UTC-1 and I have no idea why...
So this seems to match my findings.
