3

I'm using rsync to copy file from Linux to Mac over SSH.

My command looks like this:

rsync \
    --exclude FOO \
    --exclude BAR \
    -e ssh \
    -rclpvih \
    --delete \
    --stats \
    /local/dir/ \
    host:/remote/dir/

And the output looks like this:

building file list ... done
...
<fc.T...... core/FileRecordingProcessor.cpp
<fc.T...... core/Pipeline.hpp
...

Number of files: 4,910 (reg: 4,401, dir: 509)
Number of created files: 0
Number of regular files transferred: 35
Total file size: 332.34M bytes
Total transferred file size: 2.62M bytes
Literal data: 0 bytes
Matched data: 2.62M bytes
File list size: 196.59K
File list generation time: 0.457 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 209.72K
Total bytes received: 12.03K

sent 209.72K bytes  received 12.03K bytes  147.83K bytes/sec
total size is 332.34M  speedup is 1,498.70

Even if nothing changed locally, rsync transfers the same (sub)set of files on each invocation.

Those files are only a small subset of the whole tree. Other files are not resent unless they are really changed. The number of file being resent is always 35. All of them have <fc.T...... flags.

Files look identical on local and remote machines.

Local:

$ ll core/Pipeline.hpp
-rw-r--r-- 1 victor victor 5.1K Nov  2 18:24 core/Pipeline.hpp
$ md5sum core/Pipeline.hpp
7604940d777322a587d2fe2fa12c1183  core/Pipeline.hpp

Remote:

$ ll core/Pipeline.hpp
-rw-r--r-- 1 victor staff 5.1K Nov  3 18:20 core/Pipeline.hpp
$ md5sum core/Pipeline.hpp
7604940d777322a587d2fe2fa12c1183  core/Pipeline.hpp

Owner, group, and times are not the same, but rsync is not configured to preserve them and is configured to use checksums.

The time on the two machines is not synced.

Improve this question

5 Answers 5

Reset to default
3

One case where this can happen is when rsync is transferring of files between a case-sensitive filesystem (typically Linux), and a case-insensitive filesystem (typically Windows and MacOS).

If the synced two paths (eg d/x and d/X) are the same after notional conversion to, say, lowercase, then rsync does not notice, and may transfer d/x, then overwrite the same destination file with d/X.

If the files do not contain the same data, and have the same timestamp, the files will always be updated on future rsync runs.

I suggest checking whether some upper/lower-case difference is causing rsync to mistakenly re-transfer the files. One useful Linux command is:

find . | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort | LC_ALL=C uniq -d

source

Improve this answer
2
  • You're probably right, though I don't yet understand the problem well. The issue is very strange. One of the files being sent each time is core/Pipeline.hpp. It's named so on both machines. But if I rename it to core/pipeline.hpp (lower-case), on both machines, it's no longer sent each time. At the same time, core/Pipeline.cpp (.cpp instead of .hpp) is not sent each time, and I don't have to rename it.
    – gavv
    Commented Nov 5, 2020 at 15:17
  • The command with uniq prints nothing; i.e. there are no conflicts.
    – gavv
    Commented Nov 5, 2020 at 15:18
3

You're missing either the --archive (-a) or --times (-t) flags. Without one of those rsync won't track the change time and so it cannot bypass files that look like they're already been copied

You're also explicitly telling rsync to use checksums to validate that the files have been copied, so that's what rsync does.

Generally speaking, the --archive (-a) flag will do just about everything you need:

rsync --exclude FOO --exclude BAR -avi --delete --stats /local/dir/ host:/remote/dir/

You might want to add -H to keep hard links (if you have any), and -AX to keep ACLs and Extended Attributes (if you have any).

Improve this answer
1
  • Thanks for reply. I'm using checksums instead of times intentionally. Though, adding --times didn't help too, as mentioned above. The problem is that rsync re-sends some of the files which contents (and checksums, and times) didn't change.
    – gavv
    Commented Dec 20, 2020 at 8:21
2

TL;DR: If you on macOS, try using /usr/bin/rsync instead of the open source rsync that "brew" or "nix" or other systems may install. The native rsync understands Apple's special filesystem better and won't recopy unchanged files.

Longer version...

A situation where this can happen is if the version of rsync that you are using doesn't support OS-specific situations. It will re-copy files that it thinks has changed but haven't.

For example, my Mac has 2 versions of rsync:

$ which -a rsync
/usr/local/bin/rsync
/usr/bin/rsync

/usr/local/bin/rsync is version 3.2.4 which supports protocol version 31. I installed this using "brew".

/usr/bin/rsync is the Apple-provided version, which is forked from version 2.6.9 (much older) and supports protocol version 29. Apple has enhanced this version to support macOS-specific file system features.

When I use -c the open source version recopies files that haven't changed. The Apple-provided rsync doesn't have this problem.

While there are many optimizations between protocol version 29 and 31, you probably won't miss them. Getting things correct by using Apple's rsync is probably important.

Improve this answer
1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Jul 7, 2022 at 14:54
0

Using the very fine answer in https://stackoverflow.com/questions/4493525/what-does-f-mean-in-rsync-logs, it looks like those <fc.T flags in the log are showing you:

<: a file is being transferred to the remote host (sent)
f: it's a file (not a dir or symlink)
c: a local change/creation is occurring for the item
.: the item is not being updated (though it might have attributes that are being modified)

T: well, a t means the modification time is different and is being updated to the sender’s value (requires --times). An alternate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without --times and when a symlink is changed and the receiver can’t set its time.

I think the answer is in there, something is wrong w/ the timestamp differences, so you might need use the --times setting to try to preserve the times.

Improve this answer
4
  • I think T is here just because I'm not using --times and timestamps are different; but it's not the source of the problem. After adding --times, rsync still resends the same files, but the flags are now <fc......... It looks like c indicates the source of the problem, but I don't understand why, since the files contents is identical on the two machines.
    – gavv
    Commented Nov 5, 2020 at 15:03
  • Could there be something else happening on the Linux source side that's changing a file property? I'm not a big Linux person, so can't imagine what. However, I could totally see something like this happening on the Mac side, and then the Linux source thinks it needs to copy the file again, because the target has been changed by the source didn't change. Sorry, not super helpful, but maybe if you're logged out of the Mac (sitting at the login screen), that might prevent random Mac processes from changing the file. Try 2 rsync's in a row w/ the Mac logged out (but turned on).
    – jimtut
    Commented Nov 6, 2020 at 1:53
  • Could there be something else happening on the Linux source side that's changing a file property? - nope
    – gavv
    Commented Nov 22, 2020 at 10:53
  • but maybe if you're logged out of the Mac (sitting at the login screen), that might prevent random Mac processes from changing the file. - sounds promising, but unfortunately logging out didn't change anything
    – gavv
    Commented Nov 22, 2020 at 10:54
0

I ended up switching to unison for synchronization between Linux and macOS (and Windows):

unison \
    -servercmd /usr/local/bin/unison
    -ignore "Name FOO" \
    -ignore "Name BAR" \
    -auto \
    -batch \
    -force /local/dir/ \
    /local/dir/ \
    ssh://host/remote/dir/
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .