Rsync Compare-Dest not Recognizing Wildcard

Question

I am trying to use rysnc to do multi level incremental backup, however I am running into some issues. It seems as though rsync does not recognize wild cards in compare-dest. I run the following code

rsync -av /home/j/test/source/ --compare-dest=/home/j/test/destination/*/ /home/j/test/destination/00/

It will give me the following message.

--compare-dest arg does not exist: /home/j/test/destination/*

I run into this issue even after trying to run the code again and place my backup in a new directory

rsync -av /home/j/test/source/ --compare-dest=/home/j/test/destination/*/ /home/j/test/destination/01/

On the second attempt it will copy everything from the source folder to destination/01/ instead of looking for changes in all the directories within the destination directory as we would expect with a wildcard.

The intent is to be able to continually make new directories withing the destination directory to these new directories perform incremental backup of the source while comparing all previous backups. Not sure how to get wildcards to work with --compare-dest, I have seen other people use them with rsync to do a very similar thing.

I am running Ubuntu on WSL2 and have tried the same on my server running Ubuntu 22.04. Some guidance on this would be much appreciated.

Thanks,

J

Answer 1

Here's what's happening:

1. Your command contains a wildcard. Normally such things will be processed and expanded by the shell before rsync even gets to run. However, in this instance what the shell sees is the string --compare-dest=/home/j/test/destination/*/, and as this won't match a directory path it is left unchanged.
2. The rsync command is now executed and after parsing its arguments ends up with compare-dest having a directory path containing an asterisk. It's important to note that by this point the asterisk is a literal character, and that's what rsync attempts to use. (Conceptually, replace * with a word such as STAR.)
3. The rsync command cannot find a matching directory path that contains a literal asterisk as a directory name (i.e. "/home/j/test/destination/*") and fails with the error that you see.

If you can guarantee that there is going to be only one directory matching the path /home/j/test/destination/*/ you can repeat the rsync command but without the =:

rsync -av --compare-dest /home/j/test/destination/*/ /home/j/test/source/ /home/j/test/destination/00/

Semantically this is the same thing but this time the shell gets to parse /home/j/test/destination/*/ and attempt to expand the wildcard into one or more matching directories.

Two really important notes, though:

1. If the expansion of the path fails, the * will be left as a literal character and you'll get an error from `rsync again.
2. If the expansion results in more than one directory path the second and subsequent will be treated as additional source directories by rsync with ensuing chaos. THIS WOULD BE BAD.

If you cannot guarantee that there is only one match you can use the {…} brace expansion pattern with a wildcard, which will repeat the --compare-dest=/path/… as many times as necessary for the set of directories matching the wildcard path. For example:

dirs=(/home/j/test/destination/*/)
echo "dirs=( ${dirs[*]} )"

echo rsync -av "${dirs[@]/#/--compare-dest=}" /home/j/test/source/ /home/j/test/destination/00/

There will almost certainly be better solutions for your specific scenario, but without knowing further details it's impossible to recommend an ideal solution.