42

I'm using rsync to essential fetch files off a server, then delete the files from the server once I have them locally. The full command I'm running is below.

This does successfully delete the files on the source server, however empty directories still remain. I don't get any messages or errors. All output is normal. Perhaps this is the intended functionality.

How can I tell rsync to clean everything, including the directories?

rsync --progress -vrzh --remove-source-files

Version is 3.0.9 on both ends.

Improve this question
1

5 Answers 5

Reset to default
22

The behavior of --remove-source-files that you observe is exactly that specified by man rsync:

--remove-source-files

   This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the transfer and have been successfully duplicated on the receiving side.

There is no specific command to remove the directories, as these two discussions in StackExchange and ServerFault clearly show. The solution suggested there is to issue two separate commands:

 rsync -av --ignore-existing --remove-source-files source/ destination/ && \
 rsync -av --delete `mktemp -d`/ source/

The last piece of the command suggested in these two posts,

 rmdir source/

which is needed to remove the (now emptied) source directory has this form in these posts because the OPs and the answers are using rsync to move large amounts of files within the same machine. In your case you will have to do this manually.

Improve this answer
3
  • 11
    The rsync --delete suggestion is dangerous, because it ignores the possibility that the rsync wasn't completed, or there are new files at the source. @slhck's find method below is much safer.
    – Sai
    Commented Jul 5, 2017 at 9:08
  • 2
    This is an idiotic oversight over whoever wrote rsync, since directories are files too.
    – raygozag
    Commented Dec 31, 2019 at 6:22
  • Although the rsync --delete command is only executed if the first rsync is successful because of the && condition (compared to the argument of @Sai), I wouldn't use it, too, as it could be possible that something writes in the source directory, while the first rsync is doing it's job (race-condition). Instead, the find method is race-condition safe.
    – mgutt
    Commented Mar 26, 2023 at 20:18
54

The manpage mentions that it only removes non-dirs:

--remove-source-files   sender removes synchronized files (non-dirs)

This is expected behavior.

If you want to remove empty directories in your source, if there are still files left, do a:

find . -type d -empty -delete

This will recurse, because it is using find, and only apply to´ empty directories (via the -empty option). -delete deletes those directories. (You cannot delete non-empty directories this way.)

If it's just one empty source directory, a rmdir <directory> will of course suffice.

Improve this answer
8
  • 8
    yeah, this is the only solution. it's kind of a silly missing feature of rsync... rsync knows when it's processed the last file in a directory... it's easy enough to remove the dir too if it's empty.
    – Erik Aronesty
    Commented Jun 23, 2014 at 19:10
  • 8
    Beware that issuing "rm -rf" is race-condition prone and I discourage it.
    – Raúl Salinas-Monteagudo
    Commented Mar 24, 2015 at 16:38
  • 3
    Variant that doesn't erase the top level empty directory: find some_dir -depth -type d -empty -not -path some_dir -delete
    – Cameron Tacklind
    Commented Oct 19, 2019 at 0:37
  • 1
    Not sure how it was in older versions of find, but -depth and -empty aren't needed I think. I tested it and a) it starts with the deepest dir and b) it is not able to delete dirs which contain files.
    – mgutt
    Commented Mar 26, 2023 at 20:14
  • 1
    @mgutt Interesting. It seems that for GNU find, "The ‘-delete’ action will fail to remove a directory unless it is empty.", and: "The use of the ‘-delete’ action on the command line automatically turns on the ‘-depth’ option." — perhaps it's different for older/other versions.
    – slhck
    Commented Mar 27, 2023 at 9:08
7

Using "rm -rf" has an inherent race condition, you could namely delete files that were just created between the rsync and the rm invocations.

I prefer to use:

rsync --remove-source-files -a server:incoming/ incoming/ &&

ssh server find incoming -type d -delete

This will NOT remove the directories if they are not empty.

Improve this answer
4
  • 4
    The rm -rf will also remove files that were not transferred for some reason.
    – Kristian
    Commented Jun 16, 2016 at 21:23
  • 4
    This answer misses the -depth option which instructs find to process in the right order. As a result of this miss, directories that contain only empty directories (possibly recursively) will not be deleted. The variant by @slhck has it right.
    – Stéphane Gourichon
    Commented May 15, 2019 at 21:40
  • @StéphaneGourichon "man find" in my system (Kubuntu 20.04) says: "The -delete action also implies -depth". So I think adding it explicitly is redundant.
    – Mauro Molinari
    Commented Jan 28, 2022 at 12:08
  • @MauroMolinari what you write is technically true. Still it's closer to danger than the recommended answer, especially when you first test commands without delete, to be sure. Man find also says "When testing a find command line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises.".
    – Stéphane Gourichon
    Commented Jan 28, 2022 at 21:00
1

Remove source files, then remove directories to be safe.

# given this scenario where you generate folders 2014-01-01 etc.. that have an archive myfile.tar.gz
pushd $(mktemp -d)
mkdir 201{4..6}-{01..12}-{01..31}
for i in $(ls); do; touch $i/myfile.tar.gz;done;

# find and rsync on 10 CPU threads directories that match ./2015-*
find /tmp/tmp.yjDyF1jN70/src -type d -name '2015-*' | \
parallel \
--jobs 10 \
--progress \
--eta \
--round-robin \
rsync \
--hard-links \
--archive --verbose --protect-args \
--remove-source-files \
{} /tmp/tmp.yjDyF1jN70/dest

# now safely remove empty directories only
for i in $(ls /tmp/tmp.yjDyF1jN70/src); do; rmdir /tmp/tmp.yjDyF1jN70/src/$i; done;

More on GNU Parallel

Improve this answer
-1

-m, --prune-empty-dirs prune empty directory chains from file-list

--force force deletion of dirs even if not empty

Improve this answer
1
  • 4
    That just prevents rsync from copying empty dirs. It doesn't delete empty dirs.
    – Navin
    Commented Jul 23, 2017 at 4:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .