WinSCP Synchronisation files buffer (210K files!)

Question

I have a folder being synchronised (mirrored) remotely, using the synchronize -mirror -delete remote command, over FTP.

The folder I am synchronising contains 209,068 files, and 19,079 directories, and thus, even just to check that everything is synced, it takes ages.

The way it seems that WinSCP does it, is it takes 1 file a time, and checks if it is up to date. If it is not, it uploads it.

This process takes forever. 210K files, with a ping of 50ms at least, which means around 3 hours of just comparing files! Not even syncing. (not accounting for actual handling time even)

So I was wondering if perhaps there is a way to make it batch compare (ask for meta-data for like a 100 files at a time), or perhaps create a buffer of a 100 requests a time, that would go much faster (around 100 times faster), which would be 1.8 minutes instead of 3 hours.

If not, perhaps there is a different software I could use that performs better in this kind of task?

Edit: Why I think it does 1 file at a time? When running the sync command, this is the output I am seeing:

E:\sync\a.txt -> \Sync\a.txt

Less than a second after:

E:\sync\b.txt -> \Sync\b.txt

And so on, every file shows, in order of the directory, a part of a second after another

Edit2: Correction

• When syncing a folder, it does go file by file.
• When checking if a folder is synced, it goes folder by folder.

My previous information was incorrect, my apologies.

Answer 1

Both ftp and sftp are very slow when transmitting large number of small files.

If it is possible (that is, if you have access to ssh shell), please try to use rsync, it should be way faster, especially, when some files were transferred earlier and only few of them were changed.

There are several implementations of rsync for Windows. I would recomment grsync, as it has a graphical interface.