I have 16 subdirectories which all contain somewhere between 1m-1.5m files each (roughly 18m files in total), but I need all the files to be in a single directory. Each file is tiny (35-100 bytes each). The total combined size of the files is relatively small - around 600mb - but it appears to be the sheer amount of them that's causing the issues.
So far I've tried:
Windows move: Didn't even get started. It said it would take 'about a day' to calculate the move. Gave up after 2 hours of calculating
.
DOS move
: This works great for the first 500-600k files (moving around 10k files per second), but starts to slow down noticeably as it drags towards the million mark, doing about 100 files every 2 seconds.
7Zip: I've read suggestions that zipping up the entire folder and then extracting it in the destination would be WAY quicker; however using the GUI it just crashed explorer after a few minutes; using the command line was incredibly slow (100 files every few seconds)
DOS robocopy
: Having already moved ~1m files yesterday, I ran robocopy src_folder dest_folder *.log
just to shift the last of what was in the first directory. It took 27 minutes to move ~12k files.
No matter what method I choose, it seems that the number of files in the destination folder is what causes the issue. If there are more than a million files in the destination, the move/copy slows to an absolute crawl regardless of the method.
Any ideas on how to achieve this that won't take days/weeks? For reference it's on a single SSD on a single machine: 64-bit, 16gb RAM, 8 threads.
7z
- took around 50 minutes to zip up ~1m files. Moving that one file (which was only 97mb) took less than a second. Currently unpacking that in the destination folder to see how long it takes.pause
/sleep
for a specified amount of time after doing 500K files. (FYI: moving/copying files is always faster via command line in Windows - leave the Windows Shell [explorer.exe
] out of it)