Robocopy reports less than 100%

Question

I'm using robocopy to copy a large number of files (300,000 files totalling around 4TB) from one NAS drive to another. Here is a sample command:

robocopy \\nas1\myfolder \\nas2\myfolder /E /R:1 /W:5

I've done this hundreds of times before across a wide variety of folders and drives without any issue. In each case, as it finishes copying a file, it always reported 100% before moving on to the next.

However, I'm suddenly getting less-than-100% transfers being reported. Usually 98-99% but sometimes as low as 30%. See image:

The robocopy command is still running (expected to take several hours) so I cannot yet determine the validity of the copied files or do any checks.

In what circumstances would robocopy do this? Is there a problem with the transfer or is it reporting partial progress in threads or something?

Update: Small files (< 1k) tend to be worst affected, reporting < 50% in many cases. Large files (> 10MB) tend to report 98%+

Answer 1

I believe that robocopy will see files as different sizes if the underlying block size of the destination volume is different than the source volume (i.e. a several-byte file's size-on-disk may be 4k NTFS, but only 1k on the NAS)

This often shows up when people are mirroring from an NTFS to a NAS, but I have only seen comments about this regarding the compare phase rather than during the copy itself.

I think you can tell samba to use a specific block size so you can configure the destination to use the same block size as the source if they do not already match.

Answer 2

SPECULATION ON WHAT IS HAPPENING W/O ANY TESTING:

1. Files with less than 100% may have copied over, but are awaiting some integrity test later in the processing (maybe when robocopy is almost finished)
2. Files with less than 100% may have failed as you said retry once after 5sec, so a network hiccup could have caused
3. It's copying multiple files concurrently & is truly listing the partial files it has copied over

Things you could try:

• robocopy /J: Copies using unbuffered I/O (recommended for large files)
• robocopy /NOOFFLOAD: Copy files without using the Windows Copy Offload mechanism
• robocopy /z /MT 32 /Log+:<LogFile>:
  • /z = Copies files in restartable mode. In restartable mode, should a file copy be interrupted, Robocopy can pick up where it left off rather than re-copying the entire file.
  • /MT 32 = Creates multi-threaded copies with n threads. n must be an integer between 1 and 128. The default value for n is 8. For better performance, redirect your output using /log option.
  • /Log = creates a log file that is helpful to follow-up on any files that failed

Links I found helpful investigating this:

• https://ss64.com/nt/robocopy.html | https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy
• https://docs.microsoft.com/en-us/azure/databox/data-box-deploy-copy-data
• https://techcommunity.microsoft.com/t5/ask-the-performance-team/slow-large-file-copy-issues/ba-p/372377
• https://social.technet.microsoft.com/wiki/contents/articles/1073.robocopy-and-a-few-examples.aspx