Is there a way to copy with verification or just verify copied data

Question

I am a GIS specialist, part of this is to routinely copy vast quantities of data to disc - backup, archiving and delivery. Occasionally the files on the other disc do not match the source disc, in the order of for every 4TB copied there will be one error in one file (that's 0.000000025%).

Currently I am using RoboCopy or Windows Explorer copy to duplicate the files onto the other discs and WinDiff to verify the contents, which is how I know that there's a very small error rate. This process works but WinDiff is slow, sometimes taking more time to verify than it took to originally copy; RoboCopy does not have a and verify switch like XCopy.

I have previously used XCopy with the /v switch but was advised that that utility was depreciated in Windows 7 and I shouldn't be using it. Individually creating MD5, checksum or hash of files is not an option unless there's an efficient library/method/algorithm that I can implement in a programming language as there would be hundreds of thousands of individual files.

The question is: Is there a method that will verify the files as they are copied or a better verification program than WinDiff, or even better yet a guaranteed way to copy these files with 100% integrity so I don't have to verify them?

I can code for C# / VB.net / C++ / Python / DOS Batch provided the required libraries are free or built in. I'm not scared by command line, I can remember when there was only command line, so DOS utilities/instructions are fine.

So why is this now a problem if I have a working method? It's not really, I have a working method but have been doing it the same way since Windows NT in the late 90's; in the late 90's it was Exabyte tapes and 100MB or less, then there was Optical media which could be verified in less than half an hour. Now, with media being high capacity and relatively inexpensive, I find that I am copying and verifying up to 16 TB for a single delivery and the verification time is overrunning the delivery date.... I need to at least look for a better way!

Answer 1

The program you've been looking for is xxcopy. I too had this desire to do a binary file compare when copying oh, a few hundred thousand files for forensic analysis. I didn't care it took a little longer, I didn't want to have to create a SFV or MD5 file and then run another program on the files afterward, I wanted something that did the DOS "fc" command on the fly.

http://xxcopy.com/xxtb_027.htm#tag_319

You want switch /V2

Have a good day.

Answer 2

I have used Fastcopy and terracopy for years in windows, either of them make good replacements for the windows copy paste methods, both have a comparison routine. I think someone said the compare method for fastcopy is a "better" method. Both of them have caught very minor errors I caused myself by having ram timed incorrectaly.

There are other differances that make FC and TC 2 totally different acting programs, it is better to experience and adjust the options in both to be able to understand them. Like on moves FC will check each item prior to deletion one by one moves. TC can be easily set to compare all as a second action, or repeated action. TC might have issues with network drives, less people had issues with FC when using networked drives.

I prefer to use FC, but something like a single error in FC can cause more manuel effort, TC having all the items still in the list, and having a visable listing shown always has its advantages. They are both GUI style programs, not CMD style although they both will work in a command prompt or batch.
Both can isure better sequentiality of data by waiting till one copy operation is completed, prior to starting the other operation, so making fragmented messes on disks is reduced.

Neither of them make "fixing those last few files" that did not copy right like Windiff.exe, so windiff was a lot better when everything is failing :-)

Both will integrate right into the system as a replacement for the windows copy, and integrate themselves into the right click context menu also. Both can also work as fully portable. I no longer have either of them replace the systems copy, I have them in the context menu only to use them as desired.

Conclusion, try them both, see what you think.

Answer 3

You can read about the Windows FCIV tool here. That link will explain how to install the tool and also explain its usage. For your purpose, you can use the tool's recursive option and run these two commands:

fciv D:\Path\To\Source -r -xml db.xml
fciv -v -bp E:\Path\To\Destination -r -xml db.xml

Replace D:\Path\To\Source with the path of your source files and replace E:\Path\To\Destination with the path of the files that you copied.

The first command will create a database called "db.xml" in the folder where you ran the command. The second command will use that database to verify the copied files using the md5 hash checksum.

If all went well, you'll receive a message that says "All files verified successfully".