Making backups of 10 hard drives. Should I just make disk images?

Question

I am going to be making backups of about 10 hard drives, around 4 TB of data. I will be using a USB to SATA adapter and then be transferring the data to a 4tb external USB drive. I will most likely be doing this under a Linux environment running on a raspberry pi since I want the copying system to be stand alone. I know the Pi's power abilities, so I will be using a powered hub, and that shouldn't be an issue. My only question is, would it be a better idea to just make images of all 10 hard drives and then mount those images whenever I need to get data off them, or actually copy over each file to the new hard disk. The reason I thought images would be a better idea is because I have seen problems in the past in windows when dragging and dropping all of the contents of a hard disk where it will get hung up on a single file for some unknown reason and give a vague error like "could not copy file xxx", which will in turn stop the rest of the copy process. From what I understand, making an image would result in a 1 for 1 copy of the contents and doesn't pay attention to the actual data. Also the organisation of images seems nice. Are there any advantages or disadvantages of using images? Also, are there any programs that would run well under debian to do this?

-Ben

Answer 1

I'd probably consider a few things.

The raspi is not a good device for initial imaging- its underpowered, you'll need to power your drives somehow, and it'll overcomplicate things. A 'regular' desktop is what I'd use given an option

As for imaging, I tend to like gnu ddrescue for this sort of thing - it does one pass, skipping over bad sectors, and repeated passes to ensure as much data is copied as possible.

You can then mount these images with kpartx (which would mount images with more than one partition) and get the data out from there.

Answer 2

There are different tools you could use, depending on what your goals are:

rsync: If the filesystem is intact and all you want is the data on it, copy it over with rsync.

partimage ( or similar tools, depending on the filesystem you're using ): If you want to make an exact copy of the drive but without copying the sectors that don't contain any data, then use partimage. This can save lots and lots of space. But only do it, if you are certain that the filesystem isn't corrupt and all the data you want is in there, because you lose every data that is in unused sectors. If you decide on using partimage, the easiest way to do this is to use clonezilla, a nice frontend for different disk imaging tools. Keep in mind that you don't have direct access to the files, you have to recreate and mount the actual image in order to get access to them.

ddrescue: If you think the drives will fail any minute and you want to get all the data off of them as soon as possible for later recovery, then use this tool, as described in Journeyman Geek's answer. Keep in mind though that this needs a lot of disk space and additional effort to actually get the data from the images.

As for the RaspberryPi: You can use the RaspberryPi for this with the only limitation being the transfer speed of the USB ports (as long as you don't compress your data). So if you don't mind it being a little slow, using the RaspberryPi is totally fine. You could also use any other Computer with a GNU/Linux distribution running on it, or even a LiveCD ( like sysresccd, partedmagic or ClonezillaLive ).

All the above Software is available on almost all GNU/Linux distributions, including Debian and Raspbian, so availability isn't a problem.