Critique My Backup and Storage Plan

Question

My current storage (RAID-1 off of a hardware RAID card) and backup (a spare drive) solutions for my home network are inadequate. I have too much data scattered on various one-off drives. It is time to evolve. Backups seem simple enough, at least: lots of big drives. However, I am bewildered by the number of choices for small home storage. The Drobo S looks appealing. So does the ReadyNAS. I am not looking for bunches of shiny features, I'm mostly interested in reliability. I am not interested in building Yet Another PC to create a file server or doing something in the cloud, or whatever. I'm stupid, so I am keeping it simple.

Requirements for Main Volume:

1. Starting working space roughly 2TB, with options for growth up to 5TB
2. RAID or something RAID-like with at least one parity drive
3. eSATA II for speed during backups
4. Ability to shut down gracefully when alerted of low power by a UPS
5. Optional but Desirable: Will take 2TB drives now with options for the larger 3TB drives coming in 2010-2011
6. Optional but Desirable: : RAID-6 or something similar, with two parity drives
7. Optional but Desirable: : Hot spare
8. Ethernet connection not required, as the volume will be shared via the same machines which runs my home print server

Backups:

1. Backup performed via ROBOCOPY in mirror mode to an external hard drive via a eSATA II connection.
2. Start with rotating between two external 2TB hard drives, will go up to six external 2TB drives.
3. Start with a weekly backup, move to a bi-weekly backup as more drives are added.
4. Move to 3TB drives as the size of my main volume increases.
5. Backup drives will be stored on an off-site location.

Hard drives:

1. I plan on buying all of the same model, but different batches from different vendors.
2. I found a "burn-in" utility with which I can pound away on the drives for a couple of weeks before adding them to the backup pool or the main volume.

I estimate that I am looking at roughly $1,500 to start, once I start throwing in two TB drives for backup and four for storage. So, are there any obvious flaws in my plan? What have I overlooked? Any suggestions for the storage device for my main volume that fits my requirements? Or do I just keep it simple, 2 drives in RAID-1, then perform due diligence with my backups, accepting that I will have to buy a whole new unit when my data grows past 2TB?

Answer 1

Main Volume

I know you said you don't want to build another PC fileserver, but most of the ready-to-go solutions don't have any safeguards against silent data corruption.

If you're looking for data integrity and reliability, you might want to consider running an OpenSolaris fileserver with a raidz2 or raidz3 configuration (2 or 3 parity drives, respectively) on ZFS.

With larger drives, the rebuild time will increase when a drive fails--which also increases the chances of a second-drive failure during the rebuild. But the main advantage of ZFS is that it protects you against silent data corruption, since the filesystem itself is checksummed.

You can also run ZFS on other operating systems, but OpenSolaris is always the most up-do-date version since it takes a while to port new features to the other platforms. If setting up an OpenSolaris box seems a little more work than what you want, FreeNAS seems to be the next best thing, in terms of ZFS support.

On the Linux side, ZFS is not supported in the kernel (only as a user-level driver), but there is also a new filesystem under development, called btrfs. Unfortunately, there is no stable release of btrfs, as of March 2010.

Backup

For your offsite backups, it might be more cost-effective to pay for a service like CrashPlan, Carbonite, or Mozy. It's very, very easy to configure any of these to automatically backup your files. Of the three, CrashPlan has the best backup and recovery features (and even allows you to backup to other remote computers for free), while Mozy's recovery methods are either expensive or very inconvenient (if you want to download a Mozy backup, you have to wait for your job to be queued up and bundled into a zip file). I haven't personally had any experience with Carbonite.

Note that you shouldn't depend solely on an offsite backup--if you backup to the cloud or some other offsite computer, you should also have a local backup.

The Drobo reviews I've seen noted poor write performance, but if you're just using it as a nightly backup drive, it might be sufficient.

Backup rotation

If you want to rotate backups between a local and off-site location, you need at least 3 backups to guarantee one is always local and one is always safe at the off-site location. The third is either in-transit or at one of the other two locations at any given point in time.

ROBOCOPY vs. CrashPlan

ROBOCOPY will cause more wear and tear on your hardware, since it has to read every file during every backup. It's not clear to me whether it only copies changed files or if it copies all files. If ROBOCOPY fails for some reason, it may not be apparent that it has failed, unless you have set something up to reliably report its backup status.

CrashPlan monitors your hard drive for changed files, and only copies the changed files. Since it actively monitors changes to disk, it does not need to read every file in your backup source. CrashPlan automatically e-mails you to notify you how long it has been since the last backup, and how much data was transferred during the last backup.

That said, keep in mind that CrashPlan doesn't have to replace your ROBOCOPY backup scheme. You can use CrashPlan to supplement whichever other backup scheme you happen to choose.

Answer 2

Your question is specifically called "Critique My Backup and Storage Plan". So at the risk of getting down voted because I'm not going to tell you what you want to hear, here's my opinion of your plan. It's overly expensive and overly complex.

I am not interested in building Yet Another PC to create a file server or doing something in the cloud, or whatever. I'm stupid, so I am keeping it simple.

Seriously? Having to setup backup scripts with Robocopy (which can't actually backup in-use files)? Manually transporting drives off-site? Hardware raid? Buying drives from multiple vendors and running "burn-in" utilities on them? All of that is the opposite of simple.

For all the hype there is around "cloud" services, home data backup really is one of the killer uses for it. I personally use CrashPlan because the software is well behaved and dead simple to setup. But they're all roughly the same in terms of price. Expect to pay around $3-$5 bucks a month total for unlimited storage from all of your machines. The "win" with cloud based backups is that they take all the hassle out of getting your data off-site. But no matter what service you use, restorals are a pain since downloading gigs of data will take a while. So you should augment it with a local backup solution as well. And since you've already got the cloud based backup working, your local backup solution doesn't need to be bullet proof in terms of reliability to be useful.

Here's the setup I have for my house. I built a small low-powered Atom based "home server" with four 1TB disks in a software raid 5 array with no hot spares. To be clear, the raid here is not for reliability. It's for the simplicity of creating a single backup drive. The server is running Windows and a copy of CrashPlan which is not configured to backup anywhere. The rest of the machines in the house are also running CrashPlan and configured to backup to both the CrashPlan Central cloud and my home server. That's it. Everything happens automatically and I get backup status reports via email from CrashPlan to make sure everything is running smoothly.

If any client machine in the house dies, it can be restored from the home server very quickly since it's all local. If the home server dies, there's nothing to restore since it was just a local repository for everyone else's backups. I just re-build it and re-point all the clients to the new/fixed box. If my whole house is vaporized from orbit, I've still got the cloud backup to restore from as a last resort.

Answer 3

Only the one backup? That's a mistake :) Especially putting so many resources into it, it'd be a pain if, for example, a natural disaster knocked that backup, and your own, out.