What's the difference between creating mdadm array using partitions or the whole disks directly

Question

Can someone please explain to me, what the difference is between creating mdadm array using partitions or the whole disks directly? Supposing I intend to use the whole drives.

Imagine a RAID6 created in two ways, either:

mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

or:

mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

What is the difference, and possible problems arising from any of the two variants?

For example, I mean the reliability or manageability or recovery operations on such arrays, etc.

Answer 1

The most important difference is that it allows you to increase the flexibility for disk replacement. It is better detailed below along with a number of other recommendations.

One should consider to use a partition instead of the entire disk. This should be under the general recommendations for setting up an array and may certainly spare you some headaches in the future when further disk replacements get necessary.

The most important arguments is:

Disks from different manufacturers (or even different models of the "same" capacity from the same manufacturer) don't necessarily have the exact same disk size and, even the smallest size difference, will prevent you from replacing a failed disk with a newer one if the second is smaller than the first. Partitioning allows you to workaround this;

Side note on why to use different manufacturers disks: Disks will fail, this is not a matter of a "if" but a "when". Disks of the same manufacturer and the same model have similar properties, and so, higher chances of failing together under the same conditions and time of use. The suggestion so is to use disks from different manufacturers, different models and, in special, that do not belong to the same batch (consider buying from different stores if you are buying disks of the same manufacturer and model). This is not uncommon that a second disk fail happen during a resotre after a disk replacement when disks of the same batch are used. You certainly don't want this to happen to you.

So the recommendations:

1) Partition the disks that will be used with a slightly smaller capacity than the overall disk space (e.g, I have a RAID5 array of 2TB disks and I intentionally partitioned them wasting about 100MB in each). Then, use /dev/sd?1 of each one for composing the array - This will add a safety margin in case a new replacing disk has less space than the original ones used to assemble the array when it was created;

2) Use disks from different manufacturers;

3) Use disks of different models if different manufacturers are not an option for you;

4) Use disks from different batches;

5) Proactively replace disks before they fail and not all at the same time. This may be a little paranoid and really depends on the criticity of the data you have. I use to have disks that have 6 months differences in age from each other;

6) Make regular backups (always, regardless if you use an array or not). Raid doesn't serve the same purpose of backups. Arrays assure you high availability, Backups allow you to restore lost files (including the ones that get accidentally deleted or are damaged by viruses, some examples of something that using arrays will not protect you from).

OBS: Except for all the non-neglectable rational above, there aren't much further technical differences between using /dev/sd? vs /dev/sd?#.

Good luck

Answer 2

Another important argument is that some mainboards may delete your RAID superblocks if you use whole disk devices and are not super careful with wiping them when adding disks to a RAID array that once were GPT devices.

I learned this the hard way, when my ASRock mainboard's UEFI rendered my RAID unusable:

• http://forum.asrock.com/forum_posts.asp?TID=10174
• https://news.ycombinator.com/item?id=18541493
• https://www.phoronix.com/scan.php?page=news_item&px=Linux-Software-RAID-ASRock

To protect yourself against this, always run sgdisk --zap on a formerly GPT-formatted disk that you want to use for a whole-device RAID.

wipefs -a may also work to wipe all kinds of leftover partitioning info (not only GPT) before adding the device to the disk.

---

That said, just using partitions seems like a best practice to make this less likely to happen.

I never knew what the real arguments were for this best practice (beyond @Marcelo's answer explaining that it can deal with disks of slightly different sizes). Now I know.

By the way, if this happens to you, your data is not lost. You can most likely just sgdisk --zap the device, and then recreate the RAID with e.g. mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd (mdadm will tell you that it already detects past data, and asks you if you want to continue re-using that data). I tried this mutliple times and it worked, but I still recommend taking a backup before you do it.

Answer 3

There are couple of posts citing problem with using the whole device as a RAID member instead of partition, one here. I can confirm that to be the case.

So, if you create RAID using the whole device (say, /dev/sda, /dev/sdb, ... instead of on /dev/sda1, /dev/sdb1, ...) the RAID device doesn't reassemble after a reboot even if you have configuration saved under mdadm.conf. There is also a possibility of mainboard firmware overwriting or removing the RAID superblock. In my setup, running mdadm --assemble --scan --verbose doesn't get me back my RAID setup and it seems to be the case of RAID superblock not being available.

Answer 4

For saving yourself from future headaches, use partitions. All drives fail, it's not a question of "if" but "when". If you are using whole drives, then after a drive failure, you must replace it with exactly the same size model. Even a few MB larger, due to different manufacturer, or process changes, whatever, will cause the drive replacement to fail.

Using partitions, when one of your current 2TB drives fails, you get a replacement 4TB (or whatever, just needs to be larger), then create the same 2TB partition size on it. Yes, you may be wasting space on that new large drive. Using the rest of that drive as a non-raid partition may be possible, but could affect performance in weird ways. After several drive failures and replacements, you will eventually replace all of your original 2TB drives, and now have the ability to extend your RAID array to use larger partition sizes.

Answer 5

It is not uncommon at all to put disks rather then partitions together to a RAID, before then partitioning the RAID. In particular, most hardware RAID controllers cannot even read partition tables, so they always operate on whole disks. But the same is possible with software RAIDs. And if configured properly, software RAIDs will reassamble fine on reboot. Of course, make sure, that no trace of a former partition table is left on the disk. This is paricularly important, as many disks are delivered preformatted, and many tools (software RAID, filesystems etc.) tend to leave "empty sectors" at the beginning, where these traces of former partition tables then can survive.

It also pays off to buy one or two extra disks as spare, when you set up a RAID. If any disk fails, you can replace it quickly, which is particularly important in a RAID5 or two-disk RAID1, where after a single disk failure, you are left with no redundancy, until the failed disk has been replaced and the RAID has been re-synced. By having an exact matching spare, you also avoid those problems, that can arise from a "weak" disk in a RAID, like when the new disk is slower or a tad smaller than the old ones.

Nonetheless, the "tad smaller" problem can be fixed whether or not you have partitions. Just use the device mapper to create a virtual disk, that gets 99.9% of its capacity from the new disk and the missing 0.1% from a system disk or SSD. Alternatively, you can shrink the file system during a reboot in the initramfs stage (see https://serverfault.com/a/888830 how to do this), then shrink the RAID and you are back online, too.

In other words: Using entire disks for a hardware or software RAID or a ZFS memory pool is perfectly fine.