Raid 5 rebuild with wrong stripe size & no failed disks - is data lost?

Question

I'm curious if there are hidden complexities to RAID5 and RAID5 rebuilds.

My understanding is that (in a 3 disk array) a rebuild would just directly compute the parity of disks 1 & 2 and write that parity to disk 3 (or whatever disk was replaced). If it is really this simple, is there any way for a rebuild to cause data loss if no disks were actually replaced?

For example... "hypothetically", if I configured my raid array with all disks & original data still there, but set the wrong symmetry, wrong stripe size, disks in the wrong order, etc, and then for some reason I resynced or reconstructed the array, could data be lost (or even changed at all)?

Answer 1

If you remove a disk from your RAID, but the RAID still works because the missing disk was covered by redundancy, and then you add a replacement disk, then that added disk will be overwritten. There is no data loss in that the data you see on the /dev/mdX device does not change. However if you had important data on that added disk, it is lost due to overwrite.

So it all depends on whether the data on the remaining disks was intact or not.

set the wrong symmetry, wrong stripe size, disks in the wrong order, etc

Try to avoid mdadm --create to recover a RAID. It's the worst option because it's so easy to get wrong (so many variables, like disk order, chunk size, raid level, raid layout, metadata version, data offsets, ..., and mdadm's defaults for those change over time so you must specify all of them correctly).

If you must use it, you should do it in read-only mode: backups of metadata regions, on a snapshot layer, missing disk, --assume-clean to prevent it from syncing, ...

Whether a bad sync destroys or changes your data depends on how badly you got your settings wrong. Basically in a RAID5, provided all disks are intact and all parity data is correct, you can sync with any disk order and any chunk size. The sync itself won't cause any damage then. XOR is XOR, regardless of order or size, the result is the same.

# truncate -s 128M a b c
# losetup -f --show a b c
/dev/loop0
/dev/loop1
/dev/loop2
# mdadm --create /dev/md42 --level=5 --chunk=512 --raid-devices=3 /dev/loop0 /dev/loop1 /dev/loop2
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# shred -n 1 /dev/md42
# mdadm --stop /dev/md42
mdadm: stopped /dev/md42
# dd if=/dev/loop0 bs=1M skip=8 | md5sum
98f23e09198f7943e27030e8e0f9bc30  -
# dd if=/dev/loop1 bs=1M skip=8 | md5sum
90ef43c60e540b930c6db8dae831f8ab  -
# dd if=/dev/loop2 bs=1M skip=8 | md5sum
d010c8ae141fadc3347e8ed319b76db9  -
# mdadm --create /dev/md42 --level=5 --chunk=64 --raid-devices=3 /dev/loop2 /dev/loop0 /dev/loop1
mdadm: /dev/loop2 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Fri Sep  5 11:35:51 2014
mdadm: /dev/loop0 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Fri Sep  5 11:35:51 2014
mdadm: /dev/loop1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Fri Sep  5 11:35:51 2014
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# mdadm --wait /dev/md42
# mdadm --stop /dev/md42
mdadm: stopped /dev/md42
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/loop0 bs=1M skip=8 | md5sum
98f23e09198f7943e27030e8e0f9bc30  -
# dd if=/dev/loop1 bs=1M skip=8 | md5sum
90ef43c60e540b930c6db8dae831f8ab  -
# dd if=/dev/loop2 bs=1M skip=8 | md5sum
d010c8ae141fadc3347e8ed319b76db9  -

However, if a disk is missing from the set, or a surplus disk in the set; or if you change other things like partition offsets on one of the disks; or you use a different RAID level, it no longer works that way and you lose whatever is written by the sync. And whether you can recover from there depends on whether the data on the other disks still has redundancy to cover for what was overwritten. There is no simple solution to recover from such a situation.