mdadm raid cretion: apparently superblock not written to disk

Question

I am in exactly the same predicament as MDADM raid “lost” after reboot:

I followed this giude to setup a raid-1 array with two disks (I was working in a recovery shell, so no "sudo"):

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
mkfs.ext4 -F /dev/md0
e2label /dev/md0 home
echo >>/etc/fstab 'LABEL=home /home ext4 defaults 0 2
mkdir -p /mnt/md0
mount /dev/md0 /mnt/md0
( cd /home && tar cf - * ) | ( cd /mnt/md0 && tar xvf - )

Everything went ok till next reboot (I waited for synchronization to finish successfully).

After reboot /dev/md0 was lost, I reissued the initial command:

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd

... and it recreated the disk, with complete content (I checked with "rsync"), but restarted synchronization again.

I waited once more for it to terminate (a few hours!) and rebooted again.

Again the Raid device is lost and apparently superblock was never written to disk as I find two empty partition tables at end of first sector (see aa55 signature).

root@cinderella:~# mdadm --assemble /dev/md0 /dev/sdc /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: /dev/sdc has no superblock - assembly aborted
root@cinderella:~# mdadm --misc --examine /dev/sd[cd]
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
root@cinderella:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
unused devices: <none>

I am prepared to redo everything from scratch, but it takes a lot of time (those are 2T disks and content I need to copy is ~1T) and I would like to understand what went wrong before taking another spin.

Sincerely this looks like a mdadm bug (serious one!), but I could be very wrong, of course.

Now I redid the creation and...

root@cinderella:~# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
mdadm: partition table exists on /dev/sdc
mdadm: partition table exists on /dev/sdc but will be lost or
       meaningless after creating array
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: partition table exists on /dev/sdd
mdadm: partition table exists on /dev/sdd but will be lost or
       meaningless after creating array
mdadm: size set to 1953382464K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
root@cinderella:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdd[1] sdc[0]
      1953382464 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  0.1% (2260544/1953382464) finish=230.1min speed=141284K/sec
      bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>
root@cinderella:~# mount -v /dev/md0 /mnt/md0
mount: /dev/md0 mounted on /mnt/md0.
root@cinderella:~# ls -la /mnt/md0
total 36
drwxr-xr-x   6 root  root   4096 Feb 18 22:40 .
drwxr-xr-x   4 root  root   4096 Feb 18 22:34 ..
drwxr-xr-x   7 mauro mauro  4096 Feb 23  2018 etherpad
drwx------   2 root  root  16384 Feb 18 11:10 lost+found
drwxr-xr-x  16 mauro mauro  4096 Nov 10 11:42 mauro
drwxr-xr-x 127 mcon  mcon   4096 Feb 19 15:09 mcon
root@cinderella:~# 

Any idea?

Answer 1

For whoever will be in my situation:

I dont't know if this is a bug or not, but it seems mdadm is very sensitive to a Partition Table in the MBR.

After a lot of trials I finally solved my problem doing the following:

• rebooot
• zap MBR on each drive in the array:
  • dd if=/dev/zero of=/dev/sdc bs=512 count=1
  • dd if=/dev/zero of=/dev/sdd bs=512 count=1
• recreate the array (this will trigger resync):
  • mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
• check content is not lost
  • mount /dev/md0 /mnt
  • examine contents
  • umount /mnt
  • wait for resync to terminate (otherwise fsck will be very slow)
  • fsck -pfC /dev/md0
• persist the array
  • mdadm --examine --scan >> /etc/mdadm/mdadm.conf
  • remove old persistency lines, if present.
  • update-initramfs -u
• reboot again (to be sure everything is ok).
• enjoy!

Answer 2

I know this is a zombie thread, but I was totally stuck with a raid array that wouldn't persist through reboot, and this solved the issue. The disks were recycled from another server.