mdadm - Raid 1 arrays do not assemble after reboot

Question

I rebooted my server after creating 3 raid 1 arrays with mdadm, and after rebooting, systemd was spitting out errors about the raid devices timing out when trying to assemble them at boot.

I was given an option to enter the root password for maintainance, which I have done.

I tried running

mdadm --assemble --scan

but this produced no output.

I'm not sure what else I should check.

This is a debian 10 system, which runs inside a KVM (lib-virt) on another instance of debian 10. There is a HBA which is passed through as a pci device to the VM. This may or may not be relevant.

afaik the mdadm.conf is setup correctly, and fstab is set correctly.

But, again, I might be mistaken.

What should I do now?

Data is probably not particularly important as we only copied things over last night. Still it would be nice not to lose everything since it took the whole day to copy.

Update

This is the output of

mdadm --examine --verbose /dev/sda

/dev/sda:
    MBR Magic: aa55
Parition[0]: 1953525167 sectors at 1 (type ee)

Is this helpful?

requested information

cat /proc/mdstat
Personalities : [linear] .... etc ....
unused devices: <none>

That's it - there's no list of raid devices there (sorry should have said this earlier)

Commands run:

For example, after my raids failed on the first reboot, I managed to rebuild them with the following commands without losing any data as far as I can tell

sudo mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
cat /proc/mdstat
.... syncing

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo reboot
sudo mdadm --assemble --scan
cat /proc/mdstat
.... nothing there

Answer 1

I tried many guides myself, most ending up with the same issue, but this specific one by Jeff Geerling worked for me. The key difference for me was partitioning each individual drives before you create the array with mdadm.

Answer 2

Apparently I never got around to posting an answer here, but I did manage to recover my drives/raid array/data.

In a nutshell, here's what I did: I repeated the steps to create the array again, and when assembled my data was "just there".

Here's roughly what went wrong:

My array could not be re-assembled because the superblock had been overwritten.

I don't remember exactly why this happened, but it was something like:

• either I created the partitions after creating the software array but I should have created them before creating the software array
• or I should have created the array using the software partitions (/dev/sda1/) rather than the disks themselves (/dev/sda)
• or some combination of the above or slight variation of

Basically: Be careful when doing your partitioning not to overwrite a superblock. And be careful when setting up your array not to overwrite your partition information. I would advice checking your setup works by doing a hard reboot after putting some unimportant data on the device. If you can do a restart of the machine without the array failing to reassemble, you are probably ok.