Here's a brief history of how I got here:
- 4x disk Ubuntu 12.04 software RAID10 with 5x partitions (md0 - md4)
- 1x disk died
mdadm --fail
>mdadm --remove
> physically removed drive and replacemdadm --add
> disk resync'd perfectly for all partitions- decided to replace all the disks so they were identical
- repeat steps 3-4 for remaining 3 disks. 2nd and 3rd disk went perfectly.
- after final disk was replaced I added it back to the array but was notified that the file system was in read-only mode.
cat /proc/mdstat
revealed that some partitions had dropped out but it was very inconsistent.- I rebooted the machine (probably not the smartest idea)
- Machine wouldn't boot (no MBR on the new disk I assume).
- Replaced last drive I had taken out. Machine boots to
intitramfs
prompt but keyboard unresponsive. - Remove last drive so now only the 3x good disks remain.
- Boot from Ubuntu Live USB.
- Ubuntu disk utility lists the 4x RAID devices says they are inactive and partially assembled.
ubuntu@ubuntu:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md3 : inactive sdd8[6](S) sdc8[5](S) sdb8[4](S) 1464837120 blocks super 1.2 md4 : inactive sdd9[6](S) sdc9[5](S) sdb9[4](S) 718365696 blocks super 1.2 md1 : inactive sdd6[6](S) sdc6[5](S) sdb6[4](S) 146479104 blocks super 1.2 md2 : inactive sdd7[6](S) sdc7[5](S) sdb7[4](S) 585931776 blocks super 1.2 md0 : inactive sdd5[6](S) sdc5[5](S) sdb5[4](S) 14641152 blocks super 1.2 unused devices: <none>
ubuntu@ubuntu:~$ sudo mdadm --assemble --verbose /dev/md0 -f /dev/sdb5 /dev/sdc5 /dev/sdd5
mdadm: looking for devices for /dev/md0 mdadm: cannot open device /dev/sdb5: Device or resource busy mdadm: /dev/sdb5 has no superblock - assembly aborted
So now I'm a bit stuck! The 3x disks in there were all consistent at the moment that the 4th disk was replaced. SMART checks come out ok (no bad sectors, etc.).
I just need a way to restore the array with 3x disks so I can re-add the 4th. Any thoughts?
Many thanks!