I ran into what is a true mystery to me today, and would love if someone could give me an idea of how what happened is even possible.
We had physically removed a disk from our software RAID1 (Ubuntu 14.04, if it matters) previously in the day, and went to re-add it. (/dev/sda2 was the live partition in the RAID, and we wanted to re-add /dev/sdb2 and have it re-sync, /dev/md1 is the RAID array in question)
We issued the command: mdadm /dev/md1 --add /dev/sdb2
Nearly immediately we noticed that the drive didn't seem to be syncing, as a check of /proc/mdstat indicated the drive was already incorporated in the array.
Within a minute, applications crashed, ext4 failed and remounted itself read-only. It appears that the drive was re-added to the array without being sync'd with the existing one. Output from syslog, appears to confirm that:
[ 1607.017583] md: bind<sdb2>
[ 1607.019831] RAID1 conf printout:
[ 1607.019839] --- wd:1 rd:3
[ 1607.019842] disk 0, wo:1, o:1, dev:sdb2
[ 1607.019845] disk 1, wo:0, o:1, dev:sda2
[ 1607.019911] md: recovery of RAID array md1
[ 1607.019915] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1607.019917] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 1607.019923] md: using 128k window, over a total of 341795704k.
[ 1607.020187] md: md1: recovery done.
[ 1607.022356] RAID1 conf printout:
[ 1607.022359] --- wd:2 rd:3
[ 1607.022361] disk 0, wo:0, o:1, dev:sdb2
[ 1607.022364] disk 1, wo:0, o:1, dev:sda2
...
[ 1651.932310] init: mysql main process (1192) terminated with status 1
[ 1651.932329] init: mysql main process ended, respawning
[ 1654.691865] init: mysql main process (9271) terminated with status 1
[ 1654.691883] init: mysql main process ended, respawning
[ 1655.004970] init: mysql post-start process (9272) terminated with status 1
[ 1656.937186] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 494, block bitmap and bg descriptor inconsistent: 1284 vs 8190 free clusters
[ 1656.937268] Aborting journal on device md1-8.
[ 1656.937718] EXT4-fs (md1): Remounting filesystem read-only
[ 1656.937921] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 495, block bitmap and bg descriptor inconsistent: 479 vs 10240 free clusters
[ 1656.938181] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 496, block bitmap and bg descriptor inconsistent: 23892 vs 23866 free clusters
[ 1656.938411] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 497, block bitmap and bg descriptor inconsistent: 11610 vs 16699 free clusters
[ 1656.938639] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 498, block bitmap and bg descriptor inconsistent: 2560 vs 10642 free clusters
[ 1656.938782] EXT4-fs error (device md1): ext4_mb_generate_buddy:756: group 499, block bitmap and bg descriptor inconsistent: 3830 vs 14336 free clusters
Of particular note is the line that says it needs to sync 341795704k, and then a millisecond later stating "recovery done".
How did this happen and what did we do wrong, if anything, to allow this to happen?
