btrfs does not show drive as missing

Question

I am trying out btrfs and I noticed some funny behaviour:

I have two drives in a raid 1 configuration.

sudo btrfs fi show
Label: none  uuid: 52b9c6f6-ce5c-4727-861e-e2fd012e475f
    Total devices 2 FS bytes used 251.16GiB
    devid    1 size 298.09GiB used 252.01GiB path /dev/sdi
    devid    2 size 298.09GiB used 252.01GiB path /dev/sdh

As you can see they are set up as mdata=raid1, data=raid1:

sudo btrfs fi df /btr/.fsroot/
Data, RAID1: total=251.00GiB, used=250.57GiB
System, RAID1: total=8.00MiB, used=64.00KiB
Metadata, RAID1: total=1.00GiB, used=597.47Mi

Now if I fail /dev/sdh (e.g. pull out the sata cable), everything works as expected,

sudo btrfs fi show
Label: none  uuid: 52b9c6f6-ce5c-4727-861e-e2fd012e475f
    Total devices 2 FS bytes used 251.16GiB
    devid    1 size 298.09GiB used 252.01GiB path /dev/sdi
    *** Some devices missing

the filesystem is even unmounted. But what bothers me is what happens if I fail /dev/sdi:

Label: none  uuid: 52b9c6f6-ce5c-4727-861e-e2fd012e475f
    Total devices 2 FS bytes used 251.16GiB
    devid    1 size 298.09GiB used 252.01GiB path /dev/sdi
    devid    2 size 298.09GiB used 252.01GiB path /dev/sdh

So basically the failed drive is not detected at all. This can't even be fixed by running btrfs device scan. So in a production environment, how am I supposed to detect, that my raid is no more?

Answer 1

Have you scrubbed the BTRFS filesystem? A scrub should read and check all available copies. And it's recommended to schedule a scrub, for example as monthly cronjob, so this is how you would be notified.

---

For future reference: Unfortunately, BTRFS does not seem to be able to detect a failed drive "live", i.e., while the filesystem is mounted (*).
I guess they didn't think about server systems that run for months or even years without ever being unmounted. Let's hope that this has been fixed by now (while you are reading this)...

*: Quoting a mailing list entry from 09/2015:

The difference is that we have code to detect a device not being present at mount, we don't have code (yet) to detect it dropping on a mounted filesystem. Why having proper detection for a device disappearing does not appear to be a priority, I have no idea, but that is a separate issue from mount behavior.

https://www.mail-archive.com/[email protected]/msg46598.html

ZFS, for example, will realize that a drive that needs to be accessed is gone and marks it as FAULTED. For a server using BTRFS, it might be an idea to have a custom check (cron job) that sends an alert if at least one of the drives in the RAID array is not accessible anymore...

---

Degraded mounts: Anyone who manually removes a drive in order to simulate a failure will probably think about mounting the filesystem in degraded mode (e.g., to use "btrfs replace"). Mounting in degraded (rw) mode can be dangerous, it might even lead to data loss if you don't take care of resyncing your filesystem afterwards (balance):

Just an FYI to be really careful about degraded rw mounts. There is no automatic resync to catch up the previously missing device with the device that was degraded,rw mounted. You have to scrub or balance, there's no optimization yet for Btrfs to effectively just "diff" between the devices' generations and get them all in sync quickly.

Much worse is if you don't scrub or balance, and then redo the test reversing the device to make missing. Now you have multiple devices that were rw,degraded mounted, and putting them back together again will corrupt the whole file system irreparably.

https://www.mail-archive.com/[email protected]/msg46519.html