RAID5 mdadm faulty / blk_update_request: I/O error

Question

I have five raw images of a server's disks (stored on one external HDD). Each of those disks contains two partitions which are "Linux raid autodetect" (fd) and are members of RAID1 (boot-related; 3 devices (2 active + 1 spare)) and RAID5 (other data; 5 devices).

Having mapped devices using kpartx (or losetup, doesn't really matter) I run mdadm to assemble RAID arrays. RAID1 successfully assembled and I can mount it in a regular way. RAID5 is being assembled (actually "active", according to /proc/mdstat) but:

1. All devices are "faulty" and "removed";
2. I see additional errors in dmesg (blk_update_request: I/O error, dev loop4, sector 16) after which I get more errors (md: super_written gets error=-5; md/raid:md0: Disk failure on loop5, disabling device.);
3. I can't mount RAID5 array (fair enough).

I tried all suggestions I could find on the web:

1. Use --force key -> won't work;
2. Use --create and --assume-clean -> won't work;
3. Use --zero-superblock (which is similar to 2 ) -> won't work.
4. If check with --examine - all member devices are in clean state, checksum is correct and event numbers are equal.

As per 2 and 3 - it seems that I CAN'T write to those mapped devices. I strace'd --zero-superblock and, on write() syscall, I got EPERM (Operation not permitted). Can't imagine what prevents me from writing to mapped devices (according to lsblk --fpm RAID5 device is writable brw-rw----).

Another peculiar detail. Above I mentioned that I get I/O error on sector 16. I checked what's located at sector 16 (offset 8192 bytes) in every loop (mapped) RAID5 device. Appeared that it starts with magic number 0x6d746962 (ASCII: "bitm") - likely bitmap related:

00002000  62 69 74 6d 04 00 00 00  48 a7 15 94 56 79 23 ed  |bitm....H...Vy#.|<br/>
00002010  7d 7b 33 bc b9 f8 71 7a  d0 03 00 00 00 00 00 00  |}{3...qz........|<br/>
00002020  d0 03 00 00 00 00 00 00  00 b0 d8 0d 00 00 00 00  |................|

I'm not sure if it's related (at least I don't know how to play with bitmap while assembling array) but it's clear that system can't succeed with blk_update_request operation (likely it either lacks some rights or can't find what's needed).

I on purpose won't post wall of logs - please, let me know if it's needed. I'll enclose some info which describes situation better.

md0 : active raid5 dm-6[5](F) dm-8[4](F) dm-2[3](F) dm-4[2](F) dm-10[1](F)
      464609280 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/0] [_____]
      bitmap: 0/1 pages [0KB], 65536KB chunk

And here is from --detail output:

Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       0        0        2      removed
       4       0        0        4      removed
       6       0        0        6      removed
       8       0        0        8      removed

       1     252       10        -      faulty   /dev/dm-10
       2     252        2        -      faulty   /dev/dm-2
       3     252        4        -      faulty   /dev/dm-4
       4     252        8        -      faulty   /dev/dm-8
       5     252        6        -      faulty   /dev/dm-6

Today is a 3rd day in a row of my "tries harder"... I would appreciate any of your advices. Thanks.

Answer 1

So, finally, I managed to assemble the RAID5. As appeared, the problem was in the the following (I haven't mentioned it in my question). Disks' images I had were in E01 format, split to number of parts. Prior mapping partitions I used ewfmount (including -X allow_root) to "assemble" multiple E01s into one file. Turned out that FUSE, despite you see your images mounted as RW, somehow blocks them, making devices read-only (even if you edit /etc/fuse.conf).

One of my tries, while troubleshooting, was to export multiple E01s to five regular dd/raw formatted images and work with them directly, excluding ewfmount from the 'scheme".

Once export was completed I mapped raw disks using kpartx, and had been able to assemble RAID5 in a regular way with no problems.

Multiple Operation not permitted or I/O error problems were obvious and had to giveme a hint but I, due to unknown reasons, denied them (until I tried to modify some sectors on mapped devices with dd directly and failed).

That's it.