I've found myself in a complex Linux LVM-RAID5 / device-mapper problem after one temporary device failure, and one bad block encountered during that time. The logical volume can now not be activated due to what seem to be underlying device-mapper errors.
What:
- CentOS 6.10 system
- 4 disk (SATA) volume group (vg03), /dev/sd[a,b,c,d]
- single logical volume (lvol1) in RAID5.
Timeline: I've been away and wasn't able to fix one problem before a second took place.
- Physical Volume at /dev/sdc went offline, later recovered
- After /dev/sdc failed, but before it was recovered, /dev/sdb encountered SMART errors for offline uncorrectable sectors.
- At that time, the logical volume /dev/vg03/lvol1 access failed with i/o errors on all file reads.
- /dev/sdc was recovered via a physical re-seat in the enclosure.
- The logical volume can no longer be activated
I have done a vgextend --restoremissing /dev/sdc which was successful, the VG no longer complains that a PV is missing. But the LV still will not activate. LVM as a whole seems happy now to be honest. a vg config dump reports all rimage and rmeta volumes visible. In fact, the vg config dump looks identical to a backup taken from when things were working fine.
Despite a complete volume group and accessibility of all rimage and rmeta components, attempts to activate the logical volume all fail, as follows:
[root@fw log]# vgchange -a y vg03
device-mapper: reload ioctl on (253:119) failed: Input/output error
0 logical volume(s) in volume group "vg03" now active
[root@fw log]# lvchange -a y /dev/vg03/lvol1
device-mapper: reload ioctl on (253:119) failed: Input/output error
[root@fw log]# lvchange -a y --activationmode partial /dev/vg03/lvol1
PARTIAL MODE. Incomplete logical volumes will be processed.
device-mapper: reload ioctl on (253:119) failed: Input/output error
[root@fw log]# lvchange -a y --activationmode degraded /dev/vg03/lvol1
device-mapper: reload ioctl on (253:119) failed: Input/output error
The issue APPEARS to be with device-mapper reconstructing the rimage and rmeta components back into a usable RAID5 dataset.
I suspect:
- after rimage3 went offline, the RAID array became degraded, but available.
- rimage2 then encounted an unreadable block, and it's whole rimage was marked dirty
- rimage3 has now returned, but because RAID life proceeded while it was away, this RAID component is now considered by device-mapper to be stale, and unusable.
dmesg output indicates only 2 of 4 devices are present, despite vgchange -vvvv output showing all 4 rimage and rmeta volumes were found - and I can even dd read their contents without issue.
Log sections at bottom to not pollute the narrative.
I saw a similar (but for a boot LV) situation answered by Max Ehrlich to add the following to the kernel params to recover a dirty RAID LV. md-mod.start_dirty_degraded=1. Unfortunately, this has not changed any behaviour.
So.. if anyone more familiar with device mapper can offer any guidance on activating a Linux RAID5 logical volume with two complete and one of EITHER one dirty component, or one stale one?
The data on the fs was actually unchanged during this entire time period, and realistically, I would expect it should be reconstructable again even despite the bad block on one member. That stale volume is stale in timestamp only, not data content.
vgchange -vvvv -ay
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFhAiMp10vcg5xfyRWcZ00bC4Xm7RiPXGv in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFhAiMp10vcg5xfyRWcZ00bC4Xm7RiPXGv in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rmeta_0
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rmeta_0 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFuF9rQHu65xgUwdaP8AmJdLczSgj8dr5m in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFuF9rQHu65xgUwdaP8AmJdLczSgj8dr5m in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rimage_1
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rimage_1 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFJTFmG1s0VFNDeZdEwzOPjpvyiZhkWocv in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFJTFmG1s0VFNDeZdEwzOPjpvyiZhkWocv in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rmeta_1
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rmeta_1 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFbXrEcOcr4bs0ldIuifV0GQchgVd1T3jr in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFbXrEcOcr4bs0ldIuifV0GQchgVd1T3jr in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rimage_2
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rimage_2 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFjKHZikMCfUSAcWsdgoon4zHezd6eP149 in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFjKHZikMCfUSAcWsdgoon4zHezd6eP149 in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rmeta_2
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rmeta_2 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECF86L9mwnSyHLlDbDd6YfmwPddLBcQyHYT in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECF86L9mwnSyHLlDbDd6YfmwPddLBcQyHYT in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rimage_3
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rimage_3 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFb8QDktJ3UeEXm5d0xAyPjqQOY4anOPLO in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFb8QDktJ3UeEXm5d0xAyPjqQOY4anOPLO in deptree.
#activate/dev_manager.c:2575 Checking kernel supports striped segment type for vg03/lvol1_rmeta_3
#metadata/metadata.c:2516 Calculated readahead of LV lvol1_rmeta_3 is 256
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFhAiMp10vcg5xfyRWcZ00bC4Xm7RiPXGv in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFval0zTemeWZL2fu7nyWm1MBlktzPjIO6 in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFJTFmG1s0VFNDeZdEwzOPjpvyiZhkWocv in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFuF9rQHu65xgUwdaP8AmJdLczSgj8dr5m in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFjKHZikMCfUSAcWsdgoon4zHezd6eP149 in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFbXrEcOcr4bs0ldIuifV0GQchgVd1T3jr in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFb8QDktJ3UeEXm5d0xAyPjqQOY4anOPLO in deptree.
#libdm-deptree.c:570 Matched uuid LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECF86L9mwnSyHLlDbDd6YfmwPddLBcQyHYT in deptree.
#libdm-deptree.c:2693 Loading vg03-lvol1_rmeta_0 table (253:111)
#libdm-deptree.c:2637 Adding target to (253:111): 0 8192 linear 8:48 2048
#ioctl/libdm-iface.c:1838 dm table (253:111) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rmeta_0 (253:111) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rimage_0 table (253:112)
#libdm-deptree.c:2637 Adding target to (253:112): 0 7507812352 linear 8:48 10240
#ioctl/libdm-iface.c:1838 dm table (253:112) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rimage_0 (253:112) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rmeta_1 table (253:113)
#libdm-deptree.c:2637 Adding target to (253:113): 0 8192 linear 8:0 2048
#ioctl/libdm-iface.c:1838 dm table (253:113) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rmeta_1 (253:113) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rimage_1 table (253:114)
#libdm-deptree.c:2637 Adding target to (253:114): 0 7507812352 linear 8:0 10240
#ioctl/libdm-iface.c:1838 dm table (253:114) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rimage_1 (253:114) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rmeta_2 table (253:115)
#libdm-deptree.c:2637 Adding target to (253:115): 0 8192 linear 8:16 2048
#ioctl/libdm-iface.c:1838 dm table (253:115) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rmeta_2 (253:115) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rimage_2 table (253:116)
#libdm-deptree.c:2637 Adding target to (253:116): 0 7507812352 linear 8:16 10240
#ioctl/libdm-iface.c:1838 dm table (253:116) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rimage_2 (253:116) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rmeta_3 table (253:117)
#libdm-deptree.c:2637 Adding target to (253:117): 0 8192 linear 8:32 2048
#ioctl/libdm-iface.c:1838 dm table (253:117) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rmeta_3 (253:117) identical table reload.
#libdm-deptree.c:2693 Loading vg03-lvol1_rimage_3 table (253:118)
#libdm-deptree.c:2637 Adding target to (253:118): 0 7507812352 linear 8:32 10240
#ioctl/libdm-iface.c:1838 dm table (253:118) OF [16384] (*1)
#libdm-deptree.c:2728 Suppressed vg03-lvol1_rimage_3 (253:118) identical table reload.
#libdm-deptree.c:1981 Creating vg03-lvol1
#ioctl/libdm-iface.c:1838 dm create vg03-lvol1 LVM-NYeT7Gk1h6roCjIRVRdM3YQAoyuBcECFe2A4KJCjCU5J9sSzIVdJLRh1xF7xntYx NF [16384] (*1)
#libdm-deptree.c:2693 Loading vg03-lvol1 table (253:119)
#libdm-deptree.c:2637 Adding target to (253:119): 0 22523437056 raid raid5_ls 3 128 region_size 16384 4 253:111 253:112 253:113 253:114 253:115 253:116 253:117 253:118
#ioctl/libdm-iface.c:1838 dm table (253:119) OF [16384] (*1)
#ioctl/libdm-iface.c:1838 dm reload (253:119) NF [16384] (*1)
#ioctl/libdm-iface.c:1876 device-mapper: reload ioctl on (253:119) failed: Input/output error
#libdm-deptree.c:1022 Removing vg03-lvol1 (253:119)
#libdm-common.c:2338 Udev cookie 0xd4de97c (semid 720905) created
#libdm-common.c:2358 Udev cookie 0xd4de97c (semid 720905) incremented to 1
#libdm-common.c:2230 Udev cookie 0xd4de97c (semid 720905) incremented to 2
#libdm-common.c:2480 Udev cookie 0xd4de97c (semid 720905) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK (0x20)
#ioctl/libdm-iface.c:1838 dm remove (253:119) NF [16384] (*1)
#libdm-common.c:1430 vg03-lvol1: Stacking NODE_DEL [trust_udev]
device-mapper log output from dmesg
md/raid:mdX: device dm-114 operational as raid disk 1
md/raid:mdX: device dm-112 operational as raid disk 0
md/raid:mdX: allocated 0kB
md/raid:mdX: not enough operational devices (2/4 failed)
RAID conf printout:
--- level:5 rd:4 wd:2
disk 0, o:1, dev:dm-112
disk 1, o:1, dev:dm-114
disk 2, o:1, dev:dm-116
disk 3, o:1, dev:dm-118
md/raid:mdX: failed to run raid set.
md: pers->run() failed ...
device-mapper: table: 253:119: raid: Fail to run raid array
device-mapper: ioctl: error adding target to table
