I have a Synology fileserver (but this isn't a Synology-specific question) with a 5 TB RAID 5 ext4 volume that had two drives get disconnected at once. The volume "crashed" in Synology's parlance and I couldn't recover it via the web frontend; even Synology's own help pages said "too bad, contact your local data recovery company", but I eventually found some info on mdadm (see here) and was able to reattach the drives manually and rebuild the volume. Unsurprisingly, the filesystem complained of errors after that. The automated Synology process at boot just runs /sbin/e2fsck -C-1 -pvf <volume>, which generated this in the log:
1.41.12-3202: Inodes that were part of a corrupted orphan linked list found.
1.41.12-3202: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
So I held my breath and ran e2fsck -v on my volume and I think it came out pretty clean all things considered, but it's been a long time since I've had to do a deep dive on anything fsck-related and I could really use a more experienced eye. I answered y to everything on the assumption that getting back the vast majority of my data was better than ending up with a filesystem in some precarious state. Here is the final summary, I'll include some specific logs below it:
1.41.12-3202: ***** FILE SYSTEM WAS MODIFIED *****
120457 inodes used (0.03%, out of 365404160)
12198 non-contiguous files (10.1%)
87 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 8/8/8
Extent depth histogram: 109076/9963/53
1278688908 blocks used (87.49%, out of 1461594288)
0 bad blocks
993 large files
109013 regular files
10069 directories
0 character device files
0 block device files
0 fifos
438 links
1365 symbolic links (1355 fast symbolic links)
0 sockets
------------
120884 files
Pass 1 found about a half-dozen inodes that had issues, things like this:
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes
Inode 71173665 was part of the orphaned inode list. FIXED.
Inode 71173665 has imagic flag set. Clear<y>? yes
Inode 71173665 has a extra size (32032) which is invalid
Fix<y>? yes
Inode 71173666 is in use, but has dtime set. Fix<y>? yes
Inode 71173666 has imagic flag set. Clear<y>? yes
Inode 71173666 has a extra size (12336) which is invalid
Fix<y>? yes
...
Inode 71173677 has compression flag set on filesystem without compression support. Clear<y>? yes
Inode 71173677, i_size is 4189036383227115892, should be 0. Fix<y>? yes
Inode 71173677, i_blocks is 129061236121632, should be 0. Fix<y>? yes
Inode 71173678 was part of the orphaned inode list. FIXED.
Inode 71173678 has imagic flag set. Clear<y>? yes
Inode 71173678 has a extra size (28261) which is invalid
Fix<y>? yes
Then there was this:
Illegal block #1 (1768453154) in inode 71173665. CLEARED.
Illegal block #2 (1632658546) in inode 71173665. CLEARED.
Illegal block #7 (1952669986) in inode 71173665. CLEARED.
Illegal block #11 (1948393504) in inode 71173665. CLEARED.
Illegal block #12 (1671889091) in inode 71173665. CLEARED.
Illegal block #13 (3442356655) in inode 71173665. CLEARED.
Illegal block #15 (4226068168) in inode 71173665. CLEARED.
Illegal block #16 (3857826628) in inode 71173665. CLEARED.
Illegal block #17 (2248859270) in inode 71173665. CLEARED.
Illegal block #19 (2059144263) in inode 71173665. CLEARED.
Illegal block #24 (3890096442) in inode 71173665. CLEARED.
Too many illegal blocks in inode 71173665.
Clear inode<y>? yes
And then passes 2-4:
Inode 71173677 (/path/to/dir/@eaDir) is an illegal character device.
Clear<y>? yes
Entry 'filename.pdf' in /path/to/file/ (169476145) has deleted/unused inode 71173665. Clear<y>? yes
Inode 71173667 (/path/to/file/@eaDir/filename1.ext@SynoEAStream) has invalid mode (071141).
Clear<y>? yes
...
Pass 3: Checking directory connectivity
/lost+found not found. Create<y>? yes
Pass 4: Checking reference counts
Inode 1442557 ref count is 311, should be 310. Fix<y>? yes
Inode 2621587 ref count is 10, should be 9. Fix<y>? yes
Inode 71173669 (...) is an illegal character device.
Clear<y>? yes
Inode 71173670 (...) is an illegal character device.
Clear<y>? yes
Inode 71173671 (...) has invalid mode (030060).
Clear<y>? yes
Inode 71173672 (...) is an illegal block device.
Clear<y>? yes
Inode 71173673 (...) is an illegal character device.
Clear<y>? yes
Inode 71173674 (...) is an illegal block device.
Clear<y>? yes
Inode 71173680 (...) has invalid mode (030060).
Clear<y>? yes
Unattached inode 71173827
Connect to /lost+found<y>? yes
Inode 71173827 ref count is 2, should be 1. Fix<y>? yes
This one from pass 2 was interesting:
Entry 'filename.pdf' in /path/to/file/ (169476145) has deleted/unused inode 71173665. Clear<y>? yes
71173665 is the inode that had too many illegal blocks in pass 1, but the file is still intact. I thought for sure I would have lost that one.
Pass 5 was completely over my head:
Pass 5: Checking group summary information
Block bitmap differences: -(14714855--14714856) -284720047 -284720052 -284720054 -284720124 -(284727276--284727278) -284727283 -(284727288--284727289) -(284841277--284841457) -725166174
Fix<y>? yes
Free blocks count wrong for group #449 (32697, counted=32699).
Fix<y>? yes
Free blocks count wrong for group #8688 (43, counted=47).
Fix<y>? yes
Free blocks count wrong for group #8689 (34, counted=40).
Fix<y>? yes
Free blocks count wrong for group #8692 (1328, counted=1509).
Fix<y>? yes
Free blocks count wrong (182905187, counted=182905380).
Fix<y>? yes
Inode bitmap differences: -71173665
Fix<y>? yes
Free inodes count wrong for group #8688 (5902, counted=5897).
Fix<y>? yes
Directories count wrong for group #8688 (487, counted=485).
Fix<y>? yes
Free inodes count wrong (365283708, counted=365283703).
Fix<y>? yes
If anybody can help me better understand what's happened here I'd appreciate it. Even the files that were specifically mentioned in the fsck output appear to be intact after checking, so did I dodge a bullet here? (And yes, I'll be more diligent from now on about backing up at least the most important parts of the data.)
