Ext3 drive will not mount after power failure; how to recover files?

Question

After a recent power failure which caused my linux box (Ubuntu 8.10) to rapidly poweroff twice from a normal running state, I have a drive that will not mount.

UPDATE: The drive will sometimes mount, but show up as completely empty (not even Lost+Found) and show 14.9 GB free (it is a 500GB drive) When I try to do anything it gives me a permission error and the drive unmounts. (or, perhaps, was not really mounted in the first place?)

Here's the error message when I try to mount:

~$ sudo mount -a
mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

So maybe specify the fs type?

~$ sudo mount -t ext3 /dev/sdd1 /media/disk-7
mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

No, same. So something is messed up?

~$ sudo fsck /dev/sdd1
fsck 1.41.3 (12-Oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
/dev/sdd1: recovering journal
fsck.ext3: No such file or directory while trying to re-open /dev/sdd1
Warning... fsck.ext3 for device /dev/sdd1 exited with signal 11.

Googling for signal 11 wasn't encouraging, but I found a few other ways to try to repair the disk:

~$ sudo e2fsck /dev/sdd1
e2fsck 1.41.3 (12-Oct-2008)
/dev/sdd1: recovering journal
e2fsck: No such file or directory while trying to open /dev/sdd1

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 [device] 

Still hoping this failure has something to do with the power outage, I assume the superblock is corrupt or something, and try another: (I first determine that my block size is 32k using makefs -n)

~$ sudo e2fsck -b 32768 /dev/sdd1
e2fsck 1.41.3 (12-Oct-2008)
ext3 recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/sdd1: recovering journal
e2fsck: Journal must be at least 1024 blocks while recovering 
ext3 journal of /dev/sdd1

Per Avery Payne below I tried the following:

sudo mount -t ext2 -o ro /dev/sdd1 /media/disk-7

But got this error message:

mount: wrong fs type, bad option, bad superblock on /dev/sdd1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
~$ dmesg | tail
[261157.639721] EXT2-fs: sdd1: couldn't mount because of unsupported optional features (4).

And that's about where I'm stuck. I tried every backup superblock listed and get the same result. If it helps any, the "recovering journal" step takes a long time before it moves on to tell me it isn't working.

Honestly, I don't care much about getting back the state of the drive minutes before the crash, just about recovering the 400+ GB of other data that is on it. If anyone knows anything else I can try, ext3 data recovery utilities or techniques, etc, I would greatly appreciate it!

Answer 1

The problems you're having sound far more extensive than what I'd expect from mere loss of power (even during fairly heavy write activity) on a device. I have to wonder if you're really having more problems at the interface/driver level, or a corrupted partition table or something of that sort.

From the sounds of things you may have exacerbated the problem further with all the thrashing around you've done while trying to fix the issue.

I don't know if we can help with this case but don't give up yet.

For the future I'd suggest that you learn the following technique:

When you have trouble with a drive under Linux or UNIX you can usually use dd to make a bit-image copy of the whole device to some other location. Find a drive that's at least as large as the one in question and try a command like: dd if=$PROBLEMATIC of=$TARGET bs=4M ... be very careful about the if (input file) and of (output file) directives. Leave that run. It's a good idea to run tail -f /var/log/messages & (or possible variant as appropriate to your /etc/syslog.conf) ... either do that in the background or in another window. There are enhanced versions of dd which can handle retries and continuing past bad blocks more robustly (sdd is a name that comes to mind). But try just using the stock GNU dd command at first.

You can make such a copy of the whole device (/dev/sdd, for example) or just the partition (/dev/sdd1). If you get "short read or similar errors then it suggests that either the device has physical errors preventing reads past certain cylinders or, in the case of a partition, that the partition table is mangled in some way. You can even make two different dd images ... one of each.

Here's the trick: do all your fsck and mount attempts, and use your various other recovery tools such as TCT (The Coroner's Toolkit) on the copied image!

This minimizes the time spent running the drive (which is possibly degrading at the hardware level as you operate it) and minimizes the impact of failed and possibly misguided recovery attempts. (In some situations you make one image, then another based on that and always operate on the tertiary image ... depends on how much the data is worth).

I personally suggest that you run something like hexdump or strings to read through the image ... just let it scroll past for a long time and look for plain text that looks like it might be fragments of your data. I have used grep to recover useful (textual) data from otherwise completely mangled filesystems. In case I'm not suggesting it as data recovery heroics ... but as a sanity check. If you scroll through 10s of megabytes or a few gigabytes of data and don't see any recognizable text ... then you probably have a hopeless case or you've done something very wrong (were you really careful about those if= and of= options?).

I don't know if any of this will help you with the current effort. But learn these tricks now and they will definitely make your next foray into data recovery much less scary. (Yes, practice on a healthy system once or twice --- go use a hex editor and try adding your own creative corruption here and there --- to the COPY of course! Then try fix it).

Oh, and this is a really good time to review your backup and data recovery plans and procedures (or provide better advice to your customer/colleague/client/friend/whatever).

Answer 2

I don't have much to offer, except to hope you learn from my mistakes. STOP! Make a duplicate of the disk. dd can help you do this, and there are plenty of drive imaging applications which will give you a nice user interface. I made the mistake of digging down into fsck, looking for backup superblocks, etc. etc. without first duplicating the disk. I lost absolutely everything. Looks like you already opted to go without backups, but it's not too late to snag a copy of the drive before you destroy it further with well-intentioned restore attempts.

I did stumble across this article on recovering in a similar circumstance. Looks like you can tell mount to use a different superblock. Combined with -t ext3, it may offer some small hope.

Answer 3

Backtrack is a linux distro that comes packaged with a host of tools for things like this. It is also, however, aimed squarely at the upper echelon of power users. Their website may have some good tutorials for using their tools, so that may be of use to you.

Answer 4

If you're using LVM2 on your drive, you'll probably need to run that before attempting to repair your volume. Try this first to see if there are volumes present.

sudo pvscan && sudo vgscan && sudo lvscan

Any volumes you find will be the device to mount, instead of a direct reference like /dev/sdd1.

If you're not using LVM on the drive, you can always attempt to re-mount the drive as Ext*2* instead of Ext*3*, as it is backwards-compatible. While this opens the window for minor filesystem corruption (because the journal isn't replayed) it does allow you access to the remainder of your data, as you have already indicated that your filesystem "clean" bit is already set. When you go to remount the volume, you will need to specify the filesystem type directly, i.e.:

sudo mount -t ext2 /dev/sdd1 /media/disk-7 && ls /media/disk-7

From there, you can recover your data. If after you recover your data you are unable to get the existing Ext3 filesystem back to a known good state, I would back up the entire dataset, reformat the filesystem, and restore. Of course, this is not always an option - but at least you'll get your data back.

Follow-up:

A quick google turns up a result that implies that you might have a kernel versioning issue. Were you upgrading your kernel, or did you compile a custom image? Just curious.

Answer 5

Are you sure that /dev/sdd1 is really the drive you are looking for and not some other drive? The device naming is depended on the order the drives where plugged in and might be different for example when you plug in a USB drive later on vs having it plugged in on boot. Use /dev/disk/by-id/ to be sure that you touch the right disk.

Next step would then be figuring out if its the whole disk or just the partition is toast, to do that launch:

cfdisk /dev/sdd

Or another partition tool of your choice to see if partitions are still in the right place and as you expected them to be. If the partition table is toast you could try gpart to recover them or recreate them from scratch if you remember their layout.

Is dmesg outputting anything suspicious? On dieing harddrives you will most of the time get a plethora of error messages.

And as others have mentioned, copy the data before trying to modify the partition table or stuff.

Answer 6

If you can boot with livecd while booting use no swap and then you can mount that /dev/sda and copy all the data from there if you have USB HDD or over the network. Then you can reboot the system reformat and dump back data.