Wrong dd command on main drive - How to recover data?

Question

Okay, something annoyingly stupid happened. I wanted to copy an Arch Linux ISO file to my USB thumb drive, but was in a rush, and accidentally entered my main drive as the of parameter.

Here are the details:

$ sudo dd bs=4MB if=archlinux-2017.08.01-x86_64.iso of=/dev/nvme1n1

/dev/nvme1n1 should have been /dev/sdb.

My main drive /dev/nvme1n1 contained two partitions:

• One 512 MB EFI boot partition
• One ext4 partition spanning the rest of the 1 TB drive

The file size of archlinux-2017.08.01-x86_64.iso is 541065216 bytes, or 516 MB

The computer is still running and appears to be working fine, and I have the output of lsblk and df -h before running the dd command. The output is exactly the same as when I run the commands now. I assume because the data is cached:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme1n1     259:5    0 931.5G  0 disk 
├─nvme1n1p1 259:6    0   512M  0 part /boot
└─nvme1n1p2 259:7    0   931G  0 part /

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p2  916G   22G  848G   3% /
/dev/nvme1n1p1  511M   36M  476M   7% /boot

ls /boot still prints the directory content (probably cached info), but the file content is damaged, and running ls /boot/EFI, or ls /boot/loader fills the screen with random characters, including lots of Input/output error.

Here is some more info:

$ cat /proc/partitions
major minor  #blocks  name

 259        5  976762584 nvme1n1
 259        6     524288 nvme1n1p1
 259        7  976237255 nvme1n1p2

$ sudo fdisk -l /dev/nvme1n1
Disk /dev/nvme1n1: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x282bad86

Device         Boot Start     End Sectors  Size Id Type
/dev/nvme1n1p1 *        0 1056767 1056768  516M  0 Empty
/dev/nvme1n1p2        164  131235  131072   64M ef EFI (FAT-12/16/32)

Looking at the output of fdisk, it's pretty clear that the partition table (and probably all data on the boot partition) was destroyed. It should be a gpt disklabel type, and the partition sizes/types are wrong. Unfortunately, because of the ISO file size (516 MB), it also overwrote the first 4 MB of my root partition.

A slightly different output from gdisk:

$ sudo gdisk /dev/nvme1n1

# selected GPT when asked "Found valid MBR and GPT. Which do you want to use?"

Command (? for help): p
Disk /dev/nvme1n1: 1953525168 sectors, 931.5 GiB
Model: Samsung SSD 960 EVO 1TB                 
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): <guid>
Partition table holds up to 248 entries
Main partition table begins at sector 2 and ends at sector 63
First usable sector is 64, last usable sector is 1056704
Partitions will be aligned on 8-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   2             164          131235   64.0 MiB    0700  ISOHybrid1

A couple of related questions I found:

• https://askubuntu.com/questions/94421/is-there-a-way-to-recover-files-from-a-storage-device-partially-overwritten-with
• Accidentally overwrote wrong disk with dd, how to recover?

I have already installed the testdisk utility, which looks promising, but I want to make sure that I perform the correct steps while the computer is still running. If I shut it down now, it won't boot up anymore, so here are the questions:

• What is the best way to recover from this situation?
• How do I restore the partition table to the previous form, and how do I recreate the /boot partition? I am running Arch Linux with the latest kernel.
• Is there any way of knowing what was contained (and destroyed?) in the first 4 MB of my root partition?

---

EDIT: Adding more information and details here based on @WumpusQ.Wumbley's suggestion to run the dumpe2fs command.

The basic output (first 50 lines) of dumpe2fs: https://pastebin.com/fBuFRQfE

To me it looks pretty normal, even the filesystem magic number (0xEF53) is correct.

This is followed by Group 0:

Group 0: (Blocks 0-32767) csum 0x9569 [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-117
  Reserved GDT blocks at 118-1141
  Block bitmap at 1142 (+1142)
  Inode bitmap at 1158 (+1158)
  Inode table at 1174-1685 (+1174)
  21349 free blocks, 8177 free inodes, 2 directories, 8177 unused inodes
  Free blocks: 11419-32767
  Free inodes: 16-8192

Which is then followed by LOTS of groups that say [...]8192 free inodes, 0 directories, 8192 unused inodes [...] The first group that actually reports some directories is not until Group 3648, or around 25,000 lines later:

Group 3648: (Blocks 119537664-119570431) csum 0xa9ea [ITABLE_ZEROED]
  Block bitmap at 119537664 (+0)
  Inode bitmap at 119537680 (+16)
  Inode table at 119537696-119538207 (+32)
  23930 free blocks, 1670 free inodes, 614 directories, 1670 unused inodes
  Free blocks: 119546502-119570431
  Free inodes: 29890939-29892608

There are a lot of backup superblocks throughout the filesystem:

$ sudo dumpe2fs /dev/nvme1n1p2 | grep -i superblock | wc -l
dumpe2fs 1.43.5 (04-Aug-2017)
19

Answer 1

I assume the partition table and boot partition can be recreated easily, so I will focus on the ext4 partition.

The layout of the filesystem is somewhat dependent on the options used when creating it. I'll describe the common case. You can see if this matches yours by running dumpe2fs on the device (which will hopefully find all of the top-level metadata in cache rather than reading from disk).

The normal block size for ext4 filesystems is 4096 bytes, so you have lost 1024 blocks.

The first thing overwritten was block 0, the primary superblock. This is not a problem by itself, because there are backup superblocks. After that is the group descriptor table, which also has backups within the filesystem.

Then there are block bitmaps and inode bitmaps. This is where the news starts to get slightly worse. If any of these are below block 1024, which they probably are, you've lost information about which inodes and blocks are in use. This information is redundant, and will be reconstructed by fsck based on what it finds traversing all the directories and inodes, if those are intact.

But the next thing is the inode table, and here you've probably lost a lot of inodes, including the root directory, journal, and other special inodes. It will be nice to have those back. Obviously the root directory at least is still functional, or just about all commands you try to run would be failing already.

If you run a dd if=/dev/nvme1n1p2 of=/some/external/device bs=4096 count=1024 now, you'll get a backup copy of whatever is in your cache currently, mixed with the bad data for the blocks that aren't cached. Then after booting a rescue disk you can do the same dd in reverse, to put that partially-good data back on the disk, overwriting the all-bad stuff that's there now.

After this you might find automated recovery tools (fsck, testdisk) work well enough. If not, you have a map you can use to help with manual recovery. Using the "free block" lists from dumpe2fs, you know which blocks to ignore.

Most of what you lost is probably inodes. It's actually fairly likely that you had no file contents in the first 4MB of disk. (I ran mkfs.ext4 with no options on a 1TB image file, and the first non-metdata block turned out to be block 9249)

Every inode you manage to recover will identify the data blocks of a whole file. And those data blocks might be located all over the disk, not necessarily nearby.

Day 2

The dump posted on pastebin reveals great news:

Group 0: (Blocks 0-32767) csum 0x9569 [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-117
  Reserved GDT blocks at 118-1141
  Block bitmap at 1142 (+1142)
  Inode bitmap at 1158 (+1158)
  Inode table at 1174-1685 (+1174)
  21349 free blocks, 8177 free inodes, 2 directories, 8177 unused inodes
  Free blocks: 11419-32767
  Free inodes: 16-8192

Since we think only 4MB at the start of the filesystem have been overwritten, we only need to worry about blocks 0-1023. And the reserved GDT blocks go all the way out to block 1141! This is the kind of damage that should be repaired by a simple e2fsck -b $backup_superblock_number (after a reboot). You could at least try that with -n to see what it thinks.

Answer 2

If the disk used GPT, the partition table should be recoverable by using the backup GPT data at the end of the disk. You can do this with gdisk; see the gdisk documentation on data recovery for details. In brief: When you launch gdisk on the disk, it will probably notice the damage and ask you if you want to use the backup GPT data or the MBR data. If you pick the GPT option and then write the changes, the partition table will be fixed. If gdisk does not ask about which partition table to use, you might still be able to load the backup table using the c option on the recovery & transformation menu.

If this fails, you could still re-create the partition table (or at least, the partitions' start and end points) by using the data in /sys/block/nvme1n1/nvme1n1p1/start and /sys/block/nvme1n1/nvme1n1p1/size files (and similarly for /dev/nvme1n1p2). If you resort to this data, though, it's imperative that you NOT shut down the computer, contrary to what hek2mgl advised. That said, hek2mgl is not wrong that continuing to use the disk in its current state runs the risk of making matters worse. Overall, I'd say the best compromise is to try to fix the partition table problem as quickly as possible and then shut down and fix the filesystem problem from an emergency disk.

Unfortunately, your ESP is toast. Given your disk layout, I'm guessing you mounted the ESP at /boot and stored your kernels there. Thus, you'll need to re-install your kernel packages using a chroot or some other means. Ditto for your boot loader or boot manager.

Answer 3

1. Shutdown the computer (immediately)
2. Boot it with a rescue system.
3. Run testdisk to try to recover your data. (If you have enough space, take an image from the device using dd and run testdisk on that image)

Why should you shutdown the computer immediately? If a new file will be created (something in /run probaly) or appended to (/var/log/...) then the file system needs to look at the existing (bad!) information to decide where to store the data. When this decision is been made based on bad information, the risk is high that existing data blocks will get overwritten. That makes them lost forever. Even for tools like testdisk and photorec.