How can Testdisk recover more data than the size of the partition which the data is on?

Question

This is going to be a long text because I'm not sure what parts of the back story are relevant - if you want to skip the boring parts, you can just read the two last paragraphs which have the actual question.

Long story short, I was careless with a partitioning utility and messed up my partitions, possibly corrupting some files since there was a partition resizing operation going on which moved files around when I had to terminate the utility process. The result was that the last partition (about 420 GB) on the disk disappeared, taking with it my machine's /home. My backups were, let's say, a bit outdated so I decided to try to recover as much as I could and clean it up later.

I fired up a live Linux distribution meant for data recovery and launched Testdisk to see if it could find the missing partition. On the first run I set it up with "Intel partitions", analysed the disk with the quick and deeper searches and came up with nothing. I figured that since I knew what the partition table looked like before the accident, I'd use GParted to recreate the lost partition (without any formatting) and try again. With Intel partitions, Testdisk still couldn't find anything at the end of the disk, so I tried with "No partitioning" starting option. This time the analysis found the partition I had recreated and in the file listing I could actually see my missing files (along with loads of files I had previously deleted as well). Win!

Now, I have learned my lesson about frequent backups and being careful with partitioning (I didn't even want to touch the hard drive that was messed up but accidents happen). Luckily it seems that Testdisk saved the day and is copying the lost files onto an external hard drive as I write this, but there's something about the copying that has left me confused.

As I mentioned previously, the partition I lost was about 420 GB in size. The hard drive which the partition resides is 1 TB in total. At the moment, Testdisk has been at it for about 15 hours (I'm copying the files over USB2.0, sloooow) and df tells me that there's over 900 GB of stuff on the external drive at the moment. When I examine the contents of the copy, there doesn't seem to be that much content there yet, although I may have missed some big files.

How is it possible for Testdisk to copy over 900 GB of data from a partition of size 420 GB? Is there some preallocation for copying deleted files which Testdisk can see but are not actually recoverable/there anymore?

Answer 1

The answer is testdisk/photorec often over grab. Sometimes the beginning marker, ending marker, or size of the file is corrupt, or it makes a bad guess and grabs more sectors than it needs for a file. Then another file has a pointer in this region, and it grabs overlapping sectors again.

So now file 1 contains all of file 1 plus random data. File 2 contains part of the random data. Frequently, these programs don't handle fragmented files well, and that too can result in a data over grab.

In an effort to make sure it recovers as much as possible it doesn't keep track of the sectors that are part of other files, just in case it gets it wrong. Therefore, the same sectors are duplicated in many files.

I have found files, like word docs, they say 2gb, but when opened and re-saved under a new name they return to there correct size.

Pretend each symbol is a cluster of storage.

G is a GIF file W is a word document U is unused

GGGGGWWWWUUUG

Most files have a header and a footer. So when the scans detects the gif header it then looks for the footer for the GIF. So now the recovered GIF file contains GGGGGGGWWWWUUUG because the file is fragmented. Then as it scans forward it detects the W or the word header and the footer so the word document comes out correctly as WWWW despite the earlier over grab.