Why is head -c (and similar commands) much slower for large files than small files?

Question

From other things I've read (for one example of many why does it take so long to read the top few lines of my file?_ )

I gather that commands like

head -c 4 bigfile
dd if=bigfile bs=4 count=1

or c++ functions like

std::ifstream file(filePath, std::ios::binary);
char buffer[4];
file.read(buffer, 4);
std::cout << buffer[0] << buffer[1] << buffer[2] << buffer[3] << std::endl;

Should be just as fast for huge files as they are for small files, since from what I gather none of these require loading the entire file they just read the first 4 bytes.

However when I test any of these commands with a 1TB file and a 0.3GB file, with the 0.3GB file they all finish instantly, while with the 1TB file they all take many minutes.

Why is this? Is there a way to actually read the first 4 bytes of a huge file just as quickly as the first 4 bytes of a small file?

My file system is EOS.

Answer 1

This is likely the consequence of the organization of files on the storage device. There have been many kinds of filesystems historically (on Unix-like systems alone, not to mention others).

Some filesystems use a B-tree structure or similar. For larger files, the tree gets deeper: more indirect accesses are required to get to the data.

Traditiona Unix filesystems avoided this by a lobsided tree approach. But even those traditional filesystems had extra fast access to very tiny files. Very tiny files (such as most symbolic links) could be stored entirely in the inode structure, not requiring additional data blocks.

The lobsided tree aspect comes into play in that the first bunch of data blocks are directly referenced from the inode (regardless of how large the file is). After the direct datablocks comes an indirect block, which contains pointers to more data blocks. Then there can be a "doubly indirect" block. Fast access to the front of the file is retained; accessing the long tail is expensive.

To understand things further on your system, you have to drill into what filesystem you are using, and how exactly it works.

• For each file (or other objects), an "inode" structure has to be allocated.

• A very tiny file's content can be stored entirely in the inode structure, in some spare space that exists. Symbolic links are a kind of file, and usually they are very small. Symbolic links fit entirely into the inode.

• A slightly larger file requires one or more data blocks in addition to the inode. Thus, it is slower to access. We must not only access the inode, but traverse pointers to data blocks and read them. The pointers to the data blocks are in the inode.

• An even larger file has indirect blocks. The inode points to indirect blocks