NTFS support on Linux, dealing with large sparse files, impact on performance

Question

Background :

I attempted to recover data from a failing HDD using ddrescue on a recent Knoppix environment. At first I stored the output to a NTFS partition, but at some point the copying rate became consistently very slow (~600KB/s), and I read on some french guide for that tool that recovering to NTFS was not recommanded, especially for a large volume, as it had been reported to result in a major slowdown. I then switched to ext4 and the performance seemed to improve significantly. (See this question.)

I noticed that the “size on disk” for the output image files generated by ddrescue was much lower than their actual size, meaning that ddrescue somehow only allocates the data which has been successfully read from the input, resulting in a “partially sparse” output, even when the -S switch (“sparse writes”) is not used. In this case, using the -S switch results in the empty sectors which are actually read from the input, being also unallocated in the output, thus making it “fully” sparse. (See that other question.)

Question :

How could this performance gap be explained between NTFS and ext4 ? Is it indeed related to sparseness ? Is the Linux NTFS driver known to have trouble dealing with large sparse files ?

Answer 1

To start with, you were probably using NTFS-3G, which is implemented in userspace, not in the kernel. That alone means you had at least double the context switches when using NTFS that you did when using ext4.

Beyond that, I can't really comment much on what the issue may be, other than mentioning that based on my own testing, NTFS does not appear to handle sparse files very efficiently on LInux, either using NTFS-3G, or using the kernel's native driver (which almost nobody uses because NTFS-3G has much better overall support).