5

I've read in this answer https://superuser.com/a/1358858/1506333 that NTFS does double writes, uncompressed data first, then compressed, is it true?

Motive Nº1: It will kill SSD much faster, since it makes two writes; NTFS compression always writes uncompressed data prior to start compression on RAM and then re-write compressed data only if it is a gain of at least 4KiB.

also these

Improve this question

1 Answer 1

Reset to default
10

I was also interested in this, and couldn't find any source other than people making this claim. So time for some original research.

TL;DR

At least for Windows 10,

  • the claim "NTFS compression writes file data twice" is BUSTED
  • the claim "NTFS compression always leads to highly fragmented files" is BUSTED

Details of what I did:

I ran a fully updated Windows 10 in a KVM virtual machine and attached a new, empty, emulated 15GB SSD with disabled caching and instrumented KVM to measure the actual writes issued by windows to the hardware.

(SI units unless noted)

  • Result 0a: the raw sparse disk image started out using 2MB
  • Result 0b: partitioning/formatting the drive wrote 49MB
  • Result 0c: the disk image grew to 53M

Experiment 1: copy a 1GiB file consisting of binary zeroes from a samba share to the newly created disk (by dragging in explorer).

  • Result 1a: this physically wrote 3650kB to the disk
  • Result 1b: disk image did not grow
  • Result 1c: file propoerties show that the file takes up 0 bytes
  • Result 1d: contig shows file as fully defragmented

Experiment 2: delete the file, copy a 1GB file consisting of highly compressible data (0x790a repeated), using windows explorer.

  • Result 2a: this physically wrote 72MB to the disk
  • Result 2b: disk image did grow to 116MB
  • Result 2c: file properties show that the file takes up 63MB
  • Result 2d: contig shows file as using 2 fragments

Experiment 3: disable compression, copy the same file again to the disk.

  • Result 3a: this physically wrote 1003MB to the disk
  • Result 3b: disk image did grow to 1.5GB
  • Result 3c: file properties show that the file takes up 1000MB
  • Result 3d: contig shows file as using 1 fragment

Experiment 4: compress this file using explorer properties.

  • Result 4a: this physically wrote 75MB to the disk
  • Result 4b: file properties show that the file takes up 63MB
  • Result 4c: contig shows file as using 4 fragments
  • Result 4d: contig shows drive as having 4 free space fragments

Experiment 5: created a 5GB disk and partition, copy a 10GB highly compressible file to it, using "type" (explorer/copy immediately refuse as the destination drive is not large enough).

  • Result 5a: this aborted after writing 4600MB
  • Result 5b: this physically wrote 397MB to the disk
  • Result 5c: disk image did grow to 548MB
  • Result 5d: file properties show that the file takes up 280MB
  • Result 5e: contig shows file as using 57 fragments
  • Result 5f: contig shows drive as having 24 free space fragments

Overall observation 1: when writing the files, windows started writing relatively fast (5MB/s), but after the copy was "finished" it kept writing for more than 40s at ever decreasing speeds (500kB/s down to 54kB/s in the last few seconds). This is a bit bizarre, and might lead credence to the theory that something happens in the background after writing the file.

Overall observation 2: even simple commands such as "type" are shockingly slow.

Overall observation 3: windows apparently recognizes either chunks with zeroes when compression, or auto-detects runs of zeroes as sparse files (The source files were not sparse). This is surprising.

Cororally: in the (recent) past, I have found that files acquired millions of fragments with NTFS compression (mostly unity "WindowsNoEditor" game pack files), so NTFS does sometimes do this, but this seems to be limited to certain files and/or certain conditions, and is generally a rare occasion (I found 15 out of 1949979 files on my 3TB compressed disk to show this pattern).

Improve this answer
4
  • 1
    Not to discredit your observations, and neither those of the initial claims, but a reminder: There are (at least) two variants of NTFS compression. The old one that only works on 4KiB clusters, and always acts in blocks of 64K (and sees if there is a gain to be had if compressed), and the newer WoF compressio, that pretty much works by storing the compressed data as an alternate data stream (which is not accessible directly) and compresses the entire file transparently: devblogs.microsoft.com/oldnewthing/20190618-00/?p=102597
    – Ro-ee
    Commented Jul 20, 2022 at 21:42
  • 1
    Wof compression is not NTFS compression, which is also explicity stated in your source. Both initial claims and this question are about NTFS compression (and the claims would quite trivially be wrong w.r.t. Wof compression).
    – Remember Monica
    Commented Jul 21, 2022 at 3:44
  • 1
    Thank you for detailed explanation! Turns out the problem i was getting is not double writes: superuser.com/questions/1330063/… answers.microsoft.com/en-us/windows/forum/all/… superuser.com/questions/1137454/…
    – soredake
    Commented Nov 6, 2022 at 19:05
  • This is great research, thank you for conducting and publishing. This shows that NTFS compression is safe for SSDs. If anything, it prolongs the lifespan because you now need to physically write fewer bytes than the data contains.
    – Violet Giraffe
    Commented Feb 24, 2023 at 14:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .