I have a directory with
42048 File(s) 32,442,230,574 bytes
Running defrag tells me everything is fine
defrag E: /A /V
Microsoft Drive Optimizer
Copyright (c) Microsoft Corp.
Invoking analysis on New Volume (E:)...
The operation completed successfully.
Post Defragmentation Report:
Volume Information:
Volume size = 931.50 GB
Cluster size = 4 KB
Used space = 814.77 GB
Free space = 116.73 GB
Fragmentation:
Total fragmented space = 1%
Average fragments per file = 1.00
Movable files and folders = 516620
Unmovable files and folders = 4
Files:
Fragmented files = 160
Total file fragments = 2273
Folders:
Total folders = 25503
Fragmented folders = 2
Total folder fragments = 8
Free space:
Free space count = 78165
Average free space size = 1.52 MB
Largest free space size = 2.97 GB
Master File Table (MFT):
MFT size = 2.50 GB
MFT record count = 2631423
MFT usage = 100%
Total MFT fragments = 14
Note: File fragments larger than 64MB are not included in the fragmentation statistics.
You do not need to defragment this volume.
So I should not have any fragmentation issues. But when I try to delete some files in that directory I see in WPA that the deletion of one file needs 600ms and after defrag did run over night it need 2400ms for a single file!
Since Defrag says everything is ok what is so fragmented here? I have tried contig.exe to check $MFT which has 7 fragments but that does not look too bad:
E:\> contig -a $MFT
Contig v1.8 - Contig
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals
E:\$Mft is in 13 fragments
E:\$Mft::$BITMAP is defragmented
Summary:
Number of files processed: 2
Number unsuccessfully procesed: 0
Average fragmentation : 7 frags/file
The slow methods seems to be NtfsInsertCachedLcnAtIndex which is called by NtfsScanEntireBitmap. I suspect that the NTFS bitmap is degraded to a very long list. The problem is that I cannot access $BITMAP with contig.exe. It says Access is denied. Has anyone had similar issues and how were they solved except to format the hard drive? Would changing the NTFS cluster size from 4 KB to 32 KB make things better or would this make no difference for a volume where many 1-2 MB files are created and deleted all the time?
Update 1 Added Smart Data
Update 2 Added File Deletion Graph over n File Deletions - No Clear Trend Visible
Update 3 Creating 100 500K files in an empty folder with contig.exe and deleting then file by file shows this pattern
for( $i=0;$i -le 100;$i++){ contig -n "$i.json" 500000 }
Get-ChildItem *.json | foreach { (Measure-Command { Remove-Item $_ } ).TotalMilliSeconds }
5.2938
2.7115
1.2143
1.297
1.1552
1.3336
1.8351
1.1386
2230.3417
0.862
0.7522
0.7832
0.7061
0.711
0.7167
0.713
0.7137
2627.1645
Contig creates unfragmented files so it cannot be file fragmentation. But for some files the deletion is very inefficient. This more looks like an NTFS file system issue. I will check with MS.
Update 4 This is an actual limit inside NTFS. I have written some tests to create e.g. several hundred K files in a folder to check NTFS file creation query performance. That works well and despite the rumors it works up to several hundred K files with no big degradation even if the file names are nearly identical which I did suspect could cause some tree balancing issues. After doing a sustained write performance test on another machine with a Samsung SSD I was hit by the file deletion issue again. This time it takes 2-6s to delete a single file via a del . from cmd.exe in a directory with 20K files inside them.
Some NTFS tables are degraded to huge linked lists which should have O(1) lookup time but not in this case anymore. I wonder how many people were hit by this and believed some expert answer that you do not need to defrag your drive on SSDs... You do not need to move the clusters around but you still need to defrag NTFS tables which certainly need some cleanup as my Samsung SSD 860 EVO 1TB shows. The original HDD issue did get better from day to day after defrag did run several days. I need to "defrag" some things inside NTFS but I have no idea what. It should not be the volume bitmap as this will influence only seek times which are fast with a SSD.