1

I have an application that generates a stream of data as a single file. The amount of generated data is variable, but in average wouldn't nearly saturate my network or NAS speeds. However, there can be peaks for a few seconds that totally go above what's possible.

I don't have control over the application's behavior, which is just dropping data if the transfer can't keep up. If I generate the data stream onto my local SSD or even HDD, which is way slower than my NAS, everything is fine. I guess the latter one results from Windows' behavior of internally caching write operations in RAM. Is there some way to get this enabled for write operations on my NAS, too? Or are there other solutions available that would be able to solve this problem?

Also the whole data stream wouldn't fit on my SSD or even HDD, so just dumping it there and copying or syncing it later/asynchronous isn't an option for me. Ideally, if at all, I'd just want the non-synced portion of the file to reside somewhere on my local machine.

I am using Windows 10 Pro on my local machine. The NAS is a DS220+ and mounted via SMB.

Improve this question
2
  • Probably a long shot, but I think your description might implicate buffering behavior, and there is a way to bypass one (some?) using (on the client) net use /writethrough or New-SMBMapping -UseWriteThrough. Might look into that.
    – Yorik
    Commented Jul 20, 2021 at 18:52
  • Enabling writethrough doesn't help, if not even worsens the problem. I already tried that and probably should have included it in the question.
    – Num Lock
    Commented Jul 22, 2021 at 6:13

1 Answer 1

Reset to default
0

Windows uses cache memory for all file writes, which also pertains to files residing on network shares. Cache memory is important - if it is not large enough for containing the data generated during these few seconds of peak activity, the best solution would be to increase the amount of RAM in the computer until it can contain it all, catching up with writing it out when activity calms down.

You could also increase the size of the pagefile, so the RAM could swap out more. Insure that the pagefile is on the SSD rather than the HDD.

You could try to improve the network speed with the aim of helping Windows to empty the RAM faster toward the NAS, therefore postponing the time that the RAM will fill up.

The following operations might help (or not). You would need to try them and see. A good test might be measuring the speed of a large file-copy to the NAS (larger than the RAM).

  • Invest in a faster network adapter, if the NAS supports it

  • Configure in Windows and in the NAS the network adapters jumbo frames, receive buffers, and interrupt moderation rate.
    On Windows, this is done in Device Management, right-click the network adapter, Properties, Advanced tab. The available settings depend on the adapter and the driver.
    Receive buffers might be more important on the NAS side, and this is done via modprobe or ethtool, depending on the adapter.

  • Turn off the Windows firewall and disable the anti-virus.

  • Use NFS for better performance. See the article
    Network share: Performance differences between NFS & SMB.

You may find more details in the articles:

Improve this answer
9
  • I have 64GB of RAM on my workstation (around ~8GB only are utilized by other things) and am connected via 1GBit to my NAS. I already tried jumbo frames. The stream of data averages at a laughable 50MB/s with peaks around 200MB/s or more, but only for a second or two. My NAS can take data at a steady rate of 115MB/s. My workstation's HDD can only take data at 85MB/s but has no problems with the data stream because of Windows' write buffer. If I turn it off in the the drive policies I get the same problems as with the NAS. So how can I enable write buffers on my local machine for SMB shares?
    – Num Lock
    Commented Jul 22, 2021 at 6:27
  • Turning off the cache will slow everything down. But if you insist, you will find the info in Controlling write-through behaviors in SMB.
    – harrymc
    Commented Jul 22, 2021 at 9:25
  • If anything I want to increase the buffer, if that's even possible. Or even better: Have something that is equivalent to the local write cache in RAM on local drives. Have you even read my question/comment? How am I insisting? And on what anyway?
    – Num Lock
    Commented Jul 22, 2021 at 10:06
  • Your insistence is not always clear. Anyway, you don't need to increase the cache buffer - Windows will use as much of the RAM as it needs. As I wrote above, to increase the cache buffer you need to increase the RAM. Increasing the size of the pagefile might also help somewhat. But perhaps if you disable write cache buffer flushing the performance might improve.
    – harrymc
    Commented Jul 22, 2021 at 12:36
  • Turning that off (or rather on) on network drives is not possible. The closest that can be done is disabling write through, which just makes matters worse.
    – Num Lock
    Commented Jul 22, 2021 at 12:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .