0

When using memory mapped files I'm getting in situations where Windows stalls since new memory is allocated and processed faster than it can be written to disk using memory mappedfiles.

The only solution I see is to throttle my processing while the MiMappedPageWriter and the KeBalanceSetManager are doing their jobs. I would be completely fine if the application is running slower instead of a complete OS freeze.

It already helped to use SetWorkingSetSizeEx using a hard limit, because the MiMappedPageWriter is starting earlier to page-out to disk, but still on some drives the data is allocated faster. For example an SSD with 250MB/s does not manage it, but with 500MB/s it is getting better. But I have to support a wide range of hardware and cannot rely on fast drives.

I found that there once was a performance counter, for example: Memory\Mapped File Bytes Written/sec, that I could use not monitor periodically (see: https://docs.microsoft.com/en-us/windows-server/management/windows-performance-monitor/memory-performance-counter-mapped-file-bytes-written-sec) but it seems that all the links have gone.

I have searched on many places, but couldn't find the performance counters for this.

Is there still a source for this or can you think of other possibilities to throttle the process while disk paging is active?

7
  • Windows shouldn't stall, just pause while cache memory is being liberated by writing out. I don't think that slowing the copy is a good solution. Try perhaps FastCopy which doesn't depend on cache memory.
    – harrymc
    Commented Mar 6, 2023 at 14:10
  • @harrymc: The problem is that windows is stalling since it tries to page out memory while I'm requesting new pages. Ultimately I'm in a condition where the modified page list is full and I get new pages as windows as paged old pages out. Then I'm on a hard memory limit. Thats the reason why I need to wait. I'm not sure about the copy you mean, I don't copy files, I'm using the MM-files as memory backed files since the data I produce is tremedously larger than my RAM.
    – msedi
    Commented Mar 6, 2023 at 15:39
  • So the data is not a copy? How is it produced? Are you basically using mapped files as virtual memory?
    – harrymc
    Commented Mar 6, 2023 at 15:44
  • Yes, I'm using the MM-files as virtual memory, they are used as a backing storage for a stack of large image data. Not all data is required all the time so its Ok if the data is paged out. But the stack of image data is in a pipeline-based approach, so each step in the pipeline produces data that can exceed the memory. When this happens the data is page out to the MM-file, but the pipeline still produces data. The problem gets better if the write speed of the disc is higher, but I canot assume a good disk.
    – msedi
    Commented Mar 6, 2023 at 15:56
  • Slowing your software in order to avoid flushing doesn't seem like a good approach. Perhaps using real virtual memory instead of mapped files will have more efficient algorithms. Question: Have you tried to pre-allocate your file?
    – harrymc
    Commented Mar 6, 2023 at 16:01

0

You must log in to answer this question.

Browse other questions tagged .