7

On a Linux system, I have one 7MB chunk of memory of fixed size (no growth) whose contents I refresh in a real-time application.

I need to write this chunk of memory to disk (same file) once per second.

Taking into consideration modern (late 2011) CPUs and HDDs, what is the most efficient way to implement this functionality? I don't care if the write actually takes some time, but as this is a real-time app, I need to return to the running app ASAP.

What methodologies should I be trying?

My baseline is a standard baseline fopen(), binary fwrite(), fclose() cycle.

I have read that mmap() might be useful. Maybe asynchronous I/O? Are there other methodologies that I should be benchmarking? Off the top of your head, which methodology do you think would be fastest?

Improve this question
4
  • 1
    If you don't care how long it takes to do the actual write, then the modern approach is to just hand it off to a thread to do.
    – Nicol Bolas
    Commented Oct 12, 2011 at 7:18
  • Async IO could be the fastest
    – David Feurle
    Commented Oct 12, 2011 at 7:22
  • 1
    ty Nicol - a writer thread seems to be the easiest to implement given the code I have in place. I will try this out against mmap.
    – kfmfe04
    Commented Oct 12, 2011 at 8:08
  • 1
    A good answer depends on whether you want the state of the file to be consistent, can accept to sometimes miss the write, or are on a realtime OS. e.g. when you need consistent images, copy the 7MB into a secondary buffer and hand the writing of to another thread. that way you wont get interrupted in your normal flow. only do the copy/handoff when there is no write to disk running.
    – PlasmaHH
    Commented Oct 12, 2011 at 8:39

2 Answers 2

Reset to default
9

mmap(2) is the way to go. Just call msync(2) with MS_ASYNC when you want to write it.

Improve this answer
8
  • 2
    Can you cite a source for this? Why is mmap the fastest?
    – spraff
    Commented Oct 12, 2011 at 7:32
  • 1
    @spraff: It directly frobs the memory in the cache, instead of spending time copying chunks of memory as per fread(3)/fwrite(3).
    – Ignacio Vazquez-Abrams
    Commented Oct 12, 2011 at 7:34
  • Ignacio, ty for your suggestion - I am trying out the example in Stevens atm. BTW, my chunk happens to have a complex data structure, but it really is one chunk (eg no std::string). Using standard binary I/O, there is a lot of coding for individual PODs. I suspect that I don't need any of this for mmap - is this correct?
    – kfmfe04
    Commented Oct 12, 2011 at 7:45
  • @IgnacioVazquez-Abrams IIUC, his problem isn't the actual type to write the data, but the time the write will block his application. In this case, msync is best called from a separate thread.
    – James Kanze
    Commented Oct 12, 2011 at 8:10
  • 2
    hahaha - I had to look up "frob" - which redirected me to frobnicate (v.v.nice): [Poss. derived from frobnitz, and usually abbreviated to frob, but frobnicate is recognized as the official full form.:] To manipulate or adjust, to tweak. One frequently frobs bits or other 2-state devices. Thus: “Please frob the light switch” (that is, flip it), but also “Stop frobbing that clasp; you'll break it”. One also sees the construction to frob a frob. See tweak and twiddle.
    – kfmfe04
    Commented Oct 12, 2011 at 8:44
1

I'd combine the two approaches mentionned: I'd use mmap to map the memory to the file, then set up a separate thread (with lower priority) to msync it every second. (In this case, the actual arguments to msync are not too important; you don't need MS_ASYNC, since you won't be blocking the main thread.)

Another alternative possibly worth trying would be asynchronous IO. It's not clear to me from my documentation what happens if you never recover the results, however, so you may need some sort of reaper code to prevent lost resources. (Asynchronous IO seems rather underspecified in Posix, which IMHO is a good reason to avoid it.)

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.