I have two programs that pass data to each other via linux pipes (named or otherwise). I need to hit a transfer rate of ~2600 MB/s between the two programs, but am currently seeing a slower rate of about ~2200 MB/s. However, I found that if I replace my 2nd process with 'dd' instead, the transfer rate jumps to over 3000 MB/s. Is there something about the way my program is reading from the pipe that is less efficient than the way 'dd' does it? What can I do to improve this throughput? Is 'ifstream' inherently slower than other methods of reading binary data from pipe?
To summarize the two scenarios:
Scenario 1:
Program 1 -> [named pipe] -> Program 2
Yields ~2200 MB/s transfer rate
Scenario2:
Program 1 -> [named pipe] -> 'dd if=pipename of=/dev/null bs=8M'
Yields ~3000 MB/s transfer rate.
Here is the way my Program 2 currently reads from pipe:
ifstream inputFile;
inputFile.open(inputFileName.c_str(), ios::in | ios::binary);
while (keepLooping)
{
inputFile.read(&buffer[0], 8*1024*1024);
bytesRead = inputFile.gcount();
//Do something with data
}
Update:
I have now tried using 'read(fd, &buffer[0], 8*1024*1024)' instead of istream, seemed to show a mild improvement (but not as much as dd)
I also tried using stream->rdbuf()->sgetn(&buffer[0], 8*1024*1024) instead of stream->read(), which did not help.
dd
's source code: lingrok.org/xref/coreutils/src/dd.cfstream
s have some overhead dealing with locales, butdd
appears to be usingread
which has none of the associated locale-checking. Even inios::binary
mode, you still pay some of that penalty. How does your perf change if you use aFILE*
instead? It's not as C++, but if perf is your concern...Release
mode, but if not - there could be worth results inDebug