Named pipes performance issues

Question

I'm using named pipes for inter-procedural communication between C# and Delphi. C# uses the System.IO.Pipes package, whereas Delphi makes use of Libby's pipes.pas. Unfortunately, the communication is all but high-performance: Profiling showed me that the communication takes 72% of the whole runtime, the rest is used by calculations.
I was able to locate one problem that could take up resources: If I don't explicitly disconnect the sending client's connection in Delphi, C# doesn't receive any data at all.

Delphi (sending)

FClient1.Write(msg[1], Length(msg));
FClient1.FlushPipeBuffers;
FClient1.WaitForReply(20);
FClient1.Disconnect;   // disconnect to signalize C# that the writing is finished
FClient1.Connect;      // connect again to prevent synchronization problems

C# (receiving)

// Wait for a client to connect
stc.pipeServer.WaitForConnection();
while (reconnect_attempts < MAX_RECONNECT_ATTEMPTS) // 
{
   string tmp = sr.ReadLine();

   // if result is empty, try again for <MAX_RECONNECT_ATTEMPTS> times
   // so you can eliminate the chance that there's just a single empty request
   while (tmp != null)// && result != tmp)
   {
      tmp = sr.ReadLine();
      result += tmp;
   }
   // sleep, increment reconnect, write debugging...
}
stc.pipeServer.Close();

Even though I guess that the reconnecting is expensive, I'm not entirely sure about it. One flow of data (roughly 1 / 11 kb) takes 130 (respectively 270ms for the 11kb) total (sending & receiving).

My question would be:
Is it necessary to force-disconnect the pipes to signalize that the client is done writing? As far as my observations go, this is only necessary when sending with libby's. Are there any other possible causes for the poor performance? Thanks in advance.

As an addition, here's the sending and receiving done vice versa:

C# (sending)

 stc.pipeClient.Connect();
 StreamWriter sw = new StreamWriter(stc.pipeClient);
 //sw.AutoFlush = true;
 sw.WriteLine(msg);
 sw.Flush();
 stc.pipeClient.WaitForPipeDrain();  // waits for the other end to read all bytes 
 // neither disconnect nor dispose

Delphi (receiving)

 SetLength(S, Stream.Size);   Stream.Read(S[1], Length(S));  
 FPipeBuffer := FPipeBuffer + S;   { TODO 2 : switch case ID }   
// if the XML is complete, i.e. ends with the closing checksum   
if (IsFullMessage()) then
begin
   // end reading, set flag
   FIsPipeReady := true;
end

Answer 1

After a lot of (manual) profiling, I came up with two insights about the problem:

1. Libby's pipes is a complex beast. Since it seems to use multiple threads and shows a weird behavior concerning its usage, the manual use of the WinApi was more convienient after all. Furthermore, the performance taken by the actual communication increased. In other words: In a relatively simple IPC-scenario like this, libby's pipes seem to be slower than the WinApi.
2. Annonymous pipes / using the stdout & stdin seem to be even faster than named pipes.

However, I must add that I still am sort of confused and can't tell whether this is true or I've been crunching the wrong numbers here.

Here's an easy example of how the WinApi implementation in Delphi could look like:

// setup pipes, you'll need one for each direction
// init handles with 0
    CreatePipe(ReadPipe1,       // hReadpipe
               WritePipe1,      // hWritePIpe
               @SecurityAttributes,        // Security
               PIPE_SIZE)                  // Size

    // setup Startupinfo
    FillChar(StartupInfo, Sizeof(StartupInfo), 0);
    StartupInfo.cb := Sizeof(StartupInfo);
    StartupInfo.dwFlags := STARTF_USESHOWWINDOW or STARTF_USESTDHANDLES;
    StartupInfo.hStdInput := ReadPipe1;
    StartupInfo.hStdOutput := WritePipe2;
    StartupInfo.wShowWindow :=  SW_HIDE; 

    // CreateProcess [...]

    // read
    Win32Check(
            ReadFile(
                  ReadPipe1,  // source
                  (@outputBuffer[1])^,               // buffer-pointer
                  PIPE_BUFFER_SIZE,                 // size
                  bytesRead,                       // returns bytes actually read
                  nil                             // overlapped on default
                  ));
    // send           
    Win32Check(
            WriteFile(
                WritePipe2,
                (@msg[1])^,         // lpBuffer - workarround to avoid type cast
                NumberOfBytesToWrite,
                bytesWritten,       // lpNumberOfBytesWritten
                nil                 // Overlapped   
                ));                          

Answer 2

Maybe you can use named events for IPC signaling. These work fine in Win7 etc when these are local (TEvent.Create('local\myserver'); When you need to do IPC between different sessions (e.g. client app and background windows service), you need more rights etc (default global\ can not be used in win7 due to UAC?). http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/devwin32/threadswaitingforatasktobecompleted_xml.html

For example: create an event per connection (with a generated name per connection).

Or take a look at a different IPC named pipe + events implementation : https://micksmix.wordpress.com/2011/06/27/named-pipes-unit-for-delphi/

Btw: you mentioned you used profiling but you could not say what takes the most time? What kind of profiling did you use? Not a "profiler" like AQtime (http://smartbear.com/products/free-tools/aqtime-standard) or AsmProfiler (http://code.google.com/p/asmprofiler/)?

Answer 3

And simple improvement could be: first send the amount of bytes to send (so receiver knows how much data it can expect) then send the data