stdin seems much slower than stdout (python). Why?

Question

I have two python programs (one is a subprocess) that need to communicate with each other. Currently I am doing that through stdin and stdout. However, writing to the subprocess's stdin seems painfully slow.

a.py, a program that takes an arbitrary line of input and prints the time:

from time import time, sleep
from sys import stdout, stdin
while True:
    stdin.readline()
    stdout.write('%f\n' % time())
    stdout.flush()

b.py, a program that runs a.py and times how long it took to write to the program's stdin and read from it's stdout:

from time import time
from subprocess import PIPE, Popen
from threading import Thread
stdin_times = []
stdout_times = []
p = Popen(['python', 'a.py'], stdin=PIPE, stdout=PIPE)
for i in range(100000):
    t1 = time()
    p.stdin.write(b'\n')
    p.stdin.flush()
    t2 = float(p.stdout.readline().strip().decode())
    t3 = time()
    stdin_times.append(t2 - t1)
    stdout_times.append(t3 - t2)
p.kill()
print('stdin (min/ave):', min(stdin_times), sum(stdin_times) / len(stdin_times))
print('stdout (min/ave):', min(stdout_times), sum(stdout_times) / len(stdout_times))

Sample output:

stdin (min/ave): 1.69277191162e-05 0.000138891274929
stdout (min/ave): 1.78813934326e-05 2.09228754044e-05

I'm using Python 3.1.2 on Ubuntu 10.10.

Why is writing to a.py's stdin so much slower than reading from its stdout? Is there anyway I can get these two programs to communicate faster?

Answer 1

I'd see if you can reproduce this when disabling buffering on both input and output. I have a hunch that output is being (line) buffered by default (as it is in most languages: perl, .NET, C++ iostreams)

Answer 2

If I try this a few times, the variance of the numbers is very high.

Maybe you should set up a more sane test to benchmark stdin and stdout, without a lot of other overhead, so you don't measure other stuff that's going on on your cpu. Also you're measuring string operations and float conversion.