python 3.4 sub process

Question

I am trying to control subprocess and control the stdout and stdin of this process. It worked great with python 2.7 but when using python 3 it fails to run.

I wrote the following code in python 2.7:

print("python 2.7:\n")
proc = subprocess.Popen(['cat -'],
 shell = True,
 stdin = subprocess.PIPE, 
 stdout = subprocess.PIPE,
 #stderr = subprocess.PIPE,
 )
for i in range(10):
 proc.stdin.write('%d\n' % i)
 output = proc.stdout.readline()
 print (output.rstrip())
remainder = proc.communicate()[0]
remainder = remainder.replace("(","").replace(")","").replace("'","").split(",")[0]
print("-----------------")
print(remainder)

as I said, this code works fine, but when trying to write it in python 3.4 I fail to use the stdin.write properly.

print("python 3.4:\n")
proc = subprocess.Popen(['cat -'],
 shell = True,
 stdin = subprocess.PIPE, 
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE,
 universal_newlines = True
 )
for i in range(10):
 inp = str(i)
 proc.stdin.flush()
 proc.stdout.flush()
 proc.stdin.write(inp)
 output = proc.stdout.read()
 print (output.rstrip())
remainder = proc.communicate()[0]
remainder = remainder.replace("(","").replace(")","").replace("'","").split(",")[0]
print("-----------------")
print(remainder)

• tdelaney: In addition to the answer below, output = proc.stdout.read() will block reading the file until EOF which is likely when the process exits. The second round of your loop, if you get there at all, will not succeed.
• tdelaney: Your process may behave differently if its stdin is a pipe instead of a terminal. The pexpect module on unixy systems can come in handy in that case.
• jfs: @tdelaney: python 2.7 code works and therefore there is no need to use pexpect here. Though in general, pexpect may be useful.

Answer 1

You almost there. It is a buffering problem:

• bufsize=0 (unbuffered) on Python 2.7 by default
• bufsize=-1 (fully buffered) on Python 3.4 (the default has been changed)

You should put proc.stdin.flush() after proc.stdin.write() (before the .write() call, the buffer is empty -- there is nothing to flush):

#!/usr/bin/env python3
from subprocess import Popen, PIPE
with Popen(['cat'], stdin=PIPE, stdout=PIPE, bufsize=1,
 universal_newlines=True) as process:
 for i in range(10):
 print(i, file=process.stdin, flush=True)
 output = process.stdout.readline()
 print(output, end='')

It is a complete code example.

Note: in general, unless the you know exactly how much you need to read (like in the example: one line input -> one line output exactly); the input/output might become desynchronized and the deadlock might happen e.g., if the child produces less output than you expect then your process will block on .readline() while the child at the same time hangs awaiting for input from you. A more robust solution would work with child process' streams asynchronously (asyncio, threading, select).