Python talking with other application(s) using subprocess

Question

this is the idea. I'll have 'main' python script that will start (using subprocess) app1 and app2. 'main' script will send input to app1 and output result to app2 and vice versa (and main script will need to remember what was sent so I can't send pipe from app1 to app2).

This is main script.

import subprocess
import time
def main():
 prvi = subprocess.Popen(['python', 'random1.py'], stdin = subprocess.PIPE , stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
 while 1:
 prvi.stdin.write('131231\n')
 time.sleep(1) # maybe it needs to wait
 print "procitano", prvi.stdout.read()
if __name__ == '__main__':
 main()

And this is 'random1.py' file.

import random
def main():
 while 1:
 inp = raw_input()
 print inp, random.random()
if __name__ == '__main__':
 main()

First I've tried with only one subprocess just to see if it's working. And it's not. It only outputs 'procitano' and waits there. How can I read output from 'prvi' (without communicate(). When I use it, it exits my app and that's something that I don't want)?

Answer 1

Add prvi.stdin.flush() after prvi.stdin.write(...).

Explanation: To optimize communication between processes, the OS will buffer 4KB of data before it sends that whole buffer to the other process. If you send less data, you need to tell the OS "That's it. Send it now" -> flush()

[EDIT] The next problem is that prvi.stdout.read() will never return since the child doesn't exit.

You will need to develop a protocol between the processes, so each knows how many bytes of data to read when it gets something. A simple solution is to use a line based protocol (each "message" is terminated by a new line). To do that, replace read() with readline() and don't forget to append \n to everything you send + flush()

• Vizualni: Thank you sir for your answer but I've tried that before and I also tried it now to see if it's working. Unfortunately it's not.
• Aaron Digulla: Since you don't give me any information what's not working, I can't help you.
• Vizualni: I am not getting any output from that 'random1.py' script. It's same as before.
• Aaron Digulla: Are you use random1.py ever writes anything? Remove stderr = subprocess.STDOUT from your code and add sys.stderr.write() to print to the console from child processes. You may need to flush those writes, too.

Answer 2

main.py

import subprocess
import time
def main():
 prvi = subprocess.Popen(['python', 'random1.py'], stdin = subprocess.PIPE , stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
 prvi.stdin.write('131231\n')
 time.sleep(1) # maybe it needs to wait
 print "procitano", prvi.stdout.read()
if __name__ == '__main__':
 main()

random1.py

import random
def main():
 inp = raw_input()
 print inp, random.random()
 inp = raw_input()
if __name__ == '__main__':
 main()

I've tested with the above codes, then I've got the same problem as your codes. I think problem is timing. Here is my guess, When the main.py tries the code below

prvi.stdout.read() # i think this code may use the random1.py process

the code below grab the random1.py process

inp = raw_input()

To solve this problem, I think, as Aaron Digulla says, you need develope the protocol to make it.

Answer 3

• use -u flag to make random1.py output unbuffered
• use p.stdout.readline() instead of .read()

time.sleep is unnecessary due to .read blocks.