Python subprocess interaction blocked by stdout.readline()

Question

I am trying to use the subprocess module to launch another instances of Pyhton, send commands and retrieve the output. However no matter what I try, it always hangs at stdout.readline(). Here's my current attempt:

from subprocess import Popen, PIPE, STDOUT
script = "import sys;sys.stdout.write('glorp')\n"
p = Popen([r"C:\Program Files\Python37\python.exe",], stdin=PIPE, stdout=PIPE, text=True)
p.stdin.write(script)
print("readline")
print(p.stdout.readline())
print("end")

The code never gets past the line involving readline(). I've tried using flush() on stdin and stdout.

Answer 1

The main problem here is that the Python interpreter is not in interactive mode when we start it as a subprocess. As the Python documentation notes:

When called with standard input connected to a tty device, it prompts for commands and executes them

In this case, the subprocess's stdin is not connected to a terminal ("tty device"). And

In non-interactive mode, the entire input is parsed before it is executed.

So Python just sits there and does nothing until stdin is closed. But we can force it into interactive mode with the -i command-line option. Then:

When a script is passed as first argument or the -c option is used, enter interactive mode after executing the script or the command, even when sys.stdin does not appear to be a terminal.

It's also useful to pass the -q option

Don’t display the copyright and version messages even in interactive mode.

and communicate with the subprocess in unbuffered mode by setting bufsize=0. Then we are able to have the interpreter run multiple commands and we can inspect the output in between:

from subprocess import Popen, PIPE
p = Popen(['python', '-i', '-q'], stdin=PIPE, stdout=PIPE, text=True, bufsize=0)
p.stdin.write('print("First line executed.")\n')
print(p.stdout.readline(), end='')
p.stdin.write('print("Second line executed.")\n')
print(p.stdout.readline(), end='')

Note that the prompts (>>>) may come in out of order in the console:

>>> >>> First line executed.
Second line executed.
>>>

We could hide the chevrons by adding '-c', 'import sys; sys.ps1=""' to the argument list when starting the interpreter.

It is also recommended to explicitly set the encoding as UTF-8 on both ends. A more thorough demonstration can be found in this issue on the Python bug tracker.