Getting output of a process at runtime

Question

I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:

result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in result.stdout.readlines(): #read and store result in log file
 openfile.write("%s\n" %line)
 print("%s" %line)

Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.

But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).

So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.

• Adrian Ratnapala: I've never actually tried it, but I think you are supposed to send your own Python file object to the stdout (or stdin, or stderr) argument. Then you have to poll that file. Subprocess was invented to spare you that pain, but it looks like you have no choice. Good luck.
• Eric: "it first completes the process and stores the output in result" - that's just not true.
• Niyojan: "it first completes the process and stores the output in result" - that's just not true. Well, when i am running Popen command, it keeps on running until the process finishes and then it executes further coding. If its work some other way, i am eager to know that.
• glglgl: @AdrianRatnapala Sounds too complicated, and the "no choice" claim is wrong.
• EmilioPeJu: you may consider using fcntl to set result.stdout as non blocking, this will allow to read in real time and if there is no data yet, IOError will be raised

Answer 1

The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:

for line in result.stdout:
 print(line)

• jfs: note: for line in result.stdout does not return lines in "real time" on Python 2, use for line in iter(result.stdout.readline, b'') instead (your print line statement suggests that you expect it to work on Python 2. On Python 3 it works fine
• Niyojan: Yes, its working, i am getting output at real time, Thank you.

Answer 2

.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":

import sys
from subprocess import Popen, PIPE
proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE)
for line in proc.stdout:
 openfile.write(line)
 sys.stdout.buffer.write(line)
 sys.stdout.buffer.flush()
proc.stdout.close()
proc.wait()

Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.

Note: on Python 2, you might also need to use iter(), to get "real time" output:

for line in iter(proc.stdout.readline, ""):
 openfile.write(line)
 print line,

Answer 3

You can iterate over the lines one by one by using readline on the pipe:

while True: 
 line = result.stdout.readline()
 print line.strip()
 if not line:
 break

The lines contain a trailing \n which I stripped for printing. When the process terminates, readline returns an empty string, so you know when to stop.

• Niyojan: I am sorry but you didn't get my question. What you have suggested i am already accomplishing it using for loop, what i want is line by line output while process is still in running state.
• Thomas Fenzl: did you try? You're using readlines, which returns all lines. That's only possible after termination. With readline, you get only the next
• Niyojan: result.stdout works just fine for me, but your line.strip() functions is helpful as i was getting too many "b'\r\n'" before it, Thank you