Printing output in realtime from subprocess

Question

I'm trying to print stdout in realtime for a subprocess but it looks like stdout is buffered even with bufsize=0 and I can't figure out how to make it work, I always have a delay.

The code I tried :

p = subprocess.Popen(cmd, 
 stdout=subprocess.PIPE, 
 stderr=subprocess.STDOUT, 
 bufsize=0)
line = p.stdout.readline()
while line:
 sys.stdout.write(line)
 sys.stdout.flush()
 # DO OTHER STUFF
 line = p.stdout.readline()

Also tried with for line in iter(p.stdout.readline, b'') instead of the while loop and with read(1) instead of readline(). Always the same result, the output gets delayed by a lot of seconds or minutes and multiple lines appear suddenly at once.

What I think happens :

bufsize is set to 0 ( it is set to 0 by default according to the docs ) so the lines piped top.stdout should be available immediately. But since p.stdout.readline() doesn't return immediately when a new line is piped, that means that it IS buffered, hence the multiple lines at once when the buffer is finally flushed to p.stdout.

What can I do to make it work ?

• jfs: related: catching stdout in realtime from subprocess
• jfs: related: Python subprocess readlines() hangs
• Nolhian: Sorry, saw your latest comment after finding the solution. Still upvoted because the answer matches my problem.

Answer 1

Thanks to pobrelkey who found the source of the problem. Indeed, the delay is due to the fact that the child is buffering its write to stdout because it is not writing to a tty. The child uses stdio which is line buffered when writing to a tty, else it is fully buffered.

I managed to get it to work by using pexpect instead of subprocess. pexpect uses a pseudo-tty and that's exactly what we need here :

p = pexpect.spawn(cmd,args,timeout=None) 
line = p.readline() 
while line:
 sys.stdout.write(line)
 sys.stdout.flush()
 # DO OTHER STUFF
 line = p.readline()

Or even better in my case :

p = pexpect.spawn(cmd,args,timeout=None,logfile=sys.stdout)
line = p.readline() 
while line:
 # DO OTHER STUFF
 line = p.readline()

No more delay !

More infos about pexpect : wiki

• jfs: if the child uses stdio then you could make it to switch to line-buffering using stdbuf -oL (or unbuffer, script). There is a code example in my answer. See also Python: read streaming input from subprocess.communicate()
• Nolhian: Indeed, your post is awesome, I really wish I'd found it before writing this thread. I looked up a lot of "getting output in realtime with subprocess" threads but missed this one. Is there a preferred method between pexpect, stdbuf and tty ?
• jfs: it depends. stdbuf variant requires that the external program (stdbuf) is present. pexpect is a 3rd party pure Python module so you have to learn its API. pty is in stdlib but it might be harder to use (judging by the code example). None works on Windows as is.

Answer 2

I would first make sure the subprocess itself doesn't buffer its output. If the subprocess is in turn a Python program, proceed to the paragraph below to see how to disable output buffering for Python processes.

As per Python, usually the problem is that Python by default buffers stderr and stdout even if you explicitly .flush() it from the code. The solution is to pass -u to Python when starting your program.

Also, you can just do for line in p.stdout instead of the tricky while loop.

P.S. actually I tried running your code (with cmd = ['cat', '/dev/urandom']) and without -u and it outputted everything in real time already; this is on OS X 10.8.

• Nolhian: Just tried with -u and still the same. I guess that cat /dev/urandom would work because it generates a lot of data so the buffer flushes eventually in p.stdout. I'll try to see if I can dig more infos.
• Erik Kaplun: I tried with something slower such as e.g. /var/log/system.log and I'm still getting the output smoothly. Which platform are you on btw?
• jfs: could you provide a code example that demonstrates the .flush() behaviour?
• Nolhian: @Erik Allik : Debian 7 64 bits

Answer 3

If you just want stdout of your child process to go to your stdout, why not just have the child process inherit stdout from your process?

subprocess.Popen(cmd, stdout=None, stderr=subprocess.STDOUT) 

• Nolhian: Indeed but I actually do other stuff other than outputting to stdout. Sorry I edited my post.
• pobrelkey: Ah. Then it will depend on whether the child process buffers its writes to stdout - often this will automatically happen if a process isn't writing to a tty. In which case you'll need to modify the child process instead of your script, or figure out some way of piping its output through a pty (expect or the pty module might help here). Sorry.