Dynamic output from python subprocess module

Question

how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok , ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"

when i try to do :

cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)

it keep storing output in buffers

• Torxed: Possible duplicate of Printing output in realtime from subprocess. Altho that's just on the question itself, but it won't work for ngrok or other ncurses application. So just leaving this here for others who end up here wondering how to get output from subprocess.

Answer 1

I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate().

So here's a snippet that might be useful:

import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while process.poll() is None:
 print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()

This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.

This piece of code would work, if it weren't for the fact that ngrok uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host.

process.poll() checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout.

There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.

For instance, process.stdout.read() could be used in junction with select.select() to achieve better buffered output where new-lines are scares. Because if a \n never comes, the above example might hang your entire application.

There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate() instead.

Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork() and read the childs stdout via the os.read module:

#!/usr/bin/python
## Requires: Linux
## Does not require: Pexpect
import pty, os
from os import fork, waitpid, execv, read, write, kill
def pid_exists(pid):
 """Check whether pid exists in the current process table."""
 if pid < 0:
 return False
 try:
 kill(pid, 0)
 except (OSError, e):
 return e.errno == errno.EPERMRM
 else:
 return True
class exec():
 def __init__(self):
 self.run()
 def run(self):
 command = [
 '/usr/bin/ngrok',
 'http',
 '5000'
 ]
 # PID = 0 for child, and the PID of the child for the parent 
 pid, child_fd = pty.fork()
 if not pid: # Child process
 # Replace child process with our SSH process
 execv(command[0], command)
 while True:
 output = read(child_fd, 1024)
 print(output.decode('UTF-8'))
 lower = output.lower()
 # example input (if needed)
 if b'password:' in lower:
 write(child_fd, b'some response\n')
 waitpid(pid, 0)
exec()

There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.

But this would give you the output of the process at least. replacing print(output.decode('UTF-8')) would show you what that output is.

• arslanmughal: it looks that program stuck at this line of code """process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)""" after reaching this line the program start storing output in buffers and become silent .. the only thing i want to extract from ngrok while ngrok is running is the farwarded url that ngrok provides
• Torxed: @arslanmughal You are partially correct, it's not stuck on the Popen() line tho. It's stuck on print(process.stdout.readline()). This is a huge different because it just tells us that Popen() isn't able to get the output from the process. I also tried stdout.read(1) which returns nothing. So I added an additional way of getting the output.