Python subprocess timing out?

Question

I have a script that runs another command, waits for it to finish, logs the stdout and stderr and based the return code does other stuff. Here is the code:

p = subprocess.Popen(command, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
o, e = p.communicate()
if p.returncode:
 # report error
# do other stuff

The problem I'm having is that if command takes a long time to run none of the other actions get done. The possible errors won't get reported and the other stuff that needs to happen if no errors doesn't get done. It essentially doesn't go past p.communicate() if it takes too long. Some times this command can takes hours (or even longer) to run and some times it can take as little as 5 seconds.

Am I missing something or doing something wrong?

• Padraic Cunningham: communicate is going to block until the command returns, if the output is needed elsewhere in your code then you will need to wit.
• Padraic Cunningham: Also if you want the stdout and stderr logged you can redirect to a file. You can also use check_call with a try/except which will raise a calledProcessError for a non zero exit status or use call if you want your code to continue redirecting the stderr to a file
• jfs: what is your Python version? Does command generate large output (100K or more)? How do you know that the script hangs on .communicate() even long after command has finished? Do you need to handle the output while the command is still running?

Answer 1

As per the documentation located here, it's safe to say that you're code is waiting for the subprocess to finish.

If you need to go do 'other things' while you wait you could create a loop like:

while p.poll():
 # 'other things'
 time.sleep(0.2)

Pick a sleep time that's reasonable for how often you want python to wake up and check the subprocess as well as doing its 'other things'.

Answer 2

The Popen.communicate waits for the process to finish, before anything is returned. Thus it is not ideal for any long running command; and even less so if the subprocess can hang waiting for input, say prompting for a password.

---

The stderr=subprocess.PIPE, stdout=subprocess.PIPE are needed only if you want to capture the output of the command into a variable. If you are OK with the output going to your terminal, then you can remove these both; and even use subprocess.call instead of Popen. Also, if you do not provide input to your subprocess, then do not use stdin=subprocess.PIPE at all, but direct that from the null device instead (in Python 3.3+ you can use stdin=subprocess.DEVNULL; in Python <3.3 use stdin=open(os.devnull, 'rb')

---

If you need the contents too, then instead of calling p.communicate(), you can read p.stdout and p.stderr yourself in chunks and output to the terminal, but it is a bit complicated, as it is easy to deadlock the program - the dummy approach would try to read from the subprocess' stdout while the subprocess would want to write to stderr. For this case there are 2 remedies:

• you could use select.select to poll both stdout and stderr to see whichever becomes ready first and read from it then

• or, if you do not care for stdout and stderr being combined into one, you can use STDOUT to redirect the stderr stream into the stdout stream: stdout=subprocess.PIPE, stderr=subprocess.STDOUT; now all the output comes to p.stdout that you can read easily in loop and output the chunks, without worrying about deadlocks:

---

If the stdout, stderr are going to be huge, you can also spool them to a file right there in Popen; say,

stdout = open('stdout.txt', 'w+b')
stderr = open('stderr.txt', 'w+b')
p = subprocess.Popen(..., stdout=stdout, stderr=stderr)
while p.poll() is None:
 # reading at the end of the file will return an empty string
 err = stderr.read() 
 print(err)
 out = stdout.read()
 print(out)
 # if we met the end of the file, then we can sleep a bit
 # here to avoid spending excess CPU cycles just to poll;
 # another option would be to use `select`
 if not err and not out: # no input, sleep a bit
 time.sleep(0.01)

• tbeauvais: I actually do want the stdout and stderr. Your comment has me thinking though. What would be the behavior if the stdout and stderr being captured is extremely large? Since the script is not behaving as expected with long runs, the stdout can be significant. Would the python script fail while the subprocess continues running?