Trouble calling bash from within python

Question

I am trying to use python to help do some automation around an incremental build function in the Android build system. Generally, from a given directory, I would execute the following command to build whatever is in that directory and subdirectories:

mm -j8

This is analogous to a "make" command, only it is incremental build and is defined as a function in a bash file called envsetup.sh. What is does it not important, just know that it's a function defined in a bash script somewhere in the file system. To execute this, I can also do:

bash -c ". /path/to/envsetup.sh; mm -j8"

This method of calling it will be important in calling the function from python. I have followed the solution here which shows how to call a function within a bash script from python. I have used this method in a simple script that, in theory, should just spit out the STDOUT and STDERR from executing the command:

import subprocess
command = ['bash', '-c', '. /path/to/envsetup.sh; mm -j8']
(stdout, stderr) = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
print 'stdout: ' + stdout
print 'stderr: ' + stderr

The call to Popen, however, never returns. What am I doing wrong that would allow bash to execute the command properly, but Python hangs when executing the command?

Answer 1

tl; dr:

Your issue is the use of shell=True. Set it to shell=False and it'll work.

With this option set, python will just run the first element of the command array, i.e. bash as a shell script. So currently, python is launching a shell of its own, in order to run your command (bash). It'll run bash with no arguments, and bash will then wait for input, blocking your python script.

The shell=True setting is for use cases where you are passing a shell script in as a single string. When you're explicitly specifying a shell and its parameters as the process to invoke, as you are doing above, you should set shell=False.

>>> import subprocess
>>> subprocess.Popen(['bash', 'whatever'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()

Here's what the proces tree looks like when I run the above:

 \_ python
 \_ /bin/sh -c bash whatever
 \_ bash

The whatever is actually passed in, but it's a parameter to the sh, not a parameter to the inner bash, so the command being run is effectively ['/bin/sh', '-c', 'bash', 'whatever'], which is quite different from ['/bin/sh', '-c', 'bash whatever']