Python subprocess call hangs

Question

Python version: 2.6.7 I have the following subprocess.call within a for loop which is exectuted 18 times, however, the process constantly hangs on the 19th loop:

if config.get_bool_option(NAME, 'exclude_generated_code', True):
 for conf in desc.iter_configs():
 for gen in desc.iter_generators(conf):
 generator.initialize_generated_path(gen, desc)
 for genpath in gen.generated_path:
 os.rename(cov_file, cov_file+'.temp')
 exclude = ['lcov']
 exclude += ['-r', cov_file+'.temp', '"'+genpath+'/*"']
 exclude += ['-o', cov_file]
 if verbose: Tracer.log.info("Running "+ ' '.join(exclude))
 try:
 subprocess.call(' '.join(exclude), stdout=out, stderr=out, shell=True)
 except subprocess.CalledProcessError, e:
 if verbose: Tracer.log.info("TESTING: Got Exception \n") 

The console output looks as follows:

Running lcov -r /remote/XXXXXX/coverage.19.temp "/remote/XXXXXX/xml/2009a/generated/*" -o /remote/XXXXX/gcov/coverage.19

Since I am not very familiar with python scripts, I just wandered whether I am doing something wrong here...I suspect a deadlock somewhere..

Would the stdout, stderr = process.communicate() deal with these issues?

Any expert answer on in which cases would the subprocess.call hang please? Thanks very much

Answer 1

When using subprocess, I tend to do something like this:

is_running = lambda: my_process.poll() is None
my_process = subprocess.Popen(' '.join(exclude), 
 stdout=subprocess.PIPE, 
 stderr=subprocess.PIPE,
 shell=True)
# Grab all the output from stdout and stderr and log it
while is_running():
 rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1)
# Log stdout, but don't spam the log
if my_process.stdout in rlist and verbose:
 # Adjust the number of bytes read however you like, 1024 seems to work 
 # pretty well for me. 
 Tracer.log.debug(my_process.stdout.read(1024))
# Log stderr, always
if my_process.stderr in rlist:
 # Same as with stdout, adjust the bytes read as needed.
 Tracer.log.error(my_process.stderr.read(1024))

I've seen the stdout stuff just dump a bunch of empty lines in my logs in the past, which is why I log that at the debug level. That prints to my logs during the development, but never gets written in production, so I can safely leave it in the code for debugging without putting garbage in their logs.

Hopefully, this can help expose just where your program is hanging and what's causing it.