Python version: 2.6.7 I have the following subprocess.call within a for loop which is exectuted 18 times, however, the process constantly hangs on the 19th loop:
if config.get_bool_option(NAME, 'exclude_generated_code', True):
for conf in desc.iter_configs():
for gen in desc.iter_generators(conf):
generator.initialize_generated_path(gen, desc)
for genpath in gen.generated_path:
os.rename(cov_file, cov_file+'.temp')
exclude = ['lcov']
exclude += ['-r', cov_file+'.temp', '"'+genpath+'/*"']
exclude += ['-o', cov_file]
if verbose: Tracer.log.info("Running "+ ' '.join(exclude))
try:
subprocess.call(' '.join(exclude), stdout=out, stderr=out, shell=True)
except subprocess.CalledProcessError, e:
if verbose: Tracer.log.info("TESTING: Got Exception \n")
The console output looks as follows:
Running lcov -r /remote/XXXXXX/coverage.19.temp "/remote/XXXXXX/xml/2009a/generated/*" -o /remote/XXXXX/gcov/coverage.19
Since I am not very familiar with python scripts, I just wandered whether I am doing something wrong here...I suspect a deadlock somewhere..
Would the stdout, stderr = process.communicate() deal with these issues?
Any expert answer on in which cases would the subprocess.call hang please? Thanks very much
1 Answer 1
When using subprocess, I tend to do something like this:
is_running = lambda: my_process.poll() is None
my_process = subprocess.Popen(' '.join(exclude),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
# Grab all the output from stdout and stderr and log it
while is_running():
rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1)
# Log stdout, but don't spam the log
if my_process.stdout in rlist and verbose:
# Adjust the number of bytes read however you like, 1024 seems to work
# pretty well for me.
Tracer.log.debug(my_process.stdout.read(1024))
# Log stderr, always
if my_process.stderr in rlist:
# Same as with stdout, adjust the bytes read as needed.
Tracer.log.error(my_process.stderr.read(1024))
I've seen the stdout stuff just dump a bunch of empty lines in my logs in the past, which is why I log that at the debug level. That prints to my logs during the development, but never gets written in production, so I can safely leave it in the code for debugging without putting garbage in their logs.
Hopefully, this can help expose just where your program is hanging and what's causing it.