running separate python programs with subprocess

Question

I am trying to create a script that will run my other python programs. I am new to subprocess module so this is a bit confusing to me.

project structure

/qe-functional
 /qe
 /tests
 cron_functional.py
 test_web_events.py
 setup.sh

cron_functional.py

print(os.getcwd())
# print(subprocess.check_output('ls'))
runtag = "daily_run_" + datetime.today().strftime("%m_%d_%y")
testrun = "source ../../setup.sh; ./test_web_events.py -n 10 -t prf -E ctg-businessevent -p post {}".format(runtag)
cmd = testrun.split()
print(cmd)
subprocess.check_output(cmd)

output

$ python cron_functional.py 
/Users/bli1/Development/QE/qe-functional/qe/tests
['source', '../../setup.sh;', './test_web_events.py', '-n', '10', '-t', 'prf', '-E', 'ctg-businessevent', '-p', 'post', 'daily_run_05_26_15']
Traceback (most recent call last):
 File "cron_functional.py", line 11, in <module>
 subprocess.check_output(cmd)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 566, in check_output
 process = Popen(stdout=PIPE, *popenargs, **kwargs)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
 errread, errwrite)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child
 raise child_exception
OSError: [Errno 2] No such file or directory

• Baris Demiray: source is a bash command, you probably need to add /bin/sh at the very beginning of your command that you pass to subprocess.check_output.
• cdarke: Additionally, using relative path names (../../ and ./) is a problem waiting to happen. If you can the use absolute names or get directory names from the environment (like HOME).
• cdarke: Specify a shell on your call: subprocess.check_output(cmd, shell=True) Note the security hazard warning in the documentation.
• cdarke: Do you realise that you are passing parameters to setup.sh?

Answer 1

source is an internal shell command, not an executable. What you want is not to run one source command with 11 arguments, but a one-liner shell script. You need to pass the whole script as one string to be interpreted by the shell.

subprocess.check_output(testrun, shell=True)

You haven't said what setup.sh does. If it's setting up environment variables and changing the working directory, consider doing that within Python instead. Then you can run

subprocess.check_output(['./test_web_events.py', '-n', '10', ..., '-p', 'post', runtag])

... without involving the shell.