Passing environment variables to `subprocess.call`

Question

I have a bash script my_script.sh containing a line like ./my_python_script.py 2>&1 and within the Python script is contained the lines

import os
from util import do_something
project_dir = os.environ["PROJECT_DIR"]
do_something(spider_name="my_spider", project_dir=project_dir, data_dir="tmp/")

where do_something is something like

import subprocess
def do_something(spider_name, project_dir, data_dir):
 subprocess.call(["scrapy", "crawl", spider_name, 
 "--set", "FEED_URI=%s%s%s" % (project_dir, data_dir, spider_name+".json"),
 cwd=project_dir+"scrapers/"+spider_name+"/")

I'm calling my_script.sh in a cron job like so:

0 13 * * * . /home/ubuntu/.profile && cd $HOME/project_dir/ && ./my_script.sh > logs/daily_$CURR_DATE.log 2>&1

My problem is that the top level script is able to see the environment passed to it, but do_something is somehow not able to use it, reporting that

Traceback (most recent call last):
 File "./my_python_script.py", line 19, in <module>
 project_dir=project_dir, data_dir="tmp/")
 File "/home/ubuntu/project_dir/util/util.py", line 41, in 
do_something
 cwd=project_dir+"scrapers/"+spider_name+"/")
 File "/usr/lib/python2.7/subprocess.py", line 523, in call
 return Popen(*popenargs, **kwargs).wait()
 File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
 errread, errwrite)
 File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
 raise child_exception
OSError: [Errno 2] No such file or directory

The directory containing the Scrapy project does, in fact, exist, and I'm able to run my_script.sh manually without problem. If I do a

print os.listdir(project_dir+"scrapers/"+spider_name+"/")

right before the call to subprocess.call I can see the complete contents of the project's directory.

What is going on here? Why isn't subprocess able to change directories when I'm calling the script from a cronjob, but is able to when I run the script manually? I'm really at a loss. Thanks in advance for any insight.

• Anil Vaitla: Does changing the cronjob to use the bash shell change the result: unix.stackexchange.com/questions/94456/…
• user4601931: @AnilVaitla Yes, I tried that with no effect. I think it's not a cron issue, but rather a subprocess issue.

Answer 1

The problem turned out to be that subprocess.call couldn't find cd when invoked as I wrote, and hence could not make use of the cwd option. Prepending

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin

to my crontab, or calling subprocess.call with the flag shell=True, brought cd into scope and fixed the issue.

• Daniel Pryden: cd is not an executable (how could it be?), it's a shell built-in.
• Anil Vaitla: Nice, yeah I think shell=True would do it. I feel I have gone through that discovery in the past as well. Good to hear it works for you!