Calling the "source" command from subprocess.Popen

Question

I have a .sh script that I call with source the_script.sh. Calling this regularly is fine. However, I am trying to call it from my python script, through subprocess.Popen.

Calling it from Popen, I am getting the following errors in the following two scenario calls:

foo = subprocess.Popen("source the_script.sh")
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
 errread, errwrite)
 File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
 raise child_exception
OSError: [Errno 2] No such file or directory
>>> foo = subprocess.Popen("source the_script.sh", shell = True)
>>> /bin/sh: source: not found

What gives? Why can't I call "source" from Popen, when I can outside of python?

• sds: Possible duplicate of Emulating Bash 'source' in Python
• tony: Would pypa/hatch's ShellManager (license MIT) w be down your alley?
• Kaz: If Popen is like the popen function in POSIX C, then its main argument is a shell script to execute. You do whatever you would do to solve the problem in that shell script like . /path/to/env.sh ; binary. Source the environment script, semicolon, run binary. && instead of semicolon if the environment script can usefully indicate failure, or might not be found or whatever, so that we want to avoid trying to run the program in that case.

Answer 1

source is not an executable command, it's a shell builtin.

The most usual case for using source is to run a shell script that changes the environment and to retain that environment in the current shell. That's exactly how virtualenv works to modify the default python environment.

Creating a sub-process and using source in the subprocess probably won't do anything useful, it won't modify the environment of the parent process, none of the side-effects of using the sourced script will take place.

Python has an analogous command, execfile, which runs the specified file using the current python global namespace (or another, if you supply one), that you could use in a similar way as the bash command source.

• Rosh Oxymoron: Also note that while execfile is the exact analogue, in Python programs import will be almost always used where you'd typically use source in a shell script.
• coffee: Interesting. So evne if I do as suggested by phihag, any changes to environment variables won't actually stick?
• SingleNegationElimination: well, they will stick in the bash subprocess, but what good that will do you depends on what the_script.sh actually does. It's unlikely that a script meant to be invoked through source is of much use in a subprocess.
• jfs: source: not found - the issues is that /bin/sh doesn't support but /bin/bash does.
• Josh Correia: Note: execfile() was replaced with exec() in Python3

Answer 2

You could just run the command in a subshell and use the results to update the current environment.

def shell_source(script):
 """Sometime you want to emulate the action of "source" in bash,
 settings some environment variables. Here is a way to do it."""
 import subprocess, os
 pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
 output = pipe.communicate()[0]
 env = dict((line.split("=", 1) for line in output.splitlines()))
 os.environ.update(env)

• andybuckley: Credit where it's due: this came from pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (although maybe xApple == Miki?) One note, though: in general the script argument needs to be an explicit path, i.e. "myenv.sh" won't work in general, but "./myenv.sh" will. This is because of the behaviour of the sourcing built-in (.) on systems with a strict implementation of the sh shell (such as Debian/Ubuntu).
• diabloneo: @andybuckley commented right. Using "./myenv.sh" instead "myenv.sh".
• unutbu: This function may raise a ValueError if an environment variable's value contains newlines. To fix, use env -0 and output.split('\x00').
• jfs: @unutbu: I've used your suggestion to support arbitrary bytes except '0円' in environment variables' values
• Sami: I tried shell_source("./myscript.sh") it gives me error ./myscript.sh: 88: Syntax error: end of file unexpected (expecting "then") Traceback (most recent call last): File "readflags_pkg_0V01.py", line 41, in shell_source("./myscript.sh") File "readflags_pkg_0V01.py", line 38, in shell_source env = dict((line.split("=", 1) for line in output.split('\x00'))) ValueError: dictionary update sequence element #0 has length 1; 2 is required Am I doing correct? Is the argument in shell_script() is file object or directly name of the file?

Answer 3

Broken Popen("source the_script.sh") is equivalent to Popen(["source the_script.sh"]) that tries unsuccessfully to launch 'source the_script.sh' program. It can't find it, hence "No such file or directory" error.

Broken Popen("source the_script.sh", shell=True) fails because source is a bash builtin command (type help source in bash) but the default shell is /bin/sh that doesn't understand it (/bin/sh uses .). Assuming there could be other bash-ism in the_script.sh, it should be run using bash:

foo = Popen("source the_script.sh", shell=True, executable="/bin/bash")

As @IfLoop said, it is not very useful to execute source in a subprocess because it can't affect parent's environment.

os.environ.update(env) -based methods fail if the_script.sh executes unset for some variables. os.environ.clear() could be called to reset the environment:

#!/usr/bin/env python2
import os
from pprint import pprint
from subprocess import check_output
os.environ['a'] = 'a'*100
# POSIX: name shall not contain '=', value doesn't contain '0円'
output = check_output("source the_script.sh; env -0", shell=True,
 executable="/bin/bash")
# replace env
os.environ.clear() 
os.environ.update(line.partition('=')[::2] for line in output.split('0円'))
pprint(dict(os.environ)) #NOTE: only `export`ed envvars here

It uses env -0 and .split('0円') suggested by @unutbu

To support arbitrary bytes in os.environb, json module could be used (assuming we use Python version where "json.dumps not parsable by json.loads" issue is fixed):

To avoid passing the environment via pipes, the Python code could be changed to invoke itself in the subprocess environment e.g.:

#!/usr/bin/env python2
import os
import sys
from pipes import quote
from pprint import pprint
if "--child" in sys.argv: # executed in the child environment
 pprint(dict(os.environ))
else:
 python, script = quote(sys.executable), quote(sys.argv[0])
 os.execl("/bin/bash", "/bin/bash", "-c",
 "source the_script.sh; %s %s --child" % (python, script))

• Brecht Machiels: For this to work in Python 3, you need to add universal_newlines=True to the check_output call.
• jfs: @BrechtMachiels universal_newline=True along would be wrong (it would break the code on both Python 2/3). The current solution is general enough to support even undecodable envvar values. On Python 3, I would likely use os.fsdecode on the values. I've updated the answer to make it explicit that the code is for Python 2.

Answer 4

source is a bash-specific shell built-in (and non-interactive shells are often lightweight dash shells instead of bash). Instead, just call /bin/sh:

foo = subprocess.Popen(["/bin/sh", "the_script.sh"])

• jfs: if the_script.sh has proper shebang and permissions (+x) then foo = subprocess.Popen("./the_script.sh") should work.

Answer 5

Update: 2019

"""
 Sometime you want to emulate the action of "source" in bash,
 settings some environment variables. Here is a way to do it.
"""
def shell_source( str_script, lst_filter ):
 #work around to allow variables with new lines
 #example MY_VAR='foo\n'
 #env -i create clean shell
 #bash -c run bash command
 #set -a optional include if you want to export both shell and enrivonment variables
 #env -0 seperates variables with null char instead of newline
 command = shlex.split(f"env -i bash -c 'set -a && source {str_script} && env -0'")
 pipe = subprocess.Popen( command, stdout=subprocess.PIPE )
 #pipe now outputs as byte, so turn it to utf string before parsing
 output = pipe.communicate()[0].decode('utf-8')
 #There was always a trailing empty line for me so im removing it. Delete this line if this is not happening for you.
 output = output[:-1]
 pre_filter_env = {}
 #split using null char
 for line in output.split('\x00'):
 line = line.split( '=', 1)
 pre_filter_env[ line[0]] = line[1]
 post_filter_env = {}
 for str_var in lst_filter:
 post_filter_env[ str_var ] = pre_filter_env[ str_var ]
 os.environ.update( post_filter_env )

Answer 6

A variation on the answer by @xApple since it sometimes is useful to be able to source a shell script (rather than a Python file) to set environment variables, and maybe perform other shell operations, and then propagate that environment to the Python interpreter rather than losing that info when the subshell closes.

The reason for a variation is that the assumption of a one-variable-per-line format of output from "env" isn't 100% robust: I just had to deal with a variable (a shell function, I think) containing a newline, which screwed up that parsing. So here is a slightly more complex version, which uses Python itself to format the environment dictionary in a robust way:

import subprocess
pipe = subprocess.Popen(". ./shellscript.sh; python -c 'import os; print \"newenv = %r\" % os.environ'", 
 stdout=subprocess.PIPE, shell=True)
exec(pipe.communicate()[0])
os.environ.update(newenv)

Maybe there is a neater way? This also ensures that the environment parsing isn't messed up if someone puts an echo statement into the script that's being sourced. Of course, there's an exec in here so be careful about non-trusted input... but I think that's implicit in a discussion about how to source/execute an arbitrary shell script ;-)

UPDATE: see @unutbu's comment on the @xApple answer for an alternative (probably nicer) way to handle newlines in the env output.

• jfs: os.environ.update()-based approach fails if ./shellscript.sh unsets some variables. os.environ.clear() could be used.. You could json.dumps(dict(os.environ)) and json.loads(output) instead of '%r' and exec. Though simple env -0 and .split('0円') works well here.

Answer 7

Using the answers here I created a solution that fit my needs.

• no need to filter out env variables
• allows for variables with new line characters

def shell_source(script):
 """
 Sometime you want to emulate the action of "source" in bash,
 settings some environment variables. Here is a way to do it.
 """
 
 pipe = subprocess.Popen(". %s && env -0" % script, stdout=subprocess.PIPE, shell=True)
 output = pipe.communicate()[0].decode('utf-8')
 output = output[:-1] # fix for index out for range in 'env[ line[0] ] = line[1]'
 env = {}
 # split using null char
 for line in output.split('\x00'):
 line = line.split( '=', 1)
 # print(line)
 env[ line[0] ] = line[1]
 os.environ.update(env)

With this I was able to run commands with the same environment variables without issue:

def runCommand(command):
 """
 Prints and then runs shell command.
 """
 print(f'> running: {command}')
 stream = subprocess.Popen(command, shell=True,env=os.environ)
 (result_data, result_error) = stream.communicate()
 print(f'{result_data}, {result_error}')

Hope this helps anyone in the same position as me

Answer 8

If you want to apply source command to some other scripts or executables, then you might create another wrapping script file and call "source" command from it with any further logic you need. In that case, this source command will modify local context where it running - namely in the subprocess that subprocess.Popen creates.

This will not work if you need to modify python context, where your program is running.

Answer 9

There seem to be a lot of answers to this, haven't read all of them so they may have already pointed it out; but, when calling shell commands like this, you have to pass shell=True to the Popen call. Otherwise, you can call Popen(shlex.split()). make sure to import shlex.

I actually use this function for the purpose of sourcing a file and modifying the current environment.

def set_env(env_file):
 while True:
 source_file = '/tmp/regr.source.%d'%random.randint(0, (2**32)-1)
 if not os.path.isfile(source_file): break
 with open(source_file, 'w') as src_file:
 src_file.write('#!/bin/bash\n')
 src_file.write('source %s\n'%env_file)
 src_file.write('env\n')
 os.chmod(source_file, 0755)
 p = subprocess.Popen(source_file, shell=True,
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 (out, err) = p.communicate()
 setting = re.compile('^(?P<setting>[^=]*)=')
 value = re.compile('=(?P<value>.*$)')
 env_dict = {}
 for line in out.splitlines():
 if setting.search(line) and value.search(line):
 env_dict[setting.search(line).group('setting')] = value.search(line).group('value')
 for k, v in env_dict.items():
 os.environ[k] = v
 for k, v in env_dict.items():
 try:
 assert(os.getenv(k) == v)
 except AssertionError:
 raise Exception('Unable to modify environment')