Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

Running Bash commands in Python

On my local machine, I run a python script which contains this line

bashCommand = "cwm --rdf test.rdf --ntriples > test.nt"
os.system(bashCommand)

This works fine.

Then I run the same code on a server and I get the following error message

'import site' failed; use -v for traceback
Traceback (most recent call last):
File "/usr/bin/cwm", line 48, in <module>
from swap import diag
ImportError: No module named swap

So what I did then is I inserted a print bashCommand which prints me than the command in the terminal before it runs it with os.system().

Of course, I get again the error (caused by os.system(bashCommand)) but before that error it prints the command in the terminal. Then I just copied that output and did a copy paste into the terminal and hit enter and it works...

Does anyone have a clue what's going on?

Answer*

To somewhat expand on the earlier answers here, there are a number of details which are commonly overlooked.

* Prefer `subprocess.run()` over `subprocess.check_call()` and friends over `subprocess.call()` over `subprocess.Popen()` over `os.system()` over `os.popen()`
* Understand and probably use `text=True`, aka `universal_newlines=True`.
* Understand the meaning of `shell=True` or `shell=False` and how it changes quoting and the availability of shell conveniences.
* Understand differences between `sh` and Bash
* Understand how a subprocess is separate from its parent, and generally cannot change the parent.
* Avoid running the Python interpreter as a subprocess of Python.

These topics are covered in some more detail below.


# Prefer `subprocess.run()` or `subprocess.check_call()`

The `subprocess.Popen()` function is a low-level workhorse but it is tricky to use correctly and you end up copy/pasting multiple lines of code ... which conveniently already exist in the standard library as a set of higher-level wrapper functions for various purposes, which are presented in more detail in the following.

Here's a paragraph from the [documentation](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module):

> The recommended approach to invoking subprocesses is to use the `run()` function for all use cases it can handle. For more advanced use cases, the underlying `Popen` interface can be used directly.

Unfortunately, the availability of these wrapper functions differs between Python versions.

* `subprocess.run()` was officially introduced in Python 3.5. It is meant to replace all of the following.
* `subprocess.check_output()` was introduced in Python 2.7 / 3.1. It is basically equivalent to `subprocess.run(..., check=True, stdout=subprocess.PIPE).stdout`
* `subprocess.check_call()` was introduced in Python 2.5. It is basically equivalent to `subprocess.run(..., check=True).returncode` (where the return code is trivially always 0)
* `subprocess.call()` was introduced in Python 2.4 in the original `subprocess` module ([PEP-324](https://www.python.org/dev/peps/pep-0324/)). It is basically equivalent to `subprocess.run(...).returncode`

### High-level API vs `subprocess.Popen()`

The refactored and extended `subprocess.run()` is more logical and more versatile than the older legacy functions it replaces. It returns a [`CompletedProcess`](https://docs.python.org/3/library/subprocess.html#subprocess.CompletedProcess) object which has various methods which allow you to retrieve the exit status, the standard output, and a few other results and status indicators from the finished subprocess.

`subprocess.run()` is the way to go if you simply need a program to run and return control to Python. For more involved scenarios (background processes, perhaps with interactive I/O with the Python parent program) you still need to use `subprocess.Popen()` and take care of all the plumbing yourself. This requires a fairly intricate understanding of all the moving parts and should not be undertaken lightly. The simpler [`Popen` object](https://docs.python.org/3/library/subprocess.html#subprocess.Popen) represents the (possibly still-running) process which needs to be managed from your code for the remainder of the lifetime of the subprocess.

It should perhaps be emphasized that just `subprocess.Popen()` merely creates a process. If you leave it at that, you have a subprocess running concurrently alongside with Python, so a "background" process. If it doesn't need to do input or output or otherwise coordinate with you, it can do useful work in parallel with your Python program.

### Avoid `os.system()` and `os.popen()`

Since time eternal (well, since Python 2.5) the [`os` module documentation](https://docs.python.org/release/2.5.2/lib/os-process.html) has contained the recommendation to prefer `subprocess` over `os.system()`:

> The `subprocess` module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function.

The problems with `system()` are that it's obviously system-dependent and doesn't offer ways to interact with the subprocess. It simply runs, with standard output and standard error outside of Python's reach. The only information Python receives back is the exit status of the command (zero means success, though the meaning of non-zero values is also somewhat system-dependent).

[PEP-324](https://www.python.org/dev/peps/pep-0324/) (which was already mentioned above) contains a more detailed rationale for why `os.system` is problematic and how `subprocess` attempts to solve those issues.

`os.popen()` used to be even more [strongly discouraged](https://docs.python.org/2/library/os.html#os.popen):

> *Deprecated since version 2.6:* This function is obsolete. Use the `subprocess` module.

However, since sometime in Python 3, it has been reimplemented to simply use `subprocess`, and redirects to the `subprocess.Popen()` documentation for details.


### Understand and usually use `check=True`

You'll also notice that `subprocess.call()` has many of the same limitations as `os.system()`. In regular use, you should generally check whether the process finished successfully, which `subprocess.check_call()` and `subprocess.check_output()` do (where the latter also returns the standard output of the finished subprocess). Similarly, you should usually use `check=True` with `subprocess.run()` unless you specifically need to allow the subprocess to return an error status.

In practice, with `check=True` or `subprocess.check_*`, Python will throw a [`CalledProcessError` exception](https://docs.python.org/3/library/subprocess.html#subprocess.CalledProcessError) if the subprocess returns a nonzero exit status.

A common error with `subprocess.run()` is to omit `check=True` and be surprised when downstream code fails if the subprocess failed.

On the other hand, a common problem with `check_call()` and `check_output()` was that users who blindly used these functions were surprised when the exception was raised e.g. when `grep` did not find a match. (You should probably replace `grep` with native Python code anyway, as outlined below.)

All things counted, you need to understand how shell commands return an exit code, and under what conditions they will return a non-zero (error) exit code, and make a conscious decision how exactly it should be handled.


# Understand and probably use `text=True` aka `universal_newlines=True`

Since Python 3, strings internal to Python are Unicode strings. But there is no guarantee that a subprocess generates Unicode output, or strings at all.

(If the differences are not immediately obvious, Ned Batchelder's [Pragmatic Unicode](https://nedbatchelder.com/text/unipain.html) is recommended, if not outright obligatory, reading. There is a 36-minute video presentation behind the link if you prefer, though reading the page yourself will probably take significantly less time.)

Deep down, Python has to fetch a `bytes` buffer and interpret it somehow. If it contains a blob of binary data, it *shouldn't* be decoded into a Unicode string, because that's error-prone and bug-inducing behavior - precisely the sort of pesky behavior which riddled many Python 2 scripts, before there was a way to properly distinguish between encoded text and binary data.

With `text=True`, you tell Python that you, in fact, expect back textual data in the system's default encoding, and that it should be decoded into a Python (Unicode) string to the best of Python's ability (usually UTF-8 on any moderately up to date system, except perhaps Windows?)

If that's *not* what you request back, Python will just give you `bytes` strings in the `stdout` and `stderr` strings. Maybe at some later point you *do* know that they were text strings after all, and you know their encoding. Then, you can decode them.

 normal = subprocess.run([external, arg],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE,
 check=True,
 text=True)
 print(normal.stdout)

 convoluted = subprocess.run([external, arg],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE,
 check=True)
 # You have to know (or guess) the encoding
 print(convoluted.stdout.decode('utf-8'))

Python 3.7 introduced the shorter and more descriptive and understandable alias `text` for the keyword argument which was previously somewhat misleadingly called `universal_newlines`.


# Understand `shell=True` vs `shell=False`

With `shell=True` you pass a single string to your shell, and the shell takes it from there.

With `shell=False` you pass a list of arguments to the OS, bypassing the shell.

When you don't have a shell, you save a process and get rid of a [fairly substantial amount of hidden complexity, which may or may not harbor bugs or even security problems.](/questions/3172470/actual-meaning-of-shell-true-in-subprocess)

On the other hand, when you don't have a shell, you don't have redirection, wildcard expansion, job control, and a large number of other shell features.

A common mistake is to use `shell=True` and then still pass Python a list of tokens, or vice versa. This happens to work in some cases, but is really ill-defined and could break in interesting ways.


 # XXX AVOID THIS BUG
 buggy = subprocess.run('dig +short stackoverflow.com')

 # XXX AVOID THIS BUG TOO
 broken = subprocess.run(['dig', '+short', 'stackoverflow.com'],
 shell=True)

 # XXX DEFINITELY AVOID THIS
 pathological = subprocess.run(['dig +short stackoverflow.com'],
 shell=True)

 correct = subprocess.run(['dig', '+short', 'stackoverflow.com'],
 # Probably don't forget these, too
 check=True, text=True)

 # XXX Probably better avoid shell=True
 # but this is nominally correct
 fixed_but_fugly = subprocess.run('dig +short stackoverflow.com',
 shell=True,
 # Probably don't forget these, too
 check=True, text=True)

The common retort "but it works for me" is not a useful rebuttal unless you understand exactly under what circumstances it could stop working.

To briefly recap, correct usage looks like

```
subprocess.run("string for 'the shell' to parse", shell=True)
# or
subprocess.run(["list", "of", "tokenized strings"]) # shell=False
```
If you want to avoid the shell but are too lazy or unsure of how to parse a string into a list of tokens, notice that `shlex.split()` can do this for you.

```
subprocess.run(shlex.split("no string for 'the shell' to parse")) # shell=False
# equivalent to
# subprocess.run(["no", "string", "for", "the shell", "to", "parse"])
```
The regular `split()` will not work here, because it doesn't preserve quoting. In the example above, notice how `"the shell"` is a single string, and how in the shell, single quotes were used to produce that string value, where the single quotes are not part of the value itself.


### Refactoring Example

Very often, the features of the shell can be replaced with native Python code. Mundane pipelines with `grep` or `tr` or `cut` etc, and simple Awk or `sed` scripts should probably just be translated to Python instead.

To partially illustrate this, here is a typical but slightly silly example which involves many shell features.

 cmd = '''while read -r x;
 do ping -c 3 "$x" | grep 'min/avg/max'
 done <hosts.txt'''

 # Trivial but horrible
 results = subprocess.run(
 cmd, shell=True, text=True, capture_output=True, check=True)
 print(results.stdout)

 # Reimplement with shell=False
 with open('hosts.txt') as hosts:
 for host in hosts:
 host = host.rstrip('\n') # drop newline
 ping = subprocess.run(
 ['ping', '-c', '3', host],
 text=True,
 stdout=subprocess.PIPE,
 check=True)
 for line in ping.stdout.split('\n'):
 if 'min/avg/max' in line:
 print('{}: {}'.format(host, line))

Some things to note here:

* With `shell=False` you don't need the quoting that the shell requires around strings. Putting quotes anyway is probably an error.
* It often makes sense to run as little code as possible in a subprocess. This gives you more control over execution from within your Python code.
* Having said that, complex shell pipelines are tedious and sometimes challenging to reimplement in Python.

The refactored code also illustrates just how much the shell really does for you with a very terse syntax -- for better or for worse. Python says *explicit is better than implicit* but the Python code *is* rather verbose and arguably looks more complex than this really is. On the other hand, it offers a number of points where you can grab control in the middle of something else, as trivially exemplified by the enhancement that we can easily include the host name along with the shell command output. (This is by no means challenging to do in the shell, either, but at the expense of yet another diversion and perhaps another process.)

### Common Shell Constructs

For completeness, here are brief explanations of some of these shell features, and some notes on how they can perhaps be replaced with native Python facilities.

* Globbing aka wildcard expansion can be replaced with `glob.glob()` or very often with simple Python string comparisons like `for file in os.listdir('.'): if not file.endswith('.png'): continue`. Bash has various other expansion facilities like `.{png,jpg}` brace expansion and `{1..100}` as well as tilde expansion (`~` expands to your home directory, and more generally `~account` to the home directory of another user)
* Shell variables like `$SHELL` or `$my_exported_var` can sometimes simply be replaced with Python variables. Exported shell variables are available as e.g. `os.environ['SHELL']` (the meaning of `export` is to make the variable available to subprocesses -- a variable which is not available to subprocesses will obviously not be available to Python running as a subprocess of the shell, or vice versa. The `env=` keyword argument to `subprocess` methods allows you to define the environment of the subprocess as a dictionary, so that's one way to make a Python variable visible to a subprocess). With `shell=False` you will need to understand how to remove any quotes; for example, `cd "$HOME"` is equivalent to `os.chdir(os.environ['HOME'])` without quotes around the directory name. (Very often `cd` is not useful or necessary anyway, and many beginners omit the double quotes around the variable and get away with it [until one day ...](/questions/10067266/when-to-wrap-quotes-around-a-shell-variable))
* Redirection allows you to read from a file as your standard input, and write your standard output to a file. `grep 'foo' <inputfile >outputfile` opens `outputfile` for writing and `inputfile` for reading, and passes its contents as standard input to `grep`, whose standard output then lands in `outputfile`. This is not generally hard to replace with native Python code.
* Pipelines are a form of redirection. `echo foo | nl` runs two subprocesses, where the standard output of `echo` is the standard input of `nl` (on the OS level, in Unix-like systems, this is a single file handle). If you cannot replace one or both ends of the pipeline with native Python code, perhaps think about using a shell after all, especially if the pipeline has more than two or three processes (though look at the [`pipes` module in the Python standard library](https://docs.python.org/3.7/library/pipes.html) or a number of more modern and versatile third-party competitors).
* Job control lets you interrupt jobs, run them in the background, return them to the foreground, etc. The basic Unix signals to stop and continue a process are of course available from Python, too. But jobs are a higher-level abstraction in the shell which involve process groups etc which you have to understand if you want to do something like this from Python.
* Quoting in the shell is potentially confusing until you understand that *everything* is basically a string. So `ls -l /` is equivalent to `'ls' '-l' '/'` but the quoting around literals is completely optional. Unquoted strings which contain shell metacharacters undergo parameter expansion, whitespace tokenization and wildcard expansion; double quotes prevent whitespace tokenization and wildcard expansion but allow parameter expansions (variable substitution, command substitution, and backslash processing). This is simple in theory but can get bewildering, especially when there are several layers of interpretation (a remote shell command, for example).

# Understand differences between `sh` and Bash

`subprocess` runs your shell commands with `/bin/sh` unless you specifically request otherwise (except of course on Windows, where it uses the value of the `COMSPEC` variable). This means that [various Bash-only features like arrays, `[[` etc](/a/42666651/874188) are not available.

If you need to use Bash-only syntax, you can 
pass in the path to the shell as `executable='/bin/bash'` (where of course if your Bash is installed somewhere else, you need to adjust the path).

 subprocess.run('''
 # This for loop syntax is Bash only
 for((i=1;i<=$#;i++)); do
 # Arrays are Bash-only
 array[i]+=123
 done''',
 shell=True, check=True,
 executable='/bin/bash')


# A `subprocess` is separate from its parent, and cannot change it

A somewhat common mistake is doing something like

 subprocess.run('cd /tmp', shell=True)
 subprocess.run('pwd', shell=True) # Oops, doesn't print /tmp

The same thing will happen if the first subprocess tries to set an environment variable, which of course will have disappeared when you run another subprocess, etc.

A child process runs completely separate from Python, and when it finishes, Python has no idea what it did (apart from the vague indicators that it can infer from the exit status and output from the child process). A child generally cannot change the parent's environment; it cannot set a variable, change the working directory, or, in so many words, communicate with its parent without cooperation from the parent.

The immediate fix in this particular case is to run both commands in a single subprocess;

 subprocess.run('cd /tmp; pwd', shell=True)

though obviously this particular use case isn't very useful; instead, use the `cwd` keyword argument, or simply `os.chdir()` before running the subprocess. Similarly, for setting a variable, you can manipulate the environment of the current process (and thus also its children) via

 os.environ['foo'] = 'bar'

or pass an environment setting to a child process with

 subprocess.run('echo "$foo"', shell=True, env={'foo': 'bar'})

(not to mention the obvious refactoring `subprocess.run(['echo', 'bar'])`; but `echo` is a poor example of something to run in a subprocess in the first place, of course).


# Don't run Python from Python

This is slightly dubious advice; there are certainly situations where it does make sense or is even an absolute requirement to run the Python interpreter as a subprocess from a Python script. But very frequently, the correct approach is simply to `import` the other Python module into your calling script and call its functions directly.

If the other Python script is under your control, and it isn't a module, consider [turning it into one](/questions/15746675/how-to-write-a-python-module-package). (This answer is too long already so I will not delve into details here.)

If you need parallelism, you can run Python functions in subprocesses with the [`multiprocessing` module.](https://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing) There is also [`threading`](https://docs.python.org/3/library/threading.html#module-threading) which runs multiple tasks in a single process (which is more lightweight and gives you more control, but also more constrained in that threads within a process are tightly coupled, and bound to a single [GIL](https://docs.python.org/3/glossary.html#term-global-interpreter-lock).)

Draft saved

Draft discarded

Edit Summary*

Cancel

5

For a more detailed exposition of how you might avoid calling Python as a subprocess, see this answer on a tangentially similar question.

tripleee
– tripleee

2018年09月08日 10:06:48 +00:00
Commented Sep 8, 2018 at 10:06
11

it boggles my mind that I had to post a new answer to such a basic question in order to show how to run the command from the question idiomatically. Your answer is long but I don't see such example. Unrelated: avoid cargo-culting. If check_call() works in your case, use it. I had to fix a code that used run() blindly. Missing check=True caused a bug which would be avoided if check_call were used — "check" is in the name, you can't lose it —it is the correct default: don't ignore errors silently. I didn't read further.

jfs
– jfs

2018年11月01日 19:47:50 +00:00
Commented Nov 1, 2018 at 19:47
3

@jfs Thanks for the feedback, I was in fact planning to add a section about Bash vs sh but you beat me to it. I'm trying to spell out the specifics in enough detail to help beginners for whom these pitfalls are not obvious so that does get a bit longwinded. Yours should be quite sufficient otherwise; +1

tripleee
– tripleee

2018年11月02日 05:13:13 +00:00
Commented Nov 2, 2018 at 5:13
1

Does stderr/stdout = subprocess.PIPE have a higher performance overhead than the default settings?

Stringers
– Stringers

2019年04月05日 03:02:52 +00:00
Commented Apr 5, 2019 at 3:02
3

@Stringers I have not tested, but I don't see why it should. If you connect those pipes to something which does some processing, then of course that processing needs to oe accouted for; but it doesn't happen in the pipe itself. The default is to not capture stdout or stderr at all, i.e. whatever gets printed there is out of Python's visibility and control, just like with os.system().

tripleee
– tripleee

2019年04月05日 03:52:43 +00:00
Commented Apr 5, 2019 at 3:52

| Show 7 more comments

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. python-3.x), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

default

CollectivesTM on Stack Overflow

Running Bash commands in Python

Answer*