Using back-ticks in Python subprocess

Question

I want to run this git command through a Python script and get the output of it:

git diff --name-only mybranch `git merge-base mybranch develop`

The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.

To achieve this I'm using subprocess.Popen:

output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)

However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.

I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.

I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.

Any help or tips are appreciated.

• Padraic Cunningham: why are you trying to use shell=True and passing a list of args? Also why is git in a separate list?`
• Jokab: The answer in the linked question said that using back-ticks would only work with shell=True. They are in separate lists for no particular reason, I'll edit to avoid confusion.
• Padraic Cunningham: You should either use a string with shell=True or remove shell=True and pass a list of individual args
• Jokab: Thank you, this was the answer. I must have missed this in the docs.
• PM 2Ring: Note that backticks are deprecated in Bash: the $(cmd) syntax is less prone to being mis-read, and you can nest it if necessary. See BashFAQ/082 "Why is $(...) preferred over (backticks)?" in the BashGuide for further details. Also, backticks are a pain to write in SO comments. :)

Answer 1

Backticks and subprocess

The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args

So for your particular command (assuming it works in the first place)

process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)

Notice when you call Popen() you get a process, shouldn't be called output IMO

Here's a simple example that works with backticks

>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'

Or you can use the $(cmd) syntax

>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'

Here's what did NOT work (for backticks)

>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'

• bakkal: @PM2Ring Added a sample using $(cmd) with subprocess
• PM 2Ring: Thanks. I decided to move my comment to the question, since it's probably more appropriate there.
• jfs: you have a choice, you can drop shell=True
• bakkal: You'd have to do the backtick expansion manually, because from my trials it doesn't seem you can expand backticks or $(cmd) without shell=True, backticks being the crux of the question

Answer 2

On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:

On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

You don't need shell=True in this case.

#!/usr/bin/env python
from subprocess import check_output
merge_base_output = check_output('git merge-base mybranch develop'.split(), 
 universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
 [merge_base_output])