Running bash command with redirection in python

Question

I want to use redirection from bash function return values to e.g. grep, and call it from python.

Even though I am using shell=True in subprocess, this does not seem to work.

Example that works in bash:

grep foo <(echo "fooline")

Example that gives the error /bin/sh: 1: Syntax error: "(" unexpected in python 3.7.3, Ubuntu 19.04:

#!/usr/bin/env python3
import subprocess
subprocess.call('grep foo <(echo "fooline")', shell=True)

According to answers like these, redirection should work with shell=True (and it does for redirecting actual files, but not return values).

EDIT: Added shebang and python version.

• LearnOPhile: It would be nice with good documentation for all inconsistencies like these so that people are able to move from bash scripting to python scripting without too much unpredictable behavior.
• m02ph3u5: echo "fooline" | grep foo?
• Piotr Kamoda: Your example works for me. What's your python version you are running?
• Jean-François Fabre: seems that you're missing the python shebang your python script is interpreted as shell
• LearnOPhile: I'm running python 3.7.3 @PiotrKamoda

Answer 1

The inconsistency that you observe is due to the fact that shell=True gives you a sh shell, not bash.

The following is valid in bash but not in sh.

grep foo <(echo "fooline")

Example output:

sh-3.2$ grep foo <(echo "fooline")
sh: syntax error near unexpected token `('

If you use a valid sh expression your approach will work. Alternatively you can specify which shell to use with executable='/bin/bash'. You can also use something like:

subprocess.Popen(['/bin/bash', '-c', cmd])

Answer 2

The error is a shell error, nothing to do with python. sh chokes on the parentheses of python syntax.

/bin/sh: 1: Syntax error: "(" unexpected

Let's add a shebang (reference: Should I put #! (shebang) in Python scripts, and what form should it take?)

And also stop using shell=True and all. Use real pipes and command lines with arguments (note: this has been tested & works on windows using a native grep command, so now this is portable)

#!/usr/bin/env python3
import subprocess
p = subprocess.Popen(['grep','foo'],stdin = subprocess.PIPE)
p.stdin.write(b"fooline\n")
p.stdin.close()
p.wait()

• LearnOPhile: The issue with using python pipes is that it does not let us drop-in well tested bash-code into python scripts. This solution requires us to refactor everything to python-style. Any ideas on how to do this while using shell=True?
• LearnOPhile: Adding shebang does not help
• Jean-François Fabre: okay I wasn"t sure of what you really needed. If you're using python to test bash code, then you can't use my solution. Consider it when you want to make your code less pythonic, and ... well maybe it's time to drop bash entirely and switch to python for scripts.