Error calling subprocess on python when redirecting stdout

Question

I tried to filter a file that was generated by a function in a python script:

out = subprocess.check_output(["sed","-n","'s/pattern/&/p'",oldFile,">",newFile])

However, I got the followong error about my command:

returned non-zero exit status 1

What is wrong?

• devnull: Redirection operators are interpreted by the shell.

Answer 1

As devnull stated, the > is interpreted by the shell. Since it is better to avoid using shell=True, use the stdout parameter instead:

import subprocess
with open(newFile, 'w') as newFile:
 subprocess.check_call(
 ["sed", "-n", "s/S/&/p", oldFile], stdout=newFile)

• bigTree: @Martijn and unutbu: I get: stdout argument not allowed, it will be overridden
• unutbu: With check_output, the stdout argument is not allowed as it is used internally. Use Popen instead.
• Martijn Pieters: Or use call() or check_call().
• unutbu: Yes, maybe check_call would be closest.
• unutbu: Notice in my post that newFile is not a string. Use the with open(...) as newFile statement.

Answer 2

You are using > redirections, which require a shell to interpret the syntax.

When you are redirecting the output of sed, there is no point is using check_output here. Use subprocess.call() or subprocess.check_call() instead and verify the return code.

Either run the command through the shell:

import pipes
out = subprocess.call("sed -n 's/S/&/p' {} > {}".format(
 pipes.quote(oldFile), pipes.quote(newFile), shell=True)

or use a pipe:

with open(newFile, 'w') as pipetarget:
 out = subprocess.call(["sed", "-n", "s/S/&/p", oldFile],
 stdout=pipetarget)

Note that you shouldn't use quotes on the 's/S/&/p' string when used as a separate argument in the argument list; when not passing that to the shell it doesn't need escaping from shell parsing either.