Shell command fails from python, ok from shell

Question

I have a python script that generates a number of shell commands from the given input. The problem is that when it tries to execute the generated commands, it fails, but when i run the generated commands myself (that is, from the command line), they are executed successfully.

Here is the generated command:

find /home/me/downloader/0-29/ -type f | grep -i .rpm$ | xargs -i cp {} /home/me/downloader/builds/0-29/

Here is the error message when it is run by the python script:

find: paths must precede expression: |
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]

Could you help me understand what the problem is?

UPD: Here is the function i use for executing the generated commands:

def exec_command(command):
 process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
 output = process.communicate()[0]
 return output

• theAlse: how are you executing that line from python? provide some code?
• venpa: You might need to quote your command in python to execute...

Answer 1

Since your command is a pipeline, you must set shell=True so that subprocess will send the command, as is, to the shell:

command = 'find /home/me/downloader/0-29/ -type f | grep -i .rpm$ | xargs -i cp {} /home/me/downloader/builds/0-29/'
subprocess.call(command, shell=True)

Or,

process = subprocess.Popen(command, shell=True)
output = process.communicate()[0]
return output

Also, do not do splitting in python on a command with a pipeline. This will result in find being passed | as one of its arguments instead of as a shell operator.

It also appears that the command can be simplified:

command="find /home/me/downloader/0-29/ -type f -iname '*.rpm' -exec cp {} /home/me/downloader/builds/0-29/ \;"

Since the above is no longer a pipeline, it could, with a minor modification, be split and given to subprocess with shell=False. The modification is that the single-quotes around '*.rpm' are there to protect the glob from shell expansion. With shell=False, the shell doesn't remove them. So, we have to. For shell=False and for use with command.split():

command="find /home/me/downloader/0-29/ -type f -iname *.rpm -exec cp {} /home/me/downloader/builds/0-29/ \;"

• user2357112: I think you messed up your quotes in the simplified version.
• chepner: The single quotes around *.rpm are only necessary when the command line is processed by the shell, to prevent pattern expansion; the shell then removes the quotes before passing the literal string to find. With shell=False, you want to leave the single quotes out.

Answer 2

I believe what you do is you start a single program called find with many parameters, incl. | and grep and xargs - and these are not arguments to find.

What you want to do is probably to ask bash to run find and then pipe the outcome to grep, etc. One way to do that is to execute a single command called bash with two parameters (-c) and a whole string incl. piped commands, e.g.

process = subprocess.Popen(["bash", "-c", "cat /etc/issue | grep a"], stdout=subprocess.PIPE)
output=process.communicate()[0]
print output