Python subprocess call is not working

Question

I'm writing a Python program and I need to execute this rather long shell command and receive outputs within my script.

test = subprocess.Popen(shlex.split("find /home/disk1 -maxdepth 0 -name folder1* -o -name folder7*"), stdout=subprocess.PIPE, shell=True) 
test.communicate()

I've tried all the possible variations of the code, like adding 'executable="/bin/bash" in the arguments, or using subprocess.check_output, etc.

Whenever I run this and check the output by typing 'test.communicate()', I'm getting all the files in '/home/disk1' directory instead of the files that I wanted as per the command. When I type this find search in the shell, it works fine. I don't know why it isn't working. I've spent the last two hours in the internet in vain. I'm baffled here, please help me out.

EDIT: As per the comments, I've also tried omitting 'shell=True' from the arguments, the response in that case is: (b'', None) I'm expecting name of 4 files with this command. This still isn't right. Thanks.

EDIT: Sorry, I'm not looking for files, I'm looking for folders.

• Padraic Cunningham: you don't need shell=True
• Charles Duffy: ...not just "don't need", but "shouldn't use", as a general rule. shell=True is Considered Harmful, and entirely defeats the purpose of using shlex.split().
• Charles Duffy: ...by the way, you should use the -print0 argument to find to NUL-delimit its output, and .split('0円') to divide it up -- otherwise, filenames containing newlines (which, yes, are entirely legal) will mess you up. That's assuming, of course, that you don't do the sane and correct thing and use os.walk() instead.
• chepner: You don't even need to fork a child to run find here. Just use glob.glob('/home/disk1/file[17]*') to get a list of matching file names.

Answer 1

1. The command

   find /home/disk1 -maxdepth 0 -name file1* -o -name file7*
   

   will not produce any output. The only object at depth 0 is /home/disk1 and its name does not match either of the expressions. So the expected result of running that command is empty. You probably wanted -maxdepth 1. However, if you don't need a recursive search, find is overkill; you can just do a glob expansion on the patterns.

2. When you specify shell=True in the constructor for subprocess.Popen, you are expected to pass a single string containing the entire command, which will then be passed to a shell. You can do this either by providing a single string argument or a list with one element which is the string. With shell=False, you need to provide a list whose first element is the program to run, and whose remaining elements are the arguments. Thus, any of the following will produce the expected result:

   test = subprocess.Popen(
    "find /home/disk1 -maxdepth 1 -name file1* -o -name file7*"
    , stdout=subprocess.PIPE, shell=True) 
   test = subprocess.Popen(
    ["find /home/disk1 -maxdepth 1 -name file1* -o -name file7*"]
    , stdout=subprocess.PIPE, shell=True) 
   test = subprocess.Popen(
    shlex.split(
    "find /home/disk1 -maxdepth 1 -name file1* -o -name file7*")
    , stdout=subprocess.PIPE) 
   test = subprocess.Popen(["find", "/home/disk1"
    , "-maxdepth", "1"
    , "-name", "file1*"
    , "-o"
    , "-name", "file7*"]
    , stdout=subprocess.PIPE) 
   

   I would use the last one, myself.

3. It's not well-documented what subprocess.Popen does with a list of more than one element when shell=True is specified, and this usage is not recommended. The Posix implementation adds all the elements of the list to ["sh", "-c"], and then executes that. In effect, that will execute the following:

   sh -c find /home/disk1 -maxdepth 1 -name "file1*" -o -name "file7*"
   

   Here, the argument to -c is just find, /home/disk1 becomes 0ドル, and 1ドル through 7ドル are set to the remaining arguments. The end result is that find is executed with no command-line arguments (in a process whose name is /home/disk1). find with no arguments produces a recursive listing of all files starting at the current working directory.

• rrlamichhane: Thanks, glob is what I need for my scenario, I was oblivious of its existence. About the maxdepth, maxdepth 0 works perfectly for me on the shell but for some reason it only worked with maxdepth 1 with subprocess. Thanks for the help.
• rici: @rrlamichhane: according to man find: "-maxdepth 0 means only apply the tests and actions to the command line arguments." Perhaps you have an alias for find or perhaps you wrote find /home/disk1/*. Shrug. It certainly works as documented on my system.