How to execute process in Python where data is written to stdin?

Question

I have a flag in my Python script which specifies whether I setup and use an external process or not. This process is a command called my_command and it takes data from standard input. If I was to run this on the command-line, it would be something like:

$ my_command < data > result

I want to use a Python script to generate lines of data by modifying standard input and feeding it to my_command.

I'm doing something like this:

import getopt, sys, os, stat, subprocess
# for argument's sake, let's say this is set to True for now
# in real life, I use getopt.getopt() to decide whether this is True or False
useProcess = True
if useProcess:
 process = subprocess.Popen(['my_command'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for line in sys.stdin:
 # parse line from standard input and modify it
 # we store the result in a variable called modified_line
 modified_line = line + "foo"
 # if we want to feed modified_line to my_command, do the following:
 if useProcess:
 process.stdin.write(modified_line)
 # otherwise, we just print the modified line
 else:
 print modified_line

However, my_command behaves as if it does not receive any data and quits with an error state. What am I doing wrong?

EDIT

Let's say my Python script is called my_Python_script. Let's say I would normally pass my_command a file called data over standard input:

$ my_command < data > result

But now I'm passing it to my_Python_script instead:

$ my_Python_script < data > some_other_result

I want my_Python_script to conditionally set up a subprocess that runs my_command on the contents of data (which are modified by my_Python_script before being passed to my_command). Does this make more sense?

If I was using bash as a scripting language, I would conditionally decide to run one of two functions. One would pipe lines of data to my_command. The other would not. Can this be done with Python?

• Hai Vu: What kind of file is my_command? Is it a shell script? Python script? You might want to try something like ['/bin/bash', 'my_command'] or something similar for Python script.
• jfs: you could write my_python_script as a Unix filter instead. Then the python script knows nothing about my_command and just reads from stdin, modifies it somehow, and prints to stdout: `<data my_python_script | my_command >some_other_result
• jfs: If stdout=PIPE then you should read from it otherwise the process might block if it generates enough output.
• James Hirschorn: A simple example of two-way communication between a primary process and a subprocess can be found here: stackoverflow.com/a/52841475/1349673

Answer 1

After writing to the stdin, you need to close it:

 process.stdin.write(modified_line)
 process.stdin.close()

Update

I failed to notice that the process.stdin.write() was executed in a for loop. In which case, you should move the process.stdin.close() to outside the loop.

Also, Raymond mentioned that we should call process.wait() as well. So the updated code should be:

for ...
 process.stdin.write(modified_line)
process.stdin.close()
process.wait()

• Alex Reynolds: This throws a ValueError exception on the second pass through the for loop: ValueError: I/O operation on closed file. The command my_command processes multiple lines (in reality, it mimics a specific application of UNIX sort) so the stdin handle needs to stay open, I'd think.
• Hai Vu: Can you close after the for loop? My bad.

Answer 2

In addition to process.stdin.close() as mentioned by @HaiVu, did you do process.wait() to wait for the command to finish before getting the result?

• Hai Vu: I forgot about process.wait(). Good catch.

Answer 3

It seems like you may be confusing arguments and stdin. Your command should be

$ <data> | mycommand result

with data being passed in once the command is called.

Taking an input is done with the raw_input builtin function. (http://docs.python.org/2/library/functions.html)

• Alex Reynolds: I don't understand. I'm trying to pass my_command modified data from within the script, not have the end user type in data manually via the terminal. Everything should be processed within the script. Does this make more sense?
• entropy: @AlexReynolds if you were typing in at the shell. How would you pass in multiple lines of input as an argument to a command? I think you are confusing arguments(ie: command arg1 arg2 arg2) and stdin(ie: (echo arg1; echo arg2; echo arg3) | command. You have to explain which one you want
• Jeremy Blalock: I think the confusion is that if you are calling $ python name_of_script.py <filename> result you are not passing the file standard input, you are passing the file as an argument. You can get a list of arguments from sys.argv.
• Alex Reynolds: If I use $ my_Python_script < data > result without all the subprocess stuff, it handles standard input and output just fine, so I don't think file arguments are the issue here. In fact, if I leave out the redirect, then the for loop hangs, waiting for standard input. I'm just trying to figure out how to incorporate a process instance of subprocess into this script.
• Raymond Tau: @AlexReynolds, does this command equivalent to the one you provided? cat data | my_Python_script > result. Just to clean up confusion.