python subprocess

Question

Suppose I have 4 resources which I want to utilize. I create 4 subprocesses and then wait for 1 of them to be free and as soon as one of them is free I utilize the freed resource again.

I want to know if there is any way we can find out which subprocess has finished working. As of now I am storing all the pid's of children in a list and using for loop to iterate.

for p in ps:
 p.wait()

But this blocks the application until the 1st process has finished. My objective is to wait until any of the subprocesses are finished. Please let me know any ideas.

• chmullig: Are the subprocesses python? If so you should consider the multiprocessing Queue construct.

Answer 1

Retrieve the process ID from the pid field of each subprocess object in your list, and create a dictionary mapping those process IDs back to the subprocess objects. Then call os.wait, which will wait for any of the subprocesses to exit, and use your dictionary to map the PID returned back to the appropriate subprocess. For finer control use os.waitpid or os.wait3 or os.wait4 (the Python documentation doesn't really explain how these work, but AFAIK they map directly to the waitpid and wait4 system calls, so you can use the documentation for those as a guideline).

• kefeizhou: This is a more complicated solution compare to p.poll and os.wait is very much platform specific, so you'd have to handle wait differently for different OS.
• zwol: It's not significantly more code than the loop, and it's the correct approach.
• Nirmal Agarwal: Thanks for the help. I am running in solaris 10 ... I will try this approach as this looks the best possible approach

Answer 2

Use p.poll(). It will tell you if the process has terminated.

• zwol: This is a busywait (it does not yield the CPU), which should be avoided if at all possible.
• kefeizhou: @Zack For 4 parallel processes, I doubt you'll see any real performance difference compared to os.wait. Since he already has code with p.wait(), it's simpler to replace it with p.poll()
• zwol: It's not about performance of this program, it's about wasting CPU cycles that other programs could have used (or could have been spent in deep idle).

Answer 3

In python 3.2.x you can use concurrent futures. A very simple interface which implements the processing engine you described with either processes or threads. (Of course threads are subject to the GIL issue, but in 3.2.x the GIL penalty is apparently far less.) All you do is supply the function that will access the resource of interest.