Python: Control subprocess from calling thread

Question

I am working on a database interface to a MongoDB using Python Eve as an API.

The database stores several documents containing parameters and values needed for some calculations. I wrote a Python module containing all the calculation routines which need to have access to the desired data.

Since the calculations need some time to complete I would prefer multiprocessing instead of multithreading in order to use several cores for co-current simulations (doing calculations in parallel, one calculation per core, but with different input parameters, not splitting one calculation to several cores).

In order to start those parallel calculations as own (sub)processes I thought about using something like a parent thread which are observed by another thread called 'thread observer'. Since each of these calculation processes needs access to the database I thought about using the eve-API in combination with the requests module.

All in all these thoughts lead to the setup like the following:

schema

Based on this concept I am not sure about how to control the subprocess from its parent thread. How can I make sure that Process 1 is terminated if Thread 1 terminates?

Note: The desired solution needs to work both on UNIX- and Windows-based machines. So using the returned pid of the started process in order to kill it using kill would not work since this would not be doable on a Windows machine.

• albert: Should I rather ask this question on Stackoverflow? I was not sure about where it does fit since it is conceptual but very python-related.

Answer 1

Start with the Python 2 and Python 3 multiprocessing package having classes like Process with Queue and Pipe to exchange messages between them on all platforms where Python runs. You seem to be interested in the Pool class of multiprocessing to control worker processes. It is a different style of control than the usual fork/exec, wait, and signals.