basic multiprocessing with python

Question

I have found information on multiprocessing and multithreading in python but I don't understand the basic concepts and all the examples that I found are more difficult than what I'm trying to do.

I have X independent programs that I need to run. I want to launch the first Y programs (where Y is the number of cores of my computer and X>>Y). As soon as one of the independent programs is done, I want the next program to run in the next available core. I thought that this would be straightforward, but I keep getting stuck on it. Any help in solving this problem would be much appreciated.

Edit: Thanks a lot for your answers. I also found another solution using the joblib module that I wanted to share. Suppose that you have a script called 'program.py' that you want to run with different combination of the input parameters (a0,b0,c0) and you want to use all your cores. This is a solution.

import os
from joblib import Parallel, delayed
a0 = arange(0.1,1.1,0.1)
b0 = arange(-1.5,-0.4,0.1)
c0 = arange(1.,5.,0.1)
params = []
for i in range(len(a0)):
 for j in range(len(b0)):
 for k in range(len(c0)):
 params.append((a0[i],b0[j],c0[k]))
def func(parameters):
 s = 'python program.py %g %g %g' % parameters[0],parameters[1],parameters[2])
 command = os.system(s)
 return command
output = Parallel(n_jobs=-1,verbose=1000)(delayed(func)(i) for i in params)

• Marco de Wit: What do you mean by programs? Separate python scripts? For the Python library it is handy if processes are available as functions in your script.
• Henry Gomersall: If they are genuinely separate programs, you can use gridengine to handle all the task management.

Answer 1

You want to use multiprocessing.Pool, which represents a "pool" of workers (default one per core, though you can specify another number) that do your jobs. You then submit jobs to the pool, and the workers handle them as they become available. The easiest function to use is Pool.map, which runs a given function for each of the arguments in the passed sequence, and returns the result for each argument. If you don't need return values, you could also use apply_async in a loop.

def do_work(arg):
 pass # do whatever you actually want to do
def run_battery(args):
 # args should be like [arg1, arg2, ...]
 pool = multiprocessing.Pool()
 ret_vals = pool.map(do_work, arg_tuples)
 pool.close()
 pool.join()
 return ret_vals

If you're trying to call external programs and not just Python functions, use subprocess. For example, this will call cmd_name with the list of arguments passed, raise an exception if the return code isn't 0, and return the output:

def do_work(subproc_args):
 return subprocess.check_output(['cmd_name'] + list(subproc_args))

Answer 2

Hi i'm using the object QThread from pyqt From what i understood, your thread when he is running can only use his own variable and proc, he cannot change your main object variables So before you run it be sur to define all the qthread variables you will need

like this for example:

class worker(QThread)
def define(self, phase):
 print 'define'
 self.phase=phase
 self.start()#will run your thread
def continueJob(self):
 self.start()
def run(self):
 self.launchProgramme(self.phase)
 self.phase+=1
def launchProgramme(self):
 print self.phase

i'm not well aware of how work the basic python thread but in pyqt your thread launch a signal to your main object like this:

class mainObject(QtGui.QMainWindow)
 def __init__(self):
 super(mcMayaClient).__init__()
 self.numberProgramme=4
 self.thread = Worker()
 #create
 self.connect(self.thread , QtCore.SIGNAL("finished()"), self.threadStoped)
 self.connect(self.thread , QtCore.SIGNAL("terminated()"), self.threadStopped)

connected like this, when the thread.run stop, it will launch your threadStopped proc in your main object where u can get the value of your thread Variables

def threadStopped(self):
 value=self.worker.phase
 if value<self.numberProgramme:
 self.worker.continueJob()

after that you just have to lauch another thread or not depending of the value you get This is for pyqt threading of course, in python basic thread, the way to execute the def threadStopped could be different.