Python running multiple processes

Question

I have a script (a.py) that needs to call another script (b.py) multiple times, each execution of b.py takes 1 minute. If I run in a for loop, this takes a lot of time.

I want to know how I can optimize this, to reduce the time. Any help or suggestions would be very helpful.

Source code:

# a.py
import os
if __name__ == '__main__':
 inputs = ['file1', 'file2', 'file3', 'file4']
 script_path = 'b.py'
 # Some logging statements. 
 for infile in inputs:
 os.system("{} -i {}".format(script_path, infile)) 
# b.py
 # take arguments
 # perform some operations
 # log the execution

So far, I have been using os.system to call the other script. How can I call the script b.py n times in parallel ?

• Jebby: It would be helpful if you edited your post to show what you have tried. All of the generous programmers on stackoverflow are much more helpful if you provide something as a base for us to work with. The least you can do is provide a minimal, complete, and verifiable example of your code.
• RecencyEffect: have a look at the subprocess module

Answer 1

You may use muliprocessing.Process to run it in parallel:

from multiprocessing import Process
inputs = ['file1', 'file2', 'file3', 'file4']
script_path = 'b.py'
def run(script, name):
 os.system("{} -i {}".format(script, name)) 
if __name__ == '__main__':
 inputs = ['file1', 'file2', 'file3', 'file4']
 script_path = 'b.py'
 for infile in inputs:
 p = Process(target=run, args=(script_path, infile))
 p.start()
 p.join()

Note: Executing a Python script from a Python script with os.system is not very elegant. You should modify your script b.py that it works as a module which provides its main functionality with functions or classes as well. Then you can import b and use these functions or classes instead.

• R George: Subprocess looks good, I haven't used this module before. I will give it a go. Also, even if I make it a module and import the functions, it still works serially, right ? How can I invoke it in parallel.
• clemens: The parallel execution is done by subprocess. Of course, my suggestion is to execute the code of module b in parallel.

Answer 2

There isn't a way you can optimise this as such. If you want to run script b for each item in a list, then iteration is the only way to do this. It might be that you can alter the contents of script b to run faster, but that is a different problem.