Parallelize python for-loop

Question

I would like to parallelize the following python loop using a Mac Pro with 2 Quad-Core processors.

result_list = []
for a in a_range:
 for b in b_range:
 for c in c_range:
 result = call_fortran_program(a, b, c)
 result_list.append(result)

In my searches I've run across terms like Cython and GIL but it's still not clear to me on how to proceed.

• Maximilian Peters: Shouldn't result_list.append(result) follow your result = statement? Otherwise there would be only one result.

Answer 1

from itertools import product
from multiprocessing import Pool
with Pool(processes=4) as pool: # assuming Python 3
 pool.starmap(print, product(range(2), range(3), range(4)))

Answer 2

Try ProcessPoolExecutor

This can circumvent GIL lock by creating multiple processes.

An example from the website:

import concurrent.futures
import math
PRIMES = [
 112272535095293,
 112582705942171,
 112272535095293,
 115280095190773,
 115797848077099,
 1099726899285419]
def is_prime(n):
 if n % 2 == 0:
 return False
 sqrt_n = int(math.floor(math.sqrt(n)))
 for i in range(3, sqrt_n + 1, 2):
 if n % i == 0:
 return False
 return True
def main():
 with concurrent.futures.ProcessPoolExecutor() as executor:
 for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
 print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
 main()

Answer 3

You can use asyncio. (Documentation can be found [here][1]). It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem.

import asyncio
def background(f):
 def wrapped(*args, **kwargs):
 return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
 return wrapped
@background
def your_function(argument):
 #code

Now this function will be run in parallel whenever called without putting main program into wait state. You can use it to parallelize for loop as well. When called for a for loop, though loop is sequential but every iteration runs in parallel to the main program as soon as interpreter gets there.

For your case you can do:

import asyncio
import time
from itertools import product
def background(f):
 def wrapped(*args, **kwargs):
 return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs) 
 return wrapped
 
@background
def call_fortran_program(a, b, c):
 time.sleep(max(0,b-a+c)) # Added Sleep to better demonstrate parallelization
 print(f"function called for {a=}, {b=}, {c=}\n", end='')
 return (a, b, c)
a_range, b_range, c_range = range(1,4), range(1,4), range(1,4)
loop = asyncio.get_event_loop() # Have a new event loop
looper = asyncio.gather(*[call_fortran_program(a, b, c) for a in a_range for b in b_range for c in c_range]) # Run the loop
 
result_list = loop.run_until_complete(looper) # Wait until finish
print(result_list)

This gives following output:

function called for a=1, b=1, c=1
function called for a=1, b=2, c=1
function called for a=1, b=1, c=2
function called for a=2, b=1, c=1
function called for a=1, b=3, c=1
function called for a=1, b=2, c=2
function called for a=1, b=1, c=3
function called for a=2, b=1, c=2
function called for a=1, b=3, c=2
function called for a=1, b=2, c=3
function called for a=2, b=1, c=3
function called for a=2, b=2, c=1
function called for a=3, b=1, c=1
function called for a=3, b=1, c=2
function called for a=3, b=2, c=1
function called for a=2, b=2, c=2
function called for a=2, b=3, c=1
function called for a=3, b=2, c=2
function called for a=3, b=1, c=3
function called for a=2, b=2, c=3
function called for a=1, b=3, c=3
function called for a=3, b=3, c=1
function called for a=2, b=3, c=2
function called for a=3, b=2, c=3
function called for a=3, b=3, c=2
function called for a=2, b=3, c=3
function called for a=3, b=3, c=3
[(1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 3, 1), (1, 3, 2), (1, 3, 3), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 3, 1), (2, 3, 2), (2, 3, 3), (3, 1, 1), (3, 1, 2), (3, 1, 3), (3, 2, 1), (3, 2, 2), (3, 2, 3), (3, 3, 1), (3, 3, 2), (3, 3, 3)]