Python Multiprocessing to invoke a class function

Question

I am trying to multiprocess a large number of objects that I need to invoke a class function off. I cant pull the function outside of the class and so just need to figure out a way to speed up the calling of this function.

Many thanks in advance

import time
# - Bespoke Class that is standalone - #
class sub_class:
 def __init__(self, item_id, assumptions = "BlahBlahBlah"):
 self.item_id = item_id
 self.assumptions = assumptions
 self.test_count = 0
 def some_func(self, adder=1):
 self.test_count += adder
 time.sleep(0.1)
# - initialise a list of classes - #
item_ids = ['A', 'B', 'C', 'D']
items = [sub_class(_id) for _id in item_ids]
print( sum(_item.test_count for _item in items) )
# Prints out 0
# - invoke a class level function - #
[_item.some_func(adder=2) for _item in items]
print( sum(_item.test_count for _item in items) )
# Prints out 8

• stovfl: Does this answer your question? python Pool with worker Processes

Answer 1

This works with only a minor change to some_func().

import time
from concurrent.futures import ProcessPoolExecutor, wait, ALL_COMPLETED
# - Bespoke Class that is standalone - #
class MyClass:
 def __init__(self, item_id, assumptions="BlahBlahBlah"):
 self.item_id = item_id
 self.assumptions = assumptions
 self.test_count = 0
 def some_func(self, adder=1):
 self.test_count += adder
 print(f"{self.item_id} executing some_func()")
 time.sleep(0.1)
 # note: we must return ourselves
 return self
# Don't omit this on Windows
if __name__ == '__main__':
 # - initialise a list of classes - #
 item_ids = ['A', 'B', 'C', 'D']
 items = [MyClass(item_id) for item_id in item_ids]
 print(sum(item.test_count for item in items))
 # Prints out 0
 # - invoke a class level function - #
 with ProcessPoolExecutor(max_workers=4) as executor:
 future_list = []
 for item in items:
 future_list.append(executor.submit(item.some_func, adder=2))
 wait(future_list, return_when=ALL_COMPLETED)
 # note: we must use the returned self to get the test count
 print(sum(future.result().test_count for future in future_list))
 # Prints out 8
 # note: the original items haven't changed
 print(sum(item.test_count for item in items))
 # Prints out 0

The "items" get copied to the new Process so you have to use the future return() values instead. Therefore, you can replace the original items using items = [future.result() for future in future_list] at some point after the wait().

• user3112131: This works perfectly, exactly what I was looking for. ( Ive add a little progress bar and it work like a charm)
• user3112131: Perhaps, I shouldn't have been as hasty: how would you store the new outputs? e.g. Doing the below prints 0 tmp_list = [future.result().test_count for future in future_list] print(sum(tmp_list))
• FiddleStix: Are you sure? It prints 8 for me. See also my answer edit.
• user3112131: Apologies, this works; I had adapted a few things and left out the "return self" bit in MyClass.some_func (despite your comment not to forget it). thanks a lot