homepage

v.3.2a3
If the maxtasksperchild argument is used, the program will just hang after whatever that value is rather than working as expected. Tested in Windows XP 32-bit
test code:
import multiprocessing
def f(x):
 return 0;
if __name__ == '__main__':
 pool = multiprocessing.Pool(processes=2,maxtasksperchild=1);
 results = list();
 for i in range(10):
 results.append(pool.apply_async(f, (i)));
 pool.close();
 pool.join();
 for r in results:
 print(r);
 print("Done");

I can duplicate this using python-2.7-8.fc14.1 on Fedora 14, and using map_async with the pool.

This problem arises because the pool's close method is called before all the tasks have completed. Putting a sleep(1) before pool.close() won't exhibit this lockup.
The root cause is that close makes the workers handler thread exit: since the maxtasksperchild argument is used, workers exit when they've processed their max number of tasks. But since the workers handler thread exited, it doesn't maintain the pool of workers anymore, and thus the remaining tasks are not treated anymore, and the task handler thread waits indefinitely (since it waits until the cache is empty).
The solution is to prevent the worker handler thread from exiting until the cache has been drained (unless the pool is terminated in which case it must exit right away).
Attached is a patch and relevant test.
Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.

> Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.
>
Yes. We should fix those.

Here's an updated patch.
I'll open a separate issue for the thread-safety.

The patch looks good to me, thanks.

New changeset 3465a9b2d25c by Charles-François Natali in branch '2.7':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/3465a9b2d25c
New changeset 52c98a729a71 by Charles-François Natali in branch '3.2':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/52c98a729a71
New changeset c2cdabc44665 by Charles-François Natali in branch 'default':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/c2cdabc44665 

James, thanks for the report!