homepage

6d6099f7fe89 introduced a regression.
It causes the following code to hang fairly reliably for me:
"""
import concurrent.futures
import math
def is_prime(n):
 print(sqt(81)) <---- Note the type error!
def main():
 with concurrent.futures.ProcessPoolExecutor(2) as executor:
 executor.map(is_prime, [1, 2])
if __name__ == '__main__':
 main()
"""
From my limited knowledge of multiprocessing and concurrent futures, it seems that in shutdown_worker(), call_queue.put(None) hangs because the queue is full and there are no more workers consuming items in the queue (they exited early).
Increasing EXTRA_QUEUED_CALLS fixes the problem, though its probably not the correct solution.

Further analysis seems to indicate that it is a race between the shutdown code and the processes finishing.
This code hangs:
"""
executor = concurrent.futures.ProcessPoolExecutor(2)
executor.map(is_prime, [1, 2])
executor.shutdown()
"""
while this doesn't:
"""
executor = concurrent.futures.ProcessPoolExecutor(2)
executor.map(is_prime, [1, 2])
time.sleep(1)
executor.shutdown()
"""

This patch should fix the issue. Can you confirm?

Yes, the patch does appear to fix the issue.
Thanks

New changeset 51c1f2cedb96 by Antoine Pitrou in branch 'default':
Issue #12456: fix a possible hang on shutdown of a concurrent.futures.ProcessPoolExecutor.
http://hg.python.org/cpython/rev/51c1f2cedb96 

Looks like I broke the OS X buildbots:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%203.x/builds/609/steps/test/logs/stdio 

Ok, qsize() can raise NotImplementedError on OS X, which explains quite a bit:
Exception in thread Thread-2:
Traceback (most recent call last):
 File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/threading.py", line 737, in _bootstrap_inner
 self.run()
 File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/threading.py", line 690, in run
 self._target(*self._args, **self._kwargs)
 File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/concurrent/futures/process.py", line 271, in _queue_management_worker
 if not pending_work_items and call_queue.qsize() == 0:
 File "/Users/buildbot/buildarea/custom.parc-snowleopard-1/build/Lib/multiprocessing/queues.py", line 139, in qsize
 return self._maxsize - self._sem._semlock._get_value()
NotImplementedError

New changeset ce52310f61a0 by Antoine Pitrou in branch 'default':
Followup to 51c1f2cedb96 (and issue #12456):
http://hg.python.org/cpython/rev/ce52310f61a0