homepage

multiprocessing.Pool methods map, imap, etc. are said to be able to normally handle exceptions. But it seems that it is true only for synchronous exceptions inside their first func arguments.
When (typically during a long-running parallel map) a user hits ^C, an asynchronous KeyboardInterrupt isn't handled properly and leads to the interpreter hangup. More precisely, children processes become <defunct> (on Linux), so the only way to terminate the whole program is to issue the KILL signal.
As stopping a program with ^C while running potentially long parallel computations is probably quite a common scenario, the interpreter should not hang up in such a case.
I'm using Python 2.6.5 (r265:79063, Mar 23 2010, 04:44:21) [GCC 4.4.3] on linux2. I've also tried to use the current multiprocessing.pool module from the current (2.7) trunk with my 2.6.5 installation, but the bug persists.

Do you have a test case which can reproduce the issue?

Yes, here is my test case.

You might want to take a look here: http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/ 

Yes, I've come up with the same solution by myself, but it cannot cover all the cases of the bug. It works only for cases when ^C is hit during a call to the users' function: http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool/2561809#2561809
If the user is "lucky", he may hit ^C during getting or putting data into the queues in multiprocessing.pool.worker. To reproduce such a case, you may insert `sleep(10)` before `task = get()` or `put((job, i, result))`, for example. I've encountered such cases just by running test examples several times.

Despite of several workarounds available on the Web, the problem persists. Almost any exception that is rised in `worker` function while putting or getting tasks from queues result in Pool hang up. Currently, `worker` is only aware of Exception descendants rised inside of the map function parameter.
I've written a unit test that checks if KeyboardInterrupts are handled normally. The source code may be included in `Lib/test/test_multiprocessing.py`.

Here is a patch that fixes this problem. Basically, it catches all the BaseExceptions that could happen during: a) getting a task from the `inqueue`, b) calling a user function, c) putting a task into the `outqueue`. The exception handler puts the exception to the `outqueue`.
It can be cleanly applied on top of revision 78790.

Any chance of getting this patch applied? Thanks.

This is related to our discussions at #9205 as well (http://bugs.python.org/issue9205), as the final patch there will also fix this issue.

On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one.
If we keep that behavior, the real problem here is that the
result handler hangs if the process that reserved a job is gone, which is going to be handled
by #9205. Should we mark it as a duplicate?

> If we keep that behavior, the real problem here is that the
> result handler hangs if the process that reserved a job is gone, which is going to be handled
> by #9205. Should we mark it as a duplicate?
I would tend to agree with your assessment; we're better served just
gracefully handling everything per 9205

> On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one.
Yes, SystemExit should cancel all the tasks that are currently in the queue. I guess my patch doesn't handle this properly.
> If we keep that behavior, the real problem here is that the result handler hangs if the process that reserved a job is gone, which is going to be handled by #9205. Should we mark it as a duplicate?
Yes, I think that #9205 covers this issue, so #8296 may be marked as a duplicate.

Closing, as Andrey Vlasovskikh has agreed that this is a duplicate of #9205.

Note that #9205 fixed concurrent.futures, but not multiprocessing.Pool which is a different kettle of fish.

Can this issue or #9205 be reopened as this particular instance of the problem doesn't seem to be resolved? I still seem to need the workaround from http://stackoverflow.com/a/1408476