homepage

test_multiprocessing segfaults in a loop. The crash occurs in _Condition.release() on waiter.release(), called from Queue._finalize_close(). Possible related changes:
 - a5c8b6ebe895: new sigwait() test using thread (issue #8407)
 - 6d6099f7fe89: add sentinels to multiprocessing (issue #9205)
Example of a crash:
[333/356/1] test_multiprocessing
Fatal Python error: Segmentation fault
Thread 0x01cde800:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/threading.py", line 237 in wait
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/queues.py", line 252 in _feed
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/threading.py", line 690 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/threading.py", line 737 in _bootstrap_inner
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/threading.py", line 710 in _bootstrap
Current thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/threading.py", line 298 in notify
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/queues.py", line 227 in _finalize_close
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/util.py", line 202 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/process.py", line 270 in _bootstrap
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/forking.py", line 133 in __init__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/process.py", line 134 in start
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/pool.py", line 220 in _repopulate_pool
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/pool.py", line 157 in __init__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/__init__.py", line 231 in Pool
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_multiprocessing.py", line 2186 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
There are approximatively 698 crashes in last tests on "x86 Tiger 3.x"!
http://www.python.org/dev/buildbot/all/builders/x86%20Tiger%203.x/builds/2722/steps/test/logs/stdio
Most occured on Queue._finalize_close() -> _Condition.release() -> waiter.release().

> a5c8b6ebe895: new sigwait() test using thread (issue #8407)
> 6d6099f7fe89: add sentinels to multiprocessing (issue #9205)
The first segfaults occured in build #2719 (cedceeb45030):
http://www.python.org/dev/buildbot/all/builders/x86%20Tiger%203.x/builds/2719
In the timeline, 6d6099f7fe89 < cedceeb45030 < a5c8b6ebe895, so the change is more likely coming from 6d6099f7fe89.

Victor, how can there be hundreds of crashes? Isn't the process supposed to terminate when a crash occurs?
There are several crashes in test_signal, so it's not only test_multiprocessing:
Thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 172 in test_main
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 407 in _executeTestPart
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 462 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 514 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1166 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1254 in _run_suite
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1280 in run_unittest
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 687 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
Fatal Python error: Segmentation fault
Thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 248 in test_wakeup_fd_early
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 407 in _executeTestPart
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 462 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 514 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1166 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1254 in _run_suite
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1280 in run_unittest
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 687 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
Fatal Python error: Segmentation fault
Thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 372 in readpipe_interrupted
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 394 in test_siginterrupt_on
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 407 in _executeTestPart
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 462 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 514 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1166 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1254 in _run_suite
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1280 in run_unittest
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 687 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
Fatal Python error: Segmentation fault
Thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 495 in test_itimer_prof
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 407 in _executeTestPart
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 462 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 514 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1166 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1254 in _run_suite
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1280 in run_unittest
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 687 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
Fatal Python error: Segmentation fault
Thread 0xa000d000:
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 472 in test_itimer_virtual
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 407 in _executeTestPart
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 462 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 514 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 105 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/suite.py", line 67 in __call__
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1166 in run
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1254 in _run_suite
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1280 in run_unittest
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_signal.py", line 687 in test_main
 File "./Lib/test/regrtest.py", line 1043 in runtest_inner
 File "./Lib/test/regrtest.py", line 841 in runtest
 File "./Lib/test/regrtest.py", line 668 in main
 File "./Lib/test/regrtest.py", line 1618 in <module>
Fatal Python error: Segmentation fault

> Victor, how can there be hundreds of crashes?
> Isn't the process supposed to terminate when a crash occurs?
Yes, a process does terminate on SIGSEGV, but multiprocessing creates subprocesses: I suppose that crashes occur in child processes.
For test_signal, I have to investigate this.

Le vendredi 10 juin 2011 à 09:40 +0000, Antoine Pitrou a écrit :
> There are several crashes in test_signal
Commit a17710e27ea2 should fix some (all?) test_signal crashes.

It looks like the sentinel doesn't handle fatal death of the child process:
test test_multiprocessing crashed -- Traceback (most recent call last):
 File "./Lib/test/regrtest.py", line 1043, in runtest_inner
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_multiprocessing.py", line 2189, in test_main
 ManagerMixin.manager.start()
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/managers.py", line 531, in start
 self._address = reader.recv()
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/connection.py", line 273, in recv
 buf = self._recv_bytes(sentinels=sentinels)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/connection.py", line 430, in _recv_bytes
 buf = self._recv(4, sentinels)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/multiprocessing/connection.py", line 413, in _recv
 raise EOFError
EOFError

I think it might be related to Issue #6721.
Using a mutex/condition variable after fork (from the child process) is unsafe: it can lead to deadlocks, and on OS-X, it seems like it can lead to segfaults.
Normally, Queue's synchronization primitives are reinitialized after fork, see Queue._after_fork() method.
But here, what happens is the following (well, that's an hypothesis):
Lib/multiprocessing/process.py", line 270 in _bootstrap
:
 _current_process = self
 util._finalizer_registry.clear()
 util._run_after_forkers()
- the old _current_process' reference count drops to zero
- it's deallocated, and since it holds the last reference to a Queue, the queue is finalized
- as a consequence, the Queue._finalize_close() callback (registered through a Finalize object) is called
- Queue._finalize_close() tries to acquire, signal and release the _notempty Condition, but the Condition hasn't been reinitialized yet, because util._run_after_forkers() is called 2 lines below
- segfault
It's probably been triggered by Antoine's patches, but I'm pretty sure this bug has always been there.
I think that moving util._run_after_forkers() up 2 lines should fix the segfaults, but with that change test_number_of_objects fails (I didn't investigate why).

Less disruptive approach:
 old_process = _current_process
 _current_process = self
 try:
 util._finalizer_registry.clear()
 util._run_after_forkers()
 finally:
 del old_process
This will delay finalization of the old process object until after _run_after_forkers() is executed, without (hopefully) messing with semantics.

Yes, I also tried this.
This fixed the test_multiprocessing failure, but I think it triggered a failure in test_concurrent_futures (didin't look in more detail).
Would it be possible to try this on the buildbot to see if it fixes the segfaults?

> Would it be possible to try this on the buildbot to see if it fixes
> the segfaults?
You can fork cpython, modify the code, and run a custom buildbot on your
clone.

New changeset e6e7e42efdc2 by Victor Stinner in branch '3.2':
Issue #12310: finalize the old process after _run_after_forkers()
http://hg.python.org/cpython/rev/e6e7e42efdc2
New changeset a73e5c1f57d7 by Victor Stinner in branch 'default':
(Merge 3.2) Issue #12310: finalize the old process after _run_after_forkers()
http://hg.python.org/cpython/rev/a73e5c1f57d7 

Let's try on "real" buildbots. If the commit fixes the issue on 3.x, I will port the fix to Python 2.7.

test_multiprocessing pass with success on PPC Tiger 3.x (and x86 Tiger 3.x, but the segfaults only occurred on PPC), but this issue is a sporadic issue. I close the issue because I hope that it is closed, but reopen it if you still see segfaults on PPC Tiger 3.x.