homepage

test_multiprocessing_fork fails only if run all tests on FreeBSD. It is passed successfully if run it separately.
$ ./python -m test -vuall
...
======================================================================
FAIL: test_many_processes (test.test_multiprocessing_fork.WithProcessesTestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/serhiy/py/cpython3.7-debug/Lib/test/_test_multiprocessing.py", line 505, in test_many_processes
 self.assertEqual(p.exitcode, -signal.SIGTERM)
AssertionError: 1 != -15
======================================================================
FAIL: test_terminate (test.test_multiprocessing_fork.WithProcessesTestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/serhiy/py/cpython3.7-debug/Lib/test/_test_multiprocessing.py", line 360, in test_terminate
 self.assertEqual(exitcode, -signal.SIGTERM)
AssertionError: 1 != -15
----------------------------------------------------------------------
test_subprocess hangs, perhaps due to the same cause.

test_many_processes() was added by bpo-30589. Antoine Pitrou commented:
"In the end, I'm glad I added a stress test (test_many_processes) as part of this issue.".
I confirm that test_multiprocessing_fork fails with "./python -m test -vuall" on FreeBSD CURRENT (I tested on Koobs's buildbot worker). I'm currently trying to bisect the issue. It's not easy since test_curses does randomly crash and running +200 tests sequentially is slow.

Latest change in test_many_processes(): commit e6cfdefa0c2f5bda177e49b228c2c7528f7c239c of bpo-31510 which was specific to macOS.

See also commit c08177a1ccad2ed0d50898c2731b518c631aed14 of bpo-30703.

Running these two tests is enough to reproduce the issue:
test.test_curses.TestCurses.test_new_curses_panel
test.test_multiprocessing_fork.WithProcessesTestProcess.test_many_processes
Command:
CURRENT-amd64% ./python -m test -vuall test_curses test_multiprocessing_fork -m test.test_curses.TestCurses.test_new_curses_panel -m test.test_multiprocessing_fork.WithProcessesTestProcess.test_many_processes

The bug can be reproduced with attached bug.py.
With "curses.initscr(); curses.endwin()", the exit code becomes 1.
Without, the child processes exits with code -15 (-SIGTERM).

I have tracked the issue down to the call inside the call to initscr in _cursesmodule.c. The issue *seems* related to the fact that all processes are sharing the same initialization of the curses internal structures, that (probably) is also related to an error that happens when calling initscr multiple times. Notice that:
def sleep_some():
 curses.initscr()
 curses.endwin()
 time.sleep(100)
(without calling initscr in the parent)
yields the correct results:
-15 -15
-15 -15
-15 -15
while doing the same but interleaving one all to initscr after the first fork:
for p in procs:
 p.start()
 curses.initscr()
yields correct results for the first child but incorrect for the rest:
-15 -15
1 -15
1 -15
It seems that forking (because spawn or forkserver always works) after doing the call to initscr does something to the child that makes it unable to handle/receive SIGTERM. The exit statuses of this last round are:
[pid 27105] +++ killed by SIGTERM +++
[pid 27104] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=27105, si_uid=1000, si_status=SIGTERM, si_utime=0, si_stime=0} ---
[pid 27106] +++ exited with 1 +++
[pid 27104] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27106, si_uid=1000, si_status=1, si_utime=0, si_stime=0} ---
[pid 27107] +++ exited with 1 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27107, si_uid=1000, si_status=1, si_utime=0, si_stime=0} ---

Also, it seems that calling initscr registers signal handlers for SIGTERM:
Without calling initscr:
...
rt_sigaction(SIGINT, {sa_handler=0x55d9351a9155, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d22b36da0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d22b36da0}, {sa_handler=0x55d9351a9155, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d22b36da0}, 8) = 0
sigaltstack(NULL, {ss_sp=0x55d93743a1c0, ss_flags=0, ss_size=8192}) = 0
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}, NULL) = 0
...
Calling initscr:
...
 rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGTERM, {sa_handler=0x7f28d6affed0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f28d7c8eda0}, NULL, 8) = 0
 rt_sigaction(SIGWINCH, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGWINCH, {sa_handler=0x7f1046394ec0, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1047523da0}, NULL, 8) = 0
...

Pablo Galindo Salgado:
> Also, it seems that calling initscr registers signal handlers for SIGTERM: (...)
The problem is that endwin() doesn't restore the old handler.
Attached PR 4183 fixes the bug, it saves/restores signal handlers:
./python -m test -vuall test_curses test_multiprocessing_fork -m test.test_curses.TestCurses.test_new_curses_panel -m test.test_multiprocessing_fork.WithProcessesTestProcess.test_many_processes

Thank you Victor and Pablo! Great work!

New changeset 19f68301a1295a9c30d9f28b8f1479cdcccd75aa by Victor Stinner in branch 'master':
bpo-31629: Add support.SaveSignals (#4183)
https://github.com/python/cpython/commit/19f68301a1295a9c30d9f28b8f1479cdcccd75aa

New changeset 1d481822a6295e6739da2d5cca0fdbede51fda22 by Victor Stinner in branch '2.7':
bpo-31629: Add support.SaveSignals (#4183) (#4188)
https://github.com/python/cpython/commit/1d481822a6295e6739da2d5cca0fdbede51fda22

New changeset 41efc402f154e48e95dde2993901648edcb24069 by Victor Stinner in branch '3.6':
bpo-31629: Add support.SaveSignals (#4183) (#4187)
https://github.com/python/cpython/commit/41efc402f154e48e95dde2993901648edcb24069

Thanks Serhiy Storchaka for the bug report and Pablo Galindo Salgado for the analysis!