homepage

Merely importing tkinter breaks the use of parallel code on my system (Mac OSX 10.11.6, tested on Python 2.7.13 / 2.7.14 / 3.5.0 / 3.6.4, all barebones distributions installed with pyenv). I've tested this with both multiprocessing and sharedmem (see minimal scripts below).
The issue seems to apply only to functions that evoke multithreading within their respective package (e.g. `numpy.matmul()`, `cv2.SIFT.detectAndCompute()`). If I make the matrix in the scripts below much smaller (e.g. change `5000` to `5`), avoiding internal multithreading, the scripts work.
## with `multiprocessing`
```python
import numpy as np
import multiprocessing
import _tkinter
def parallel_matmul(x):
 R = np.random.randn(3, 3)
 return np.matmul(R, x)
pool = multiprocessing.Pool(4)
results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```
> *Code never exits and Python has to be force quit*
## with `sharedmem`
```python
import numpy as np
import sharedmem
import _tkinter
def parallel_matmul(x):
 R = np.random.randn(3, 3)
 return np.matmul(R, x)
with sharedmem.MapReduce() as pool:
 results = pool.map(parallel_matmul,
 [np.random.randn(3, 5000) for i in range(2)])
```
> sharedmem.sharedmem.SlaveException: slave process 1 killed by signal 11

Are you by any chance trying to run this under IDLE?

I'm fairly sure this is expected behavior on macOS: _tkinter loads the Tk library, which loads Apple GUI frameworks. Those frameworks are not save w.r.t. fork(2) and that can lead to all kinds of unwanted behavior (although I would have expected a more explicit crash than this)

I have tried running the script with:
- command line (python <file>): Works without (breaks with) `import _tkinter`.
- basic python console (python): Same as with command line.
- ipython: Fails with or without `import _tkinter`. However, if for example I make the matrix in the scripts smaller or use only functions that don't resort to multithreading, code runs successfully in parallel in ipython.
- IDLE: Perhaps unsurprisingly, the code fails with or without `import _tkinter` (never terminates, or prompts endless "Your program is still running! Do you want to kill it?")
Suspecting a problem with Tcl/Tk (Apple's original 8.5.9), I rebuilt python pointing to tcl-tk installed with brew (8.6.8), which I checked with IDLE and _tkinter.TCL_VERSION() / _tkinter.TK_VERSION(). However, this did not fix the problem.

I've tried with additional backends: WX, WXAgg, WXCairo, Qt5Agg (in matplotlib speak).
With these, I can at least import matplotlib.pyplot, but as soon as say matplotlib.pyplot.plot is called and the backend is loaded, the code breaks (error message for WXAgg below). Is multiprocessing in an interactive shell simply not meant to be supported on MacOS?
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all
 manager.destroy()
 File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py", line 1303, in destroy
 self.frame.Destroy()
 File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py", line 1256, in Destroy
 if not self.IsBeingDeleted():
RuntimeError: wrapped C/C++ object of type FigureFrameWxAgg has been deleted

I cannot reproduce the problem on macOS 10.13.3 with python 3.6 (but haven't spent much time on this).
The combination of GUI code with multiprocessing is a known source of problems on macOS, the system libraries are hostile to that.
Changing the start method from "fork" to "spawn" or "forkserver" should help with this, using multiprocessing.set_start_method(). This should be changed before doing anything that might Apple system frameworks related to GUIs.

The examples in the multiprocessing doc all put all multiprocessing calls within a "if __name__ == '__main__':" statement. I know that this is necessary on Windows, but don't know if or when it helps on other OSes.

Terry Reedy: I just tried your suggestion of a main clause (code below) and it made no difference.
```
import _tkinter
import numpy as np
# multiprocessing.set_start_method("spawn")
import multiprocessing
def parallel_matmul(x):
 R = np.random.randn(3, 3)
 return np.matmul(R, x)
if __name__ == '__main__':
 pool = multiprocessing.Pool(4)
 results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```
Ronald Oussoren: Good to know that perhaps upgrading to the latest OSX would resolve this issue. I tried 'spawn' and 'forkserver' with multiprocessing.set_start_method() (see above) but both result in an endless stream of errors (with or without importing a graphics backend), although that may be my not knowing how to properly use them.

@ezwelty: The import of multiprocessing and the call to set_start_method should be at the very start of the code, before other imports, to avoid the annoying Apple behavior I mentioned.

@ronaldoussoren: The order of the imports made no difference. Even with the call at the top, I got endless errors with both 'spawn' and 'forkserver', with or without importing a graphics backend. Only 'fork' works, and only if a graphics package is not imported.
** HOWEVER **
Setting method='spawn' or 'forkserver' and force=True at top, or calling multiprocessing.set_start_method in __main__ does work (when run as scripts from command line). It's a shame to have to give up the convenience of 'fork', but this is a start.
```
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
import numpy as np
# import _tkinter
def parallel_matmul(x):
 R = np.random.randn(3, 3)
 return np.matmul(R, x)
if __name__ == '__main__':
 # multiprocessing.set_start_method("spawn", force=False)
 pool = multiprocessing.Pool(4)
 results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```

On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8. The same is true for the 3.6.6 64-bit installer. Do tkinter and multiprocessing work together better with these installations?
I want to consider using multiprocessing and pipes instead subprocess and socket for IDLE's user-code execution process, started from and communicating with the initial tkinter gui process. idlelib.run, which runs in the execution process to communicae with the gui process and supervise running user code, imports tkinter. Besides which, users have to be able to import tkinter in their programs.

As I mentioned in msg314304 this is almost certainly expected behavior on macOS because Apple’s GUI frameworks are not fork()-safe. There is nothing we can do about this other than changing the default spawn mechanism for multiprocessing on macOS. 
--
On the road, hence brief. 
Op 14 jul. 2018 om 19:30 heeft Terry J. Reedy <report@bugs.python.org> het volgende geschreven:
> 
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
> 
> On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8. The same is true for the 3.6.6 64-bit installer. Do tkinter and multiprocessing work together better with these installations?
> 
> I want to consider using multiprocessing and pipes instead subprocess and socket for IDLE's user-code execution process, started from and communicating with the initial tkinter gui process. idlelib.run, which runs in the execution process to communicae with the gui process and supervise running user code, imports tkinter. Besides which, users have to be able to import tkinter in their programs.
> 
> ----------
> nosy: +pitrou
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33111>
> _______________________________________

I was uncertain from Ethan's post what does work best. "endless errors with both 'spawn", "Only 'fork' works...but without graphics", "method='spawn'and ... does work". Do the 1st and 3rd sentences mean that spawn + graphics fails without the proper additional code but does work with it? Should the 2nd sentence be "fork only works without graphics" rather than starting with 'only fork'?
If so, this would suggest that the default should depend on whether there is a graphic import ;-). In any case, how to use successfully use multiprocessing with graphics on Mac should be documented.

macOS users, feel free to propose a doc PR for multiprocessing.

I can provide a patch, but that will likely be during EuroPython as I’m currently taking a roundabout route towards Edinburgh. 
Btw. I’m not sure yet how impactful changing the default spawning mechanism would be for 3.8. How likely is that to break user code? 
Op 16 jul. 2018 om 10:24 heeft Antoine Pitrou <report@bugs.python.org> het volgende geschreven:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> macOS users, feel free to propose a doc PR for multiprocessing.
> 
> ----------
> assignee: -> docs@python
> components: +Documentation
> nosy: +docs@python
> versions: +Python 3.7, Python 3.8 -Python 3.5
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33111>
> _______________________________________

It will definitely break *some* user code. Also, everyone not using tkinter (which I think is the majority of users) isn't affected by this.

Ronald Oussoren> I can provide a patch, but that will likely be during 
EuroPython as I’m currently taking a roundabout route towards Edinburgh.
Can you add a test of the procedure you document for using _tkinter with 
multiprocessing?

That's annoying, I cannot reproduce the issue on macOS 10.13. I don't have access to my test VM running 10.11 at the moment, I'll work on documenting this when I get back home (and do have access to 10.11 again).
BTW. I don't know if providing a reliable test is possible other than testing that the documented procedure works in general.

In Python 3.8, the default start method for multiprocessing when run on macOS is now "spawn" instead of "fork".
"Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725."