homepage

I experience a problem with multiprocessing and print.
I tried to make a minimal working example, please see the attached file.
WITHOUT the offending print statement in the queue filler thread, everything works:
- pytest experiments/mp_problem.py
- pytest experiments/mp_problem.py -s
- python experiments/mp_problem.py
WITH the offending print statement, not so much:
- pytest experiments/mp_problem.py WORKS (Probably because pytest captures fd 1)
- pytest experiments/mp_problem.py -s FAILS eventually (after a couple of workers have been started)
- python experiments/mp_problem.py FAILS eventually (same).
WITH the offending print statement AND PYTHONUNBUFFERED=1, everything works again:
- pytest experiments/mp_problem.py
- pytest experiments/mp_problem.py -s
- python experiments/mp_problem.py
Environment:
Ubuntu 18.04.5 LTS
python 3.8.5 (hcff3b4d_1) on conda 4.8.3

python experiments/mp_problem.py also fails for:
- 3.8.3, 3.8.2, 3.8.1, 3.8.0
- 3.7.7
- 3.6.10

Try putting a lock on the print statement. See this: https://stackoverflow.com/questions/40356200/python-printing-in-multiple-threads/ 

While I appreciate your suggestion, it does not help me much.
The problem that people usually have is that the output is scrambled. That is not the problem I'm dealing with.
I'm experiencing a deadlock caused by the print statement which seems like a python bug to me.
Furthermore, the problem appears in a library that is supposed to be used by other people and I have no control over their use of IO.
The particular behavior seems to be specific to using threading and multiprocessing together:
- If I use multiprocessing.dummy (multiprocessing API implemented with threads; so only a single processes are involved), my example works fine.
- If I use a process instead of a thread to fill the queue (filler = multiprocessing.Process(...); so no threads are involved), my example also works fine.
- Only if I have two threads in the main process and additional processes, the example fails.

You're right, this is a different issue. I debugged it a bit and I think the race may be between your print statement and the util._flush_std_streams() in Popen.__init__() of Lib/multiprocessing/popen_fork.py.

This may be relevant: https://stackoverflow.com/questions/9337711/subprocess-popen-not-thread-safe
It points to print() being not thread safe and suggests to use sys.stdout.write instead. That worked for me with your script.

Is there anything left here?
I've seen it mentioned on other issues of this sort that mixing multiprocessing and threading leads to problems. Should we document that?

Yes, I think it should at least be documented.
But then it practically says: "Do not use print in your library because it might be used in a threading context" This sounds unacceptable to me. 
It would be great to "just make it work".
> I debugged it a bit and I think the race may be between your print statement and the util._flush_std_streams()
Why would print deadlock with a flush?

This is just issue6721 again.
The workaround is easy: just add `multiprocessing.set_start_method("forkserver")` at the start of your program.
Also, this is more or less documented, though quite tersely:
"""Note that safely forking a multithreaded process is problematic.""
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods 

Thanks for the pointer, @pitrou!