homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Multiprocessing spawn/forkserver fails to pass Queues
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Sean Murphy, davin, kumaar.nd, mouse07410, pitrou
Priority: normal Keywords:

Created on 2016年12月14日 00:51 by Sean Murphy, last changed 2022年04月11日 14:58 by admin. This issue is now closed.

Messages (6)
msg283150 - (view) Author: Sean Murphy (Sean Murphy) Date: 2016年12月14日 00:51
Python fails to pass a Queue when calling Process with multiprocessing.set_start_method set to "spawn" or "forkserver".
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
 exitcode = _main(fd)
 File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
 self = pickle.load(from_parent)
 File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__
 self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Here is a minimized example:
```
#!/usr/bin/env python3
import multiprocessing
def check_child(q):
 print("Queue", q)
if __name__ == '__main__':
 multiprocessing.set_start_method('spawn')
 # multiprocessing.set_start_method('fork')
 # multiprocessing.set_start_method('forkserver')
 q = multiprocessing.Queue(-1)
 print("q", q)
 proc = multiprocessing.Process(target=check_child, args=(q,))
 proc.start()
```
Also, this fails when the Queue is implicitly passed to the child.
```
class Blerg():
 def __init__(self):
 self.q = multiprocessing.Queue(-1)
 def print_queue(self):
 print("Queue", self.q)
if __name__ == '__main__':
 multiprocessing.set_start_method('spawn')
 blerg = Blerg()
 blerg.print_queue()
 proc = multiprocessing.Process(target=blerg.print_queue)
 proc.start()
```
$ python3 --version
Python 3.5.2
Windows (which defaults to "spawn" style multiprocessing) does not seem to have this issue (at least in 2.7.12).
msg283305 - (view) Author: kumaar.nd (kumaar.nd) Date: 2016年12月15日 13:58
hi,
few comments on this: [when tested with 'spawn']
1. the document(3x, 17.2.1.2) clearly mentions that 'spawn' doesnt inherit the file-handles (hence FileNotFoundError).
2. the document's examples has spawn() followed by join() where parent waits for the Child to finish and exits gracefully. The error "OSError: [Errno 9] Bad file descriptor" is seen for Child process.
3. Isnt exception-handling for null file descriptors sufficient (with corresponding Document updates)
am not a core developer yet and still trying to learn and contribute.
Thanks
msg298871 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017年07月22日 21:21
Ok, this is really an error in your script. If you add `p.join()` to wait for the process to end, you'll see no error pops up. What really happens is that the parent process ends before the child is fully started, and the resource underlying the queue is cleaned up at this point.
msg298872 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017年07月22日 21:25
More exactly, since the child process is not "daemonic", the parent process waits for it to end, but after collecting the queue's resources.
msg298873 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017年07月22日 21:26
So, while it might be possible to improve on this, it would risk fragilizing other use cases, and besides, this is not a problem that is likely to happen in real usage. Therefore, I'd rather close this issue.
msg365280 - (view) Author: Mouse (mouse07410) Date: 2020年03月29日 21:52
On MacOS Catalina 10.15.4, I still see this problem occasionally even with p.join() added. See https://bugs.python.org/msg365251 and subsequent messages.
Also, see https://bugs.python.org/issue40106.
History
Date User Action Args
2022年04月11日 14:58:40adminsetgithub: 73151
2020年03月29日 21:52:57mouse07410setnosy: + mouse07410
messages: + msg365280
2017年07月22日 21:26:54pitrousetstatus: open -> closed
resolution: wont fix
messages: + msg298873

stage: resolved
2017年07月22日 21:25:29pitrousetmessages: + msg298872
2017年07月22日 21:21:47pitrousetnosy: + pitrou
messages: + msg298871
2016年12月15日 13:58:20kumaar.ndsetnosy: + kumaar.nd
messages: + msg283305
2016年12月14日 16:44:03davinsetnosy: + davin
type: crash -> behavior
2016年12月14日 00:51:48Sean Murphycreate

AltStyle によって変換されたページ (->オリジナル) /