homepage

Here's a report from Ismail Donmez (cartman), extracted from the
issue 3419 discussion in an effort to avoid putting multiple
problems under one tracker issue.
[cartman]
With trunk when running test_multiprocessing in a tight loop I saw 
another problem:
test_multiprocessing
Process Process-61:
Traceback (most recent call last):
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-60:
Traceback (most recent call last):
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
Process Process-62:
Traceback (most recent call last):
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", 
line 229, in _bootstrap
 util._run_after_forkers()
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
 util._run_after_forkers()
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
 util._run_after_forkers()
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 
138, in _run_after_forkers
 items = list(_afterfork_registry.items())
 items = list(_afterfork_registry.items())
 File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
 File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
 items = list(_afterfork_registry.items())
 File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items
 for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
 for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration
 for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration

I don't know if this is reproducible in 2.6 but I can reproduce on py3k 
branch, and also thanks for creating the bug!

Mmmh, the problem with the list(X.items()) idiom is that it's
thread-safe only if X.items() is implemented in C. Otherwise X can be
mutated if there is a thread-switch while executing bytecode in X.items().
In weakref.py (line 103), by replacing:
 for key, wr in self.data.items():
with:
 for key, wr in list(self.data.items()):
This particular error should disappear.
But this doesn't say why the dictionary is mutated at all. Does
multiprocessing (or at least that particular test) launch several
threads in a given process? Otherwise there may be something fishy going on.

py3k branch gives another error now, when running test_multiprocessing 
in a tight loop:
test test_multiprocessing failed -- Traceback (most recent call last):
 File "/Users/cartman/Sources/py3k/Lib/test/test_multiprocessing.py", 
line 1163, in test_remote
 queue = manager2.get_queue()
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 641, in temp
 authkey=self._authkey, exposed=exp
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 893, in AutoProxy
 incref=incref)
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 702, in __init__
 self._incref()
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 749, in _incref
 dispatch(conn, None, 'incref', (self._id,))
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 85, in dispatch
 raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
------------------------------------------------------------------------
---
Traceback (most recent call last):
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 187, in handle_request
 result = func(c, *args, **kwds)
 File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", 
line 403, in incref
 self.id_to_refcount[ident] += 1
KeyError: '7389d0'

Ismail, that's the incref bug in issue 3419 

Ah cool, we might be at the end of multiprocessing problems then I guess 
:-)

Le lundi 18 août 2008 à 19:42 +0000, Ismail Donmez a écrit :
> Ismail Donmez <ismail@namtrac.org> added the comment:
> 
> Ah cool, we might be at the end of multiprocessing problems then I guess 
> :-)
Well, not really, it should be diagnosed why the dictionary mutates in
the first place. This dictionary is apparently processed in a child
process just after a fork() occurred, so logically there shouldn't be
several threads running.
Unless processing the dictionary itself can create some new threads?

Actually, it's probably a duplicate of #7105.