homepage

= Error message =
pickle.dumps and pickle.dump generate the following error:
 File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
 for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration
= Test case = 
The test case is as follows:
 import pickle
 import sys
 import numpy #version 1.8.1
 import scipy.signal #version 0.13.3
 pickle.dumps(numpy.cos)
Sometimes this results in the error
Traceback (most recent call last):
 File "whichmodule_bug.py", line 13, in <module>
 pickle.dumps(numpy.cos)
 File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 61, in _ufunc_reduce
 return _ufunc_reconstruct, (whichmodule(func, name), name)
 File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
 for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration
NOTE: This is not necessarily a problem due to numpy and scipy packages. Explanation follows.
= Relevant code = 
The last version of pickle http://hg.python.org/cpython/file/f6f691ff27b9/Lib/pickle.py contains:
...
def _getattribute(obj, name, allow_qualname=False):
 dotted_path = name.split(".")
 if not allow_qualname and len(dotted_path) > 1:
 raise AttributeError("Can't get qualified attribute {!r} on {!r}; " +
 "use protocols >= 4 to enable support"
 .format(name, obj))
 for subpath in dotted_path:
 if subpath == '<locals>':
 raise AttributeError("Can't get local attribute {!r} on {!r}"
 .format(name, obj))
 try:
 obj = getattr(obj, subpath)
 except AttributeError:
 raise AttributeError("Can't get attribute {!r} on {!r}"
 .format(name, obj))
 return obj
def whichmodule(obj, name, allow_qualname=False):
 """Find the module an object belong to."""
 module_name = getattr(obj, '__module__', None)
 if module_name is not None:
 return module_name
 for module_name, module in sys.modules.items():
 if module_name == '__main__' or module is None:
 continue
 try:
 if _getattribute(module, name, allow_qualname) is obj:
 return module_name
 except AttributeError:
 pass
 return '__main__'
...
= History =
The iteration 
 for module_name, module in sys.modules.items():
in pickle.py was changed in 2007 to
 for name, module in list(sys.modules.items()):
by the BDFL [http://hg.python.org/cpython/rev/0d2dc3611e3b?revcount=800] to "Fix a bizarre error...", then it was reverted by [http://hg.python.org/cpython/rev/992ef855b3ed] in 2013. I hope the BDFL never discover this :-)
= Explanation = 
It seems that while iterating in sys.modules, getattr is called on an instance of scipy.lib.six.MovedModule, which triggers a change in sys.modules (due to an import). This result in the RuntimeError: dictionary changed size during iteration.
Freezing the list, with list(sys.modules.items()) resolves this particular issue.
= Notes on reproducing this error =
Reproducing my test case might be difficult for the following reason.
I installed scipy version 0.13.3 on Kubuntu 14.04 (sudo apt-get install python3-scipy). In the online source code, module six.py version 1.2.0 is included [https://github.com/scipy/scipy/releases/tag/v0.13.3]. However in my system the module six.py that is actually used is 1.5.2, which probably has been installed independently from scipy. 
Finally, the new version of scipy, 0.14.0, doesn't use the class scipy.lib.six.MovedModule which triggers this error. Unfortunately this new version isn't available in Ubuntu 14.04.

A simple patch to freeze the list of sys.modules.

> A simple patch to freeze the list of sys.modules.
Which patch?

diff -r fb3aee1cff59 Lib/pickle.py
--- a/Lib/pickle.py	Wed Aug 27 09:41:05 2014 -0700
+++ b/Lib/pickle.py	Thu Aug 28 10:59:37 2014 +0200
@@ -280,7 +280,7 @@
 module_name = getattr(obj, '__module__', None)
 if module_name is not None:
 return module_name
- for module_name, module in sys.modules.items():
+ for module_name, module in list(sys.modules.items()):
 if module_name == '__main__' or module is None:
 continue
 try:

Duplicate of #21905
Fixed in https://hg.python.org/cpython/rev/86ba3bdfac15