homepage

One of the buildbots suffered a test_multiprocessing failure, which
itself is not such an unusual thing. But the error that was
subsequently thrown looks like it might point to a place where
importlib's error handling could be improved. The error that was thrown
certainly gives no clear indication of what really caused the problem.
The full buildbot report was:
http://www.python.org/dev/buildbot/all/builders/ia64%20Ubuntu%203.1/builds/93/steps/test/logs/stdio
Here are the tracebacks (the second one being the interesting one):
 Traceback (most recent call last):
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/process.py",
line 233, in _bootstrap
 self.run()
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/process.py",
line 88, in run
 self._target(*self._args, **self._kwargs)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/test/test_multiprocessing.py",
line 1211, in _putter
 queue = manager.get_queue()
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 644, in temp
 token, exp = self._create(typeid, *args, **kwds)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 544, in _create
 id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 79, in dispatch
 kind, result = c.recv()
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/connection.py",
line 408, in recv
 s = self._conn.recv_bytes()
 IOError: [Errno 104] Connection reset by peer
 test test_multiprocessing failed -- Traceback (most recent call last):
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/test/test_multiprocessing.py",
line 1222, in test_rapid_restart
 queue = manager.get_queue()
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 644, in temp
 token, exp = self._create(typeid, *args, **kwds)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 542, in _create
 conn = self._Client(self._address, authkey=self._authkey)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/connection.py",
line 427, in XmlClient
 import xmlrpc.client as xmlrpclib
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 151, in decorated
 return fxn(self, module)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 399, in load_module
 return self._load_module(module)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 324, in _load_module
 code_object = self.get_code(module.__name__)
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 411, in get_code
 pyc_timestamp = marshal._r_long(data[4:8])
 File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/__init__.py",
line 65, in _r_long
 x = int_bytes[0]
 IndexError: index out of range

I will obviously have a look, but why the heck is test_multiprocessing 
calling importlib? Is it meant to do that? Or is this part of the -j flag 
for regrtest?

And it looks like the error was triggered from a malformed .pyc file that 
was truncated. Guess I need to be more careful with that possibility. =)

This was on a buildbot, and regrtest -j isn't used there, but in any
case it wouldn't affect importlib as far as I know. From the traceback
it looks like multiprocessing is just doing an import.

Huh. I wonder how importlib is ending up being the implementation of 
import then? Guess I also need to double-check everything is still 
cleaning itself up properly.

I can't reproduce the failure as running test_multiprocessing and then test_importlib does not show any left over stuff in the interpreter what would lead to importlib being used for __import__. But as the IndexError part is a valid issue I am renaming the issue.

issue #7875 brought up the original failure that caused the creation of this bug again and led to me finally diagnosing the problem.

A quick check of things shows that a totally empty file, a missing timestamp, or no marshalled code triggers the use of source. I still need to check what happens when the magic number is truncated by a byte or so along with the timestamp.
Any screw-up with the marshalled code should still be an error.

Committed to py3k as r78242. Still need to backport.

Backported to 3.1 in r78244.