homepage

In the build --without-threads, catching an ImportError in support.py
fails. Fedora buildbot:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/1986/steps/test/logs/stdio
./python ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600 
Traceback (most recent call last):
 File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 160, in _run_module_as_main
 "__main__", fname, loader, pkg_name)
 File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 73, in _run_code
 exec(code, run_globals)
 File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/__main__.py", line 1, in <module>
 from test import regrtest, support
 File "<frozen importlib._bootstrap>", line 1038, in _handle_fromlist
 File "<frozen importlib._bootstrap>", line 977, in _find_and_load
 File "<frozen importlib._bootstrap>", line 561, in load_module
 File "<frozen importlib._bootstrap>", line 218, in module_for_loader_wrapper
 File "<frozen importlib._bootstrap>", line 446, in _load_module
 File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/regrtest.py", line 243, in <module>
 from test import support
 File "<frozen importlib._bootstrap>", line 1038, in _handle_fromlist
 File "<frozen importlib._bootstrap>", line 977, in _find_and_load
 File "<frozen importlib._bootstrap>", line 561, in load_module
 File "<frozen importlib._bootstrap>", line 218, in module_for_loader_wrapper
 File "<frozen importlib._bootstrap>", line 446, in _load_module
 File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/support.py", line 34, in <module>
 import multiprocessing.process
SystemError: NULL result without error in PyObject_Call
[110893 refs]
make: *** [buildbottest] Error 255

The SystemError has changed to a KeyError:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/2004/steps/test/logs/stdio
At least that makes it easy to spot the location in import.c. The
shortest way to reproduce the error is to compile --without-threads,
then:
import multiprocessing
import multiprocessing.process

Just to clarify the failure for the bug history, somehow multiprocessing is not ending up in sys.modules as expected. It changed from a SystemError to a KeyError because I started to properly check a PyDict_GetItem() return value instead of blindly assuming something was going to be in sys.modules.

This issue is now apparently causing a segfault:
(gdb) r ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600
Starting program: /home/stefan/pydev/cpython/python ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600
[...]
Program received signal SIGSEGV, Segmentation fault.
0x000000000041b038 in PyObject_Repr (v=<unknown at remote 0x7ffff43b7660>) at Objects/object.c:377
377 if (Py_TYPE(v)->tp_repr == NULL)
(gdb) l
[...]
#0 0x000000000041b038 in PyObject_Repr (v=<unknown at remote 0x7ffff43b7660>) at Objects/object.c:377
#1 0x0000000000460db0 in PyUnicode_FromFormatV (format=0x63e3e8 "%R not in sys.modules as expected", vargs=
 0x7fffffff1ae0) at Objects/unicodeobject.c:2609
#2 0x00000000004e1be2 in PyErr_Format (exception=<type at remote 0x8e68c0>, format=
 0x63e3e8 "%R not in sys.modules as expected") at Python/errors.c:697
#3 0x00000000004ed53f in PyImport_ImportModuleLevelObject (name='multiprocessing.process

Here is a patch. The __import__ function's crazy API never ceases to amaze me.

Oops, there was a duplicate test.

Only two comments, otherwise LGTM (and I can't believe the solution was to go back through the import system just to pull out the cached module; the things we would change if we were doing this from scratch).
One, you have some "XXX False" markers in the tests. Should those get deleted or replaced with something?
Two, in your first test (at least) you only test what is in sys.modules once instead of after each attempted import. I would repeat the test after each import.

> One, you have some "XXX False" markers in the tests. Should those get
> deleted or replaced with something?
Well, I don't know what to replace them with. I would have expected
pkg.module to end up in sys.modules, but as mentioned in the comments
the relative import line actually fails with an ImportError. There's
probably some logic behind that, but that's beyond me.
> Two, in your first test (at least) you only test what is in
> sys.modules once instead of after each attempted import. I would
> repeat the test after each import.
Ok, will do.

> > One, you have some "XXX False" markers in the tests. Should those get
> > deleted or replaced with something?
> 
> Well, I don't know what to replace them with. I would have expected
> pkg.module to end up in sys.modules, but as mentioned in the comments
> the relative import line actually fails with an ImportError. There's
> probably some logic behind that, but that's beyond me.
I should mention that the same happens with 2.7 and 3.2, so it's not a
regression. It's just that I don't understand the logic :-)

So I was going to try to figure out the logic, so I manually created the test files to start debugging, but I didn't get the ImportError but instead the 1/0 error for the relative import. Maybe it's specific to lack of threads or the change you made? I mean if that's how it has always worked then I'm not arguing that it's wrong, just that it's a weird side-effect:
>>> import pkg
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<frozen importlib._bootstrap>", line 977, in _find_and_load
 File "<frozen importlib._bootstrap>", line 596, in load_module
 File "<frozen importlib._bootstrap>", line 262, in module_for_loader_wrapper
 File "<frozen importlib._bootstrap>", line 484, in _load_module
 File "./pkg/__init__.py", line 3, in <module>
 1/0
ZeroDivisionError: division by zero
[70552 refs]
>>> import sys
[70554 refs]
>>> sys.modules['pkg']
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
KeyError: 'pkg'
[70462 refs]
>>> sys.modules['pkg.module']
<module 'pkg.module' from './pkg/module.py'>
[70465 refs]
>>> with open('pkg/__init__.py') as file: print(file.read())
... 
from . import module
#import pkg.module
1/0

> So I was going to try to figure out the logic, so I manually created
> the test files to start debugging, but I didn't get the ImportError
> but instead the 1/0 error for the relative import. Maybe it's specific
> to lack of threads or the change you made? I mean if that's how it has
> always worked then I'm not arguing that it's wrong, just that it's a
> weird side-effect:
The first "import pkg" gets you a ImportError but the second one should
get you a ZeroDivisionError (yes, weird).

New changeset d6324941b739 by Antoine Pitrou in branch 'default':
Issue #14583: Fix importlib bug when a package's __init__.py would first import one of its modules then raise an error.
http://hg.python.org/cpython/rev/d6324941b739 

Should be fixed now. The buildbot has been able to launch the test suite.