homepage

This ticket will track the implementation for PEP 451 (ModuleSpec). I'll have a patch up in the next couple days.

Here's an initial patch. Some key things left to do:
* unit tests
* documentation
* implement exec_module() for the various importlib loaders.
Once that's squared away there are further things that will be addressed in new tickets (or at least separate patches). This includes:
* remove init_module_attrs() and module_to_load()
* deprecations
* clear a bunch of helper functions out of _bootstrap.py
* use ModuleSpec with __main__
* address impact on stdlib (pkgutil, pickle, etc.)

Here's a patch just for the changes to the import page in the language reference. It's not perfect, but should represent the target pretty closely.

Let's try that again with the proper diff. :)

Thanks, that looks pretty good - so much less work for loaders to do.
After this, we may even be able to finally fix the circular import bug and the one where submodule imports succeed but the overall package import fails.

I've added a review

Something else we may be able to fix (albeit probably not in 3.4): several of the issues noted in PEP 395 (since __main__.__spec__.name will give the real module name, even when __name__ is set to "__main__" or otherwise modified)

Here's an update of the language reference patch based on Brett's comments.

Here's an updated patch that implements the meat of the current PEP. Docs are still lacking and I could probably add a few more tests. Other key things left to do:
* deprecations and removals
* refactor importlib finders and loaders to use the new Finder/Loader APIs
* refactor pythonrun.c to make use of __spec__
* adjust other APIs to use __spec__ (pickle, etc.)

This time against tip. :)
I should also point out that this patch is on top of the path in issue19413.

Here's a new patch that is mostly up to date with the PEP. I still need to work on Docs and add more tests. There are a few failing tests (due to the recent reload patch) that I need to fix when I get a minute.
This patch also implements find_spec() on FileFinder and PathFinder. Left to do:
* deprecations and removals
* refactor importlib loaders to use the new Finder/Loader APIs
* refactor pythonrun.c to make use of __spec__
* adjust other APIs to use __spec__ (pickle, etc.)

I've created a server-side clone for the implementation:
http://hg.python.org/features/pep-451/ 

Okay, I've updated the pep451 branch in the clone to include as much of the implementation as I've completed, which is the bulk of the functional changes. It's enough to pass the test suite.
Here's what I'd like to get done before the feature freeze (in this priority order):
1. wrap up the current functional changes in the clone;
2. change module.__initializing__ to module.__spec__._initializing;
3. refactor importlib loaders to use the new Finder/Loader APIs;
4. refactor pythonrun.c to make use of __spec__;
5. implement the deprecations and removals;
6. adjust other APIs to use __spec__ (pickle, etc.);
7. add rudimentary doc additions for the new APIs.
Other things that can (but don't have to) wait until after the beta release:
* finish doc changes;
* fill in gaps in test coverage (there shouldn't be much due to Brett's mother of all test suites for importlib)
I haven't had a chance yet to make any changes to Doc/reference/import.rst in response to Brett's review, but I did make the Doc/library/importlib.rst changes he recommended.

I don't quite know what you mean by "current functional changes in the clone". Is this changes to import to use exec_module() and such? Just a guess since step 3 suggests there are no loaders implementing the new API currently.
As for the tests, I'm hoping that simply refactoring some of them will allow reusing a bunch of the finder & loader tests instead of having to do it from scratch.

I should clarify why I want a clarification on step 1. On Friday my current plan (barring other bugs I need to squash for b1) is to just start working on the pep-451 repo and I want a TODO list to follow in this issue so I don't have to waste time trying to figure out what to work on next. That's why I want it spelled out if we want the loaders fixed first or import, etc.

Sorry that wasn't more clear. I committed the changes from the modulespec-primary-changes.diff patch to the pep451 branch in the server-side clone. Those changes are:
Step 1
------
1. added ModuleSpec class
2. added _SpecMethods wrapper class
3. added ModuleSpec factory functions
4. added tests for ModuleSpec, _SpecMethods, and the factories
5. exposed ModuleSpec in importlib.machinery
6. exposed ModuleSpec factories in importlib.util
7. added basic docs for ModuleSpec and the factory functions
Step 2
------
1. changed _find_module() to _find_spec()
2. changed _find_and_load_unlocked() to use _find_spec() and _SpecMethods
3. changed _setup() to use specs
4. changed pydoc to recognize __spec__
Step 3
------
1. updated the import reference doc
2. changed importlib.reload() to use specs
3. added importlib.find_spec()
4. changed importlib.find_loader() to wrap find_spec()
5. updated importlib.abc to reflect the new APIs
6. changed pkgutil to use specs
7. changed imp to use specs
8. fixed a bunch of broken tests to use spec
Step 3
------
1. implemented find_spec() on PathFinder
2. implemented find_spec() on FileFinder
3. re-implemented FileFinder.find_loader() to wrap find_spec()
4. re-implemented PathFinder.find_module() to wrap find_spec()
5. changed _NamespacePath to use specs
Step 5
------
1. added _module_repr function
2. changed ModuleType.__repr__ to wrap _module_repr
Others
------
* removed _NamespaceLoader
* added comments indicating deprecations and removals
At this point, the test suite passes and the fundamental changes of the PEP are implemented (on the server-side clone). Here's what's left to do before the feature freeze:
1. 
2. change module.__initializing__ to module.__spec__._initializing
3. refactor importlib loaders to use the new Finder/Loader APIs
4. refactor pythonrun.c to make use of specs
5. check pkgutil for any missed changes
6. implement the deprecations and removals
7. adjust other APIs to use __spec__ (pickle, runpy, inspect, pydoc, others?)
8. evaluate any impact on setuptools
Other things that can (but don't have to) wait until after the beta release:
* finish doc changes
* fill in any gaps in test coverage
* ensure new docstrings exist and are correct
* ensure existing docstrings are still correct

"1." was intentionally left blank <wink>

Regarding tests, a bunch of importlib (and other?) tests make direct calls to find_module(), find_loader(), or load_module(). These are still working, as expected. However, we should probably either replace them or supplement them with equivalent tests that make use of specs directly.

Yeah, don't replace any tests, add new ones for the new APIs. Given the
needs of 2/3 compatible loader implementations, the deprecations referred
to in the PEP should also be documentation-only for 3.4.
A more conservative approach also gives us a chance to make sure we have
provided a full replacement for load_module - it occurred to me after the
PEP was accepted that we may eventually need a "can_load_into(target)"
loader API after all, since loaders may not be finder specific. That means
that with the current API of passing the target to find_spec, I potentially
talked us into repeating the "load_module" mistake on the finder side of
things by asking the finder to handle more than it needed to.

> Yeah, don't replace any tests, add new ones for the new APIs.
That's what makes the most sense to me too.
> Given the
> needs of 2/3 compatible loader implementations, the deprecations referred
> to in the PEP should also be documentation-only for 3.4.
I'm fine with that. I seem to remember one place where an actual deprecation warning would be good, but I'll have to double-check.
> A more conservative approach also gives us a chance to make sure we have
> provided a full replacement for load_module - it occurred to me after the
> PEP was accepted that we may eventually need a "can_load_into(target)"
> loader API after all, since loaders may not be finder specific. That means
> that with the current API of passing the target to find_spec, I potentially
> talked us into repeating the "load_module" mistake on the finder side of
> things by asking the finder to handle more than it needed to.
Nice analogy. "can_load_into()" makes sense. Do you think it's important enough to include in 3.4? I'm not sure it is, since I'd expect it to be a pretty uncommon case that can be worked around pretty easily (since the finder is fully aware of the loader it's creating). The extra loader method would help with that boilerplate operation, but...
As we found out (and you expounded) there are a variety of reload/load-into cases that we could address more explicitly. Perhaps there's a better API that could address those needs more broadly, or maybe they're just not worth addressing specifically. Exploring all this is something that can wait, IMHO.

Although, a boolean query method would bring back the problem of the loader
not reporting any details on *why* it can't load into a particular target
module. So it may be better to have an optional loader
"check_existing_target" API that throws a suitable exception if the target
is unacceptable. Loaders that don't support reloading at all would just
always raise an exception, while those that don't care would just not
implement the method.
That would address my concern about the lack of useful error information in
Eric's original boolean check idea, without bothering finders with loader
related details as the accepted PEP does (I still like the idea of passing
a target to importlib.find_spec - I just no longer think we should be
passing that down to the finders themselves).
Another thing we need to check we have a test for: ensuring reloading a
namespace module picks up new directories added since it was first loaded.
This would all be so much easier if reloading wasn't supported in the first
place :)

On 12 Nov 2013 09:36, "Eric Snow" <report@bugs.python.org> wrote:
>
> As we found out (and you expounded) there are a variety of
reload/load-into cases that we could address more explicitly. Perhaps
there's a better API that could address those needs more broadly, or maybe
they're just not worth addressing specifically. Exploring all this is
something that can wait, IMHO.
Yes, that's an option, too, and probably a good one. To go down that path,
we would drop the various "target" parameters and say loaders that need to
check for the reloading case should continue to provide load_module without
exec_module (at least for 3.4). runpy would use the rule that it supports
anything that exposes exec_module without create_module.

> (I still like the idea of passing
> a target to importlib.find_spec - I just no longer think we should be
> passing that down to the finders themselves).
Passing the target to the finders isn't just for the sake of any implicit "check_target" test, though that was the original motivator. It also allows the finder to decide between multiple loaders based on other criteria related to the target (but not necessarily the loader). I think it was a good addition to the API regardless.
> Another thing we need to check we have a test for: ensuring reloading a
> namespace module picks up new directories added since it was first loaded.
Agreed. Furthermore, such a test is worthwhile outside the context of PEP 451. I'm tempted to say we're already covered with existing tests, but reload is goofy enough that an explicit test is worth it.
> This would all be so much easier if reloading wasn't supported in the first
> place :)
So very true. :)

And coming full circle: there's no *harm* in letting finders reject loading
into a target module, and that's orthogonal to having loaders reject it.
It's just that loaders that want to do that will currently still need to
implement load_module.
That means the only thing we need to postpone is the load_module
deprecation, since it still covers at least one advanced use case the new
API doesn't handle.

Sounds good. It will be worth adding a note to the load_module() docs indicating the limited cases where it is still appropriate, and encouraging the use of exec_module() instead.

I'm running the pep-451 clone against 2 buildbots for now:
* http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%20custom
* http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%20custom
The ubuntu one is happy with the feature clone, but the windows buildbot keeps finding problems. At first it was issue17116 all over again. I've now set __spec__ to None by default (a la issue17115). Then it fussed about test_everyone_has___loader__. I'm sure something else will turn up.
Any thoughts on why import-related tests are responding differently depending on platform? FWIW, I think the Windows failures should have also failed on the Ubuntu buildbot.
I'm also going to run on a Windows 7 and OS X to see where things stand.

The OS X buildbot I used [1] did not exhibit any of the failures that Windows server 2003 showed.
[1] http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%20custom 

The Windows 7 buildbot I tried [1] shows the exact same failures that the server 2003 does.
[1] http://buildbot.python.org/all/builders/x86%20Windows7%20custom 

Naturally the other Windows buildbot I tried did not fail any import-related tests. So much for consistency. :(
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%20custom 

The three Windows machines look like they have very different system
versions of Python (ActiveState, cygwin, vanilla CPython). Any chance we're
accidentally invoking the system Python instead of the built one somewhere?

I think worrying about more API expansions to handle edge case/advanced reload scenarios is not worth it (or at least not in Python 3.4). And if we are shifting importlib.reload() back to looking for a new finder then we are only worrying about when people decide to directly call exec_module() manually and want some pre-condition way to verify that the reload would succeed w/o putting that logic into exec_module() itself.
All I'm seeing is a mole hill for some advanced users who should be able to deal with it on their own.

> The three Windows machines look like they have very different system
> versions of Python (ActiveState, cygwin, vanilla CPython). Any chance we're
> accidentally invoking the system Python instead of the built one somewhere?
Good point. I'll check that out. That said, on my local clone (Ubuntu 12.10) I'm also getting no failures. It could also be related to the speed of the buildbot, but I doubt it. Anyway, I'll take a closer look at the stdout artifacts.

> I think worrying about more API expansions to handle edge case/advanced
> reload scenarios is not worth it (or at least not in Python 3.4).
Agreed.

Regarding the to-do list, I'm going to add a TODO file to the root of the feature clone (in the pep451 branch) so we don't have to keep updating this ticket. :)

FYI, as of cc7ebd952777 the bulk of the implementation in features/pep-451 is working.
There is a failing test on the Windows Server 2003 buildbot (which I'm trying to track down). I haven't been able to reproduce the failure on other buildbots, on my local Ubuntu 12.10 workstation, or on my local Windows laptop. See http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%20custom/builds/30/steps/test/logs/stdio.

Cool. Roundup linked the revision hash to the feature clone. :)

Just a heads up - since the two of you appear to have this well in hand,
don't wait for a review from me before committing it. I have plenty of
other things to look at before the beta deadline :)

Brett: looks like something frozen-related broke due to 6d1656ab2c85a527c.
test_frozen in test_frozen.py is failing now because frozen modules no longer have a __cached__ attribute (which was previously set to None). Previously it was set in PyImport_ExecCodeModuleObject() in import.c. I'm guessing that simply setting __cached__ (and __file__?) in FrozenImporter.exec_module() should restore the previous behavior. Doing so makes the test pass at least.

That's not the right fix for that test failure. If exec_module() is not meant to set import-related attributes like __cached__ because import itself (through _SpecMethods.init_module_attrs()) then test_frozen needs to be updated to not expect such attributes.
Honestly test_frozen should probably either be rolled into test_importlib.frozen or deleted instead of existing separately.

I've updated the TODO a little as well as cleaning up the XXX markers. The relevant ones are mostly just open questions on implementation tweaks, so nothing major. Otherwise my goal is to finish as much as possible of the non-critical items before the beta.
Brett: Any opinions on priority for the non-critical items?
Also, do you have a plan in mind for "Python can function w/ find_module(), find_loader(), and load_module() removed"? Do you mean comment out the deprecated methods and see what happens?

Semantic-affecting changes first, then fluffy bits in terms of priority. Basically go based on what is most critical to be in 3.4.
And yes, basically can you comment out find_module/find_loader/load_module and execute the test suite (sans import-related tests) without import-related failures.

No longer blocking b1, now just blocking rc1.

Your commit doesn't compile on Windows.
Category 	None
Changed by 	Eric Snow <ericsnowcurrently@gmail.com>
Changed at 	Fri 22 Nov 2013 16:17:09
Branch 	default
Revision 	07229c6104b16d0ab7cc63f3306157d3d2819fed
Comments
Implement PEP 451 (ModuleSpec).
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/3348/steps/compile/logs/stdio 

There is a commented-out line in Lib/importlib/_bootstrap.py:
+# if hasattr(loader, 'get_data'):
+ if hasattr(loader, 'get_filename'):
http://hg.python.org/cpython/rev/07229c6104b1#l6.369 

Removed the commented-out line in c8a84eed9155 

Failing buildbot:
http://buildbot.python.org/all/builders/x86%20XP-4%203.x/builds/9630/steps/test/logs/stdio
test.test_module.ModuleTests.test_module_repr_source
test.test_pkgutil.ExtendPathTests.test_iter_importers (passed when retried)

Same failures on http://buildbot.python.org/all/builders/x86%20Gentoo%203.x/builds/5444/steps/test/logs/stdio 

The test_module failure can be triggered by running test_importlib test_module. For some reason unittest's __spec__ gets set to one for the BuiltinImporter (and nothing else).

New changeset 98bab4a03120 by Brett Cannon in branch 'default':
Issue #18864: Don't try and use unittest as a testing module for
http://hg.python.org/cpython/rev/98bab4a03120 

I just realized that there is no way to make ModuleSpec.has_location return True if the spec is created by simply instantiating ModuleSpec. As it stands now you **must** go through importlib.util.spec_from_file_location(), which makes it not a helper but a **required** API to use. I think this is the only thing that cannot be stated directly through ModuleSpec instantiation or public attribute assignment.
This means either (a) the docs on ModuleSpec need to be **very** clear that you must use spec_from_file_location() to make spec.has_location be true, (b) add another keyword-only argument to ModuleSpec.__init__(), or (c) make has_location settable (e.g. be able to only set it to True/False or alternatively assigning it sets both has_location to True and origin to the assigned value).

Good catch. My preference would be for (c) simply add a setter for the property that sets the underlying private variable to True or False.

(c) it is then! I would just take the time to call bool() on the arg to coalesce it into a True/False thing (which maybe makes sense in __init__() as well).

Actually, ignore the __init__() part of my last comment since it's set to False directly.

Here's a patch for adding a setter for ModuleSpec.has_location.

LGTM

New changeset e961a166dc70 by Eric Snow in branch 'default':
Issue #18864: Add a setter for ModuleSpec.has_location.
http://hg.python.org/cpython/rev/e961a166dc70 

About the only thing left for this ticket is to finish up writing a few tests and round out some documentation.

There are a few lingering to-do comments that will need to be addressed as part of the remaining work:
Lib/test/test_importlib/extension/test_case_sensitivity.py:# XXX find_spec tests
Lib/test/test_importlib/extension/test_finder.py:# XXX find_spec tests
Lib/test/test_importlib/source/test_file_loader.py: mod = loader.load_module('_temp') # XXX
Lib/test/test_importlib/source/test_file_loader.py: # XXX Change to use exec_module().
Lib/test/test_importlib/test_windows.py: # XXX Need a test that finds the spec via the registry.

Failure with 3.4.0c1:
[170/394/5] test_importlib
test test_importlib failed -- Traceback (most recent call last):
 File "C:\Programs\Python34.32\lib\unittest\case.py", line 57, in testPartExecutor
 yield
 File "C:\Programs\Python34.32\lib\unittest\case.py", line 574, in run
 testMethod()
 File "C:\Programs\Python34.32\lib\unittest\loader.py", line 32, in testFailure
 raise exception
ImportError: Failed to import test module: test.test_importlib.source.test_abc_loader
Traceback (most recent call last):
 File "C:\Programs\Python34.32\lib\unittest\loader.py", line 312, in _find_tests
 module = self._get_module_from_name(name)
 File "C:\Programs\Python34.32\lib\unittest\loader.py", line 290, in _get_module_from_name
 __import__(name)
 File "C:\Programs\Python34.32\lib\test\test_importlib\source\test_abc_loader.py", line 73, in <module>
 class PyLoaderMock(abc.PyLoader):
AttributeError: 'module' object has no attribute 'PyLoader'
On the other hand, the test passes on a fresh debug build.

Terry has admitted in other bugs he filed that he didn't use a fresh install location, so the test failure is most likely a red herring.

I've gone ahead and moved all remaining work into separate issues. Once the 4 remaining sub-issues are resolved, this one can be closed. I've *finally* got a couple days free so I want to close this out. :)

Given that PEP 451 is listed in the "Finished PEPs (done, implemented in code repository)" section of the PEP index can we close this and associated issues out?

Dependencies 19711 and 21099 are still open.

I've asked for patch reviewss on the three dependencies outstanding, #19711, #19714 and #21099.

I've asked for updates on the two remaining dependencies #19711 and #21099.

The last dependencies have now been closed!