homepage

It seems that test.support.import_fresh_module gets tripped up with its module blocking when you attempt to get a fresh copy of a submodule of a module where you are also importing the module that you are trying to block (bit of a doozy of a sentence there...). So, for example, with the following configuration in mymodule/__init__.py:
 from .other import other
 
 try:
 from ._a import attr
 except ImportError:
 from ._b import attr
(Assuming _a.attr = "A" and _b.attr = "B"), if you attempt to do:
 m = test.support.import_fresh_module("mymodule", fresh=("mymodule._other",), blocked=("mymodule._a"))
Then you'll find that m.attr is pulled from _a.attr. Here's a small script to demonstrate:
 from test.support import import_fresh_module
 import sys
 def import_ab(fresh_other):
 fresh = ("mymodule._other", ) if fresh_other else ()
 mods_out = []
 for to_block in "_b", "_a":
 blocked = (f"mymodule.{to_block}",)
 mods_out.append(import_fresh_module("mymodule",
 fresh=fresh, blocked=blocked))
 return mods_out
 for fresh_other in [True, False]:
 mymodule_a, mymodule_b = import_ab(fresh_other)
 qualifier = "With" if fresh_other else "Without"
 print(f"{qualifier} a fresh import of mymodule._other")
 print(f"a: {mymodule_a.attr}")
 print(f"b: {mymodule_b.attr}")
 print()
When you run it with a suitably configured module on Python 3.8:
$ python importer.py 
With a fresh import of mymodule._other
a: A
b: A
Without a fresh import of mymodule._other
a: A
b: B
It also happens if you add `mymodule._a` or `mymodule._b` to the fresh list when you are trying to block the other one.
I *think* the problem is that in the step where _save_and_remove_module is called on fresh_name (see here: https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329), it's necessarily populating `sys.modules` with a fresh import of the top-level module we're trying to import (mymodule) *before* the blocking goes into effect, then the final call to importlib.import_module just hits that cache.
I think either of the following options will fix this issue:
1. Switching the order of how "fresh" and "blocked" are resolved or
2. Deleting `sys.modules[name]` if it exists immediately before calling `importlib.import_module(name)
That said, I'm still having some weird statefulness problems if I block a C module's import and *then* block a Python module's import, so there may be some other underlying pathology to the current approach.

> I *think* the problem is that in the step where _save_and_remove_module is called on fresh_name (see here: https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329)
Looks like deleting a module name after `__import__(name)` is not good enought in https://github.com/python/cpython/blob/master/Lib/test/support/__init__.py#L244.(some redundant submodules should be removed, no?)
paul's example can be passed in PR19390.

I just encountered this bug after rewriting datetime tests in issue45229. It is also the cause of issue40058.
Not knowing about this issue I have wrote different patch than Hai Shi. It makes the code of import_fresh_module() clearer and fixes many other issues.

New changeset ec4d917a6a68824f1895f75d113add9410283da7 by Serhiy Storchaka in branch 'main':
bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654)
https://github.com/python/cpython/commit/ec4d917a6a68824f1895f75d113add9410283da7

New changeset 7873884d4730d7e637a968011b8958bd79fd3398 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654) (GH-28657)
https://github.com/python/cpython/commit/7873884d4730d7e637a968011b8958bd79fd3398

New changeset f7e99c98130a705f88348dde60adb98d5bfd8b98 by Serhiy Storchaka in branch '3.9':
[3.9] bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654) (GH-28658)
https://github.com/python/cpython/commit/f7e99c98130a705f88348dde60adb98d5bfd8b98

_save_and_remove_module in the old code did too much and too little. It tested that the fresh module is importable, saved the current state of the fresh module and its submodules and removed the fresh module and its submodules. Since it tested importability before saving all states, it can have a side effect of importing other fresh module, and since it did it before blocking, the imported "fresh" module could import the blocked modules.
> 1. Switching the order of how "fresh" and "blocked" are resolved or
It does not help, because the step of resolving "fresh" includes saving the sys.modules state, and the step of resolving "blocked" modifies sys.modules. Also the step of resolving "fresh" can modify sys.modules before saving its sate.
2. Deleting `sys.modules[name]` if it exists immediately before calling `importlib.import_module(name)
It would be not enough, because the problem can occur in one of additional fresh modules imported by the initial module. We need to remove all fresh modules before attempting to import them.
The new import_fresh_module() consists of the following steps:
1. Save the state of all fresh and blocked modules and their submodules.
2. Remove the fresh modules and their submodules and block the blocked modules and their submodules.
3. Import all additional fresh modules and return None if any of them is not importable.
4. Import the initial module and raise ImportError if it is not importable.
5. Restore states of all fresh and blocked modules and their submodules.
It does not save and restore the state of all modules, because some indirectly imported modules can not support repeated importing. It can be reconsidered in new versions.