How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with .py) in Python 3.3+?
I used imp.load_module as follows:
>>> import imp
>>> path = '/tmp/a-b.txt'
>>> with open(path, 'U') as f:
... mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE))
...
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
It still works in Python 3.3, but according to imp.load_module documentation, it is deprecated:
Deprecated since version 3.3: Unneeded as loaders should be used to load modules and find_module() is deprecated.
and imp module documentation recommends to use importlib:
Note New programs should use importlib rather than this module.
What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated imp.load_module function?
5 Answers 5
Found a solution from importlib test code.
Using importlib.machinery.SourceFileLoader:
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = loader.load_module()
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
NOTE: only works in Python 3.3+.
UPDATE Loader.load_module is deprecated since Python 3.4. Use Loader.exec_module instead:
>>> import types
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = types.ModuleType(loader.name)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b'>
>>> import importlib.machinery
>>> import importlib.util
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> spec = importlib.util.spec_from_loader(loader.name, loader)
>>> mod = importlib.util.module_from_spec(spec)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
17 Comments
load_module ignores via warnings.catch_warnings. If you instead use mod = imp.load_source('a_b', '/tmp/a-b.txt'), it raises the following warning (use -Wall): DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead.Updated for Python >= 3.8:
Short version:
>>> # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
>>> import importlib.util, sys
>>> spec = importlib.util.spec_from_file_location(modname, fname)
>>> module = importlib.util.module_from_spec(spec)
>>> sys.modules[modname] = module
>>> spec.loader.exec_module(module)
Full version:
>>> import importlib.util
>>> import sys
>>> from pathlib import Path
>>> from typing import TYPE_CHECKING
>>>
>>>
>>> if TYPE_CHECKING:
... import types
...
...
>>> def import_source_file(fname: str | Path, modname: str) -> "types.ModuleType":
... """
... Import a Python source file and return the loaded module.
... Args:
... fname: The full path to the source file. It may container characters like `.`
... or `-`.
... modname: The name for the loaded module. It may contain `.` and even characters
... that would normally not be allowed (e.g., `-`).
... Return:
... The imported module
... Raises:
... ImportError: If the file cannot be imported (e.g, if it's not a `.py` file or if
... it does not exist).
... Exception: Any exception that is raised while executing the module (e.g.,
... :exc:`SyntaxError). These are errors made by the author of the module!
... """
... # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
... spec = importlib.util.spec_from_file_location(modname, fname)
... if spec is None:
... raise ImportError(f"Could not load spec for module '{modname}' at: {fname}")
... module = importlib.util.module_from_spec(spec)
... sys.modules[modname] = module
... try:
... spec.loader.exec_module(module)
... except FileNotFoundError as e:
... raise ImportError(f"{e.strerror}: {fname}") from e
... return module
...
>>> import_source_file(Path("/tmp/my_mod.py"), "my_mod")
<module 'my_mod' from '/tmp/my_mod.py'>
Original answer for Python 3.5 and 3.6
Shorter version of @falsetru 's solution:
>>> import importlib.util
>>> spec = importlib.util.spec_from_file_location('a_b', '/tmp/a-b.py')
>>> mod = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>
I tested it with Python 3.5 and 3.6.
According to the comments, it does not work with arbitrary file extensions.
3 Comments
importlib.util.spec_from_file_location(..) returns None for me; causing an exception for the following importlib.util.module_from_spec(..) call. (See i.imgur.com/ZjyFhif.png)
importlib.util.spec_from_file_location works for known file name extensions (.py, .so, ..), but not for others (.txt...)Similar to @falsetru but for Python 3.5+ and accounting for what the importlib doc states on using importlib.util.module_from_spec over types.ModuleType:
This function [
importlib.util.module_from_spec] is preferred over usingtypes.ModuleTypeto create a new module as spec is used to set as many import-controlled attributes on the module as possible.
We are able to import any file with importlib alone by modifying the importlib.machinery.SOURCE_SUFFIXES list.
import importlib
importlib.machinery.SOURCE_SUFFIXES.append('') # empty string to allow any file
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# if desired: importlib.machinery.SOURCE_SUFFIXES.pop()
3 Comments
importlib.machinery.EXTENSION_SUFFIXES.append('') still makes importlib.util.spec_from_file_location return None.importlib.util.spec_from_file_location should still work with extensions if you specify a loader__init__importlib helper function tested on Python 3.10
Here is a convenient, ready-to-use helper to replace imp, with an example. The technique is the same as that of https://stackoverflow.com/a/19011259/895245 , this is just providing a more convenient function.
main.py
#!/usr/bin/env python3
import os
from importlib import util, machinery
def import_path(path):
module_name = os.path.basename(path).replace('-', '_')
spec = util.spec_from_loader(
module_name,
importlib.machinery.SourceFileLoader(module_name, path)
)
module = util.module_from_spec(spec)
spec.loader.exec_module(module)
sys.modules[module_name] = module
return module
notmain = import_path('not-main')
print(notmain)
print(notmain.x)
not-main
x = 1
Run:
python3 main.py
Output:
<module 'not_main' from 'not-main'>
1
I replace - with _ because my importable Python executables without extension have hyphens as in my-cmd. This is not mandatory, but produces better module names like my_cmd.
Related:
Python 3.7
In that version I was instead using the following invocations for the helper, but they do not seem to work on Python 3.10
import importlib
spec = importlib.util.spec_from_loader(
module = importlib.util.module_from_spec(
This pattern is also mentioned in the docs at: https://docs.python.org/3.7/library/importlib.html#importing-a-source-file-directly
I ended up moving to it because after updating to Python 3.7, import imp prints:
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
and I don't know how to turn that off, this was asked at:
Comments
after many failure solutions this one works for me
def _import(func,*args):
import os
from importlib import util
module_name = "my_module"
BASE_DIR = "wanted module directory path"
path = os.path.join(BASE_DIR,module_name)
spec = util.spec_from_file_location(func, path)
mod = util.module_from_spec(spec)
spec.loader.exec_module(mod)
return getattr(mod,func)(*args)
and to call it just write the function name and it's parameters _import("function",*args)
imp.load_module()over a straight import statement. Do you expect to import the module by name later (e.g.import a_b)? Do you care that any custom importers won't be used in this approach? Do you expect the module to be full-featured (e.g. define__name__and__loader__)?THIS='blah'like lines). The name of the file is not ended with.py. My program read that file.