How should I perform imports in a python module without polluting its namespace?

Question

I am developing a Python package for dealing with some scientific data. There are multiple frequently-used classes and functions from other modules and packages, including numpy, that I need in virtually every function defined in any module of the package.

What would be the Pythonic way to deal with them? I have considered multiple variants, but every has its own drawbacks.

• Import the classes at module-level with from foreignmodule import Class1, Class2, function1, function2
  Then the imported functions and classes are easily accessible from every function. On the other hand, they pollute the module namespace making dir(package.module) and help(package.module) cluttered with imported functions

• Import the classes at function-level with from foreignmodule import Class1, Class2, function1, function2
  The functions and classes are easily accessible and do not pollute the module, but imports from up to a dozen modules in every function look as a lot of duplicate code.

• Import the modules at module-level with import foreignmodule
  Not too much pollution is compensated by the need to prepend the module name to every function or class call.

• Use some artificial workaround like using a function body for all these manipulations and returning only the objects to be exported... like this

  def _export():
   from foreignmodule import Class1, Class2, function1, function2
   def myfunc(x):
   return function1(x, function2(x))
   return myfunc
  myfunc = _export()
  del _export
  

  This manages to solve both problems, module namespace pollution and ease of use for functions... but it seems to be not Pythonic at all.

So what solution is the most Pythonic? Is there another good solution I overlooked?

• Manny D: If you're that concerned about namespace pollution, your best pythonic method would probably be good ol' fashioned import foreignmodule.
• chown: Check out PEP 8 -- Style Guide for Python Code and PEP 328 -- Imports: Multi-Line and Absolute/Relative.
• Bryan Oakley: @chown: why did you write your answer as an answer, only to convert it to a comment? It doesn't belong in the comments if it answers the question. If there is a lot of discussion your comment may get hidden and become irrelevant.
• chown: @bryan I didn't want to distract from answers that had more explanation since mine was just copy/paste

Answer 1

Go ahead and do your usual from W import X, Y, Z and then use the __all__ special symbol to define what actual symbols you intend people to import from your module:

__all__ = ('MyClass1', 'MyClass2', 'myvar1', ...)

This defines the symbols that will be imported into a user's module if they import * from your module.

In general, Python programmers should not be using dir() to figure out how to use your module, and if they are doing so it might indicate a problem somewhere else. They should be reading your documentation or typing help(yourmodule) to figure out how to use your library. Or they could browse the source code yourself, in which case (a) the difference between things you import and things you define is quite clear, and (b) they will see the __all__ declaration and know which toys they should be playing with.

If you try to support dir() in a situation like this for a task for which it was not designed, you will have to place annoying limitations on your own code, as I hope is clear from the other answers here. My advice: don't do it! Take a look at the Standard Library for guidance: it does from ... import ... whenever code clarity and conciseness require it, and provides (1) informative docstrings, (2) full documentation, and (3) readable code, so that no one ever has to run dir() on a module and try to tell the imports apart from the stuff actually defined in the module.

• Tanriol: Actually I didn't know that all is important not only for from module import *, but also for help() output... now everything is much more clear.
• Ethan Furman: dir() is great for a quick reminder as to what is there when using the REPL, and __all__ goes hand in hand with dir(), as well as help(), and many (most?) third party introspection packages also play well with __all__. What do you think dir() is for?
• Brandon Rhodes: I use dir() to inspect objects, because usually everything on an object is somehow "useful" — in the sense that it belongs to either that particular object instance, or its class. I find dir() of much less utility on a module, because usually my question is not "what symbols are defined in this module?" — because that includes 3rd party things imported in — but "what does this module provide?" So I use help() or the docs or the source. :)
• Ethan Furman: Sounds reasonable. However, much like Python becoming more Object Oriented than it was in the early '90s, __all__ now does double duty -- even if you don't want your users doing from ... import *, a well defined module will still define __all__ for the purpose of defining the public API, and in so doing dir() becomes useful even on modules. docs.python.org/reference/…
• Brandon Rhodes: But dir() shows everything in a module, not just __all__ — or am I misunderstanding you?

Answer 2

One technique I've seen used, including in the standard library, is to use import module as _module or from module import var as _var, i.e. assigning imported modules/variables to names starting with an underscore.

The effect is that other code, following the usual Python convention, treats those members as private. This applies even for code that doesn't look at __all__, such as IPython's autocomplete function.

An example from Python 3.3's random module:

from warnings import warn as _warn
from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
from math import sqrt as _sqrt, acos as _acos, cos as _cos, sin as _sin
from os import urandom as _urandom
from collections.abc import Set as _Set, Sequence as _Sequence
from hashlib import sha512 as _sha512

Another technique is to perform imports in function scope, so that they become local variables:

"""Some module"""
# imports conventionally go here
def some_function(arg):
 "Do something with arg."
 import re # Regular expressions solve everything
 ...

The main rationale for doing this is that it is effectively lazy, delaying the importing of a module's dependencies until they are actually used. Suppose one function in the module depends on a particular huge library. Importing the library at the top of the file would mean that importing the module would load the entire library. This way, importing the module can be quick, and only client code that actually calls that function incurs the cost of loading the library. Further, if the dependency library is not available, client code that doesn't need the dependent feature can still import the module and call the other functions. The disadvantage is that using function-level imports obscures what your code's dependencies are.

Example from Python 3.3's os.py:

def get_exec_path(env=None):
 """[...]"""
 # Use a local import instead of a global import to limit the number of
 # modules loaded at startup: the os module is always loaded at startup by
 # Python. It may also avoid a bootstrap issue.
 import warnings

• godaygo: The only answer in which autocomplete issues are discussed. Although it seems to me that this is the key point for not to pollute the namespace.

Answer 3

Import the module as a whole: import foreignmodule. What you claim as a drawback is actually a benefit. Namely, prepending the module name makes your code easier to maintain and makes it more self-documenting.

Six months from now when you look at a line of code like foo = Bar(baz) you may ask yourself which module Bar came from, but with foo = cleverlib.Bar it is much less of a mystery.

Of course, the fewer imports you have, the less of a problem this is. For small programs with few dependencies it really doesn't matter all that much.

When you find yourself asking questions like this, ask yourself what makes the code easier to understand, rather than what makes the code easier to write. You write it once but you read it a lot.

• Tanriol: Thanks, this position seems very well-thought... unfortunately now I'm writing the code and huge multi-line expressions with several functions from multiple imports are not much fun to write. However, planning for the future seems to be the right way.
• agf: @Tanriol If you're accessing an import many, many times inside a loop and you've found the attribute access is actually a performance problem, you can always rebind it to the local scope, either in a function body or as a default argument in the function definition if you only want to do it once.
• Voo: @Tanriol At least in Python3 (no idea about 2.x) you can use import urllib.request as insert_something_here - which combines the advantages (and disadvantages) of both solutions. If you go overboard it makes reading your code more complex for others, but with the right balance you can get something that's informative and not too much to type ;)
• Lutz Prechelt: When using as, you can often find nicely mnemonic abbreviations that remove most of the verbosity, yet keep most of the benefit. Such as: import numpy as np, import matplotlib as mpl, import django as dj, and so on.

Answer 4

For this situation I would go with an all_imports.py file which had all the

from foreignmodule import .....
from another module import .....

and then in your working modules

import all_imports as fgn # or whatever you want to prepend
...
something = fgn.Class1()

Another thing to be aware of

__all__ = ['func1', 'func2', 'this', 'that']

Now, any functions/classes/variables/etc that are in your module, but not in your modules's __all__ will not show up in help(), and won't be imported by from mymodule import * See Making python imports more structured? for more info.

• Tanriol: Thanks, that's one more good solution. The idea of using __all__ is only partially usable due to having the same problem in the package, as for packages it has different semantics.
• Brandon Rhodes: This can hurt readability, since a programmer reading your code now has to step through two (at least) modules to see where a symbol is coming from and how it is defined.

Answer 5

I would compromise and just pick a short alias for the foreign module:

import foreignmodule as fm

It saves you completely from the pollution (probably the bigger issue) and at least reduces the prepending burden.

• Tanriol: Not completely - the alias is still there - but yes, that's a good solution with seemingly no major drawbacks.

Answer 6

I know this is an old question. It may not be 'Pythonic', but the cleanest way I've discovered for exporting only certain module definitions is, really as you've found, to globally wrap the module in a function. But instead of returning them, to export names, you can simply globalize them (global thus in essence becomes a kind of 'export' keyword):

def module():
 global MyPublicClass,ExportedModule
 import somemodule as ExportedModule
 import anothermodule as PrivateModule
 class MyPublicClass:
 def __init__(self):
 pass
 class MyPrivateClass:
 def __init__(self):
 pass
module()
del module

I know it's not much different than your original conclusion, but frankly to me this seems to be the cleanest option. The other advantage is, you can group any number of modules written this way into a single file, and their private terms won't overlap:

def module():
 global A
 i,j,k = 1,2,3
 class A:
 pass
module()
del module
def module():
 global B
 i,j,k = 7,8,9 # doesn't overwrite previous declarations
 class B:
 pass
module()
del module

Though, keep in mind their public definitions will, of course, overlap.