relative paths for modules in python

Question

I've attempted a few different techniques trying to do something that to me seems doable but I guess I am missing some gotchas about python (using 2.7 but would like this to work also for 3.* if possible).

I am not sure about terminology like package or module, but to me the following seems quite a "simple" doable scenario.

This is the directory structure:

.
├── job
│  └── the_script.py
└── modules
  ├── __init__.py
  └── print_module.py

The content of the_script.py:

# this does not work
import importlib
print_module = importlib.import_module('.print_module', '..modules')
# this also does not work
from ..modules import print_module
print_module.do_stuff()

The content of print_module:

def do_stuff():
 print("This should be in stdout")

I would like to run all this "relative paths" stuff as:

/job$ python2 the_script.py

But the importlib.import_module gives various errors:

• if I just use 1 input parameter ..modules.print_module, then I get: TypeError("relative imports require the 'package' argument")
• if I use 2 input parameters (as in the example above), then I get: ValueError: Empty module name

On the other hand using the from ..modules syntax I get: ValueError: Attempted relative import in non-package.

I think the __init__.py empty file should be enough to qualify that code as "packages" (or modules? not sure about the terminology), but it seems there's something I am missing about how to manage relative paths.

I read that in the past people was hacking this using the path and other functions from import os and import sys, but according to the official docs (python 2.7 and 3.*) this should not be needed anymore.

What am I doing wrong and how could I achieve the result of printing the content modules/print_module.do_stuff calling it from a script in the "relative directory" job/?

• Daniel Roseman: Why would you use importlib here? Why not just import it directly?
• BrenBarn: __init__.py only creates a package for the directory it is in. You need to add __init__.py to all directories in the tree you showed. You will also need to make sure the directory containing the top-level package directory is on sys.path, and if you want to be able to execute one of the files as a script, you'll need to think about the issues described here and here.

Answer 1

If you follow the structure of this guide here: http://docs.python-guide.org/en/latest/writing/structure/#test-suite (highly recommend reading it all, it is very helpful) you will see this:

To give the individual tests import context, create a tests/context.py file:

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import sample

Then, within the individual test modules, import the module like so:

from .context import sample

This will always work as expected, regardless of installation method.

Translated in your case this means:

root_folder
├── job
│ ├── context.py <- create this file
│ └── the_script.py
└── modules
 ├── __init__.py
 └── print_module.py

In the context.py file write the lines shown above, but import modules instead of import samples

Finally in your the_script.py: from .context import module and you will be set to go!

Good luck :)

• TPPZ: Thanks :) but this gives: ImportError: No module named modules.print_module
• TPPZ: In python 2 this gives ImportError: No module named airflow_stuff.modules.print_module, in python 3 it gives a similar error message
• TPPZ: That way of using a context via sys.path gives ValueError: Attempted relative import in non-package, sorry to keep pointing out these details but it does not seem to work, however I found a similar solution I posted in an answer here though it seems quite close to yours so not sure why in your case it raises an error.

Answer 2

I found a solution using sys and os.

The script the_script.py should be:

import sys
import os
lib_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../modules'))
sys.path.append(lib_path)
# commenting out the following shows the `modules` directory in the path
# print(sys.path)
import print_module
print_module.do_stuff()

Then I can run it via command line no matter where I am in the path e.g.:

• /job$ python2 the_script.py
• <...>/job$ python2 <...>/job/the_script.py

• mike rodent: Thanks. Makes sense. Also puzzling: almost as if they have deliberately made things so you can't use a relative path for imports (or have to jump through hoops to do so).

Answer 3

If you are not sure about terminology go to very nice tutorials:

http://docs.python-guide.org/en/latest/writing/structure/#modules

and

http://docs.python-guide.org/en/latest/writing/structure/#packages

But for your structure:

.
├── job
│ └── the_script.py
└── modules
 ├── __init__.py
 └── print_module.py

just say in the the_script.py:

import sys
sys.append('..')
import modules.print_module

This will add parent directory to PYTHONPATH, and python will see directory 'parallel' to job directory and it will work.

I think that at the most basic level it is sufficent to know that:

1. package is any directory with __init__.py file
2. module is a file with .py, but when you are importing module you omit extension.

• TPPZ: Thanks for pointing out the docs :) but this gives: ImportError: No module named modules.print_module
• TPPZ: In python 2 this gives AttributeError: 'module' object has no attribute 'append', however if I put instead sys.path.append('..'), then this gives ImportError: No module named modules.print_module. In python 3 it gives similar error messages.