Python module importing with sys.path and os.path issue

Question

I spent some time researching this and I just cannot work this out in my head.

I run a program in its own directory home/program/core/main.py

In main.py I try and import a module called my_module.py thats located in a different directory, say home/program/modules/my_module.py

In main.py this is how I append to sys.path so the program can be run on anyone's machine (hopefully).

import os.path
import sys
# This should give the path to home/program
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__), '..'))
# Which it does when checking with
print os.path.join(os.path.abspath(os.path.dirname(__file__), '..')
# So now sys.path knows the location of where modules directory is, it should work right?
import modules.my_module # <----RAISES ImportError WHY?

However if I simply do:

sys.path.append('home/program/modules')
import my_module

It all works fine. But this is not ideal as it now depends on the fact that the program must exist under home/program.

Answer 1

that's because modules isn't a valid python package, probably because it doesn't contain any __init__.py file (You cannot traverse directories with import without them being marked with __init__.py)

So either add an empty __init__.py file or just add the path up to modules so your first snippet is equivalent to the second one:

sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__), '..','modules'))
import my_module

note that you can also import the module by giving the full path to it, using advanced import features: How to import a module given the full path?

• miro_x: But that is my problem why should I have pointless init.py file?? I just want that directory where my_module.py is located clean, my_module.py is actually the only file in that directory. Why does sys.path.append('home/programs/modules') work without the __init__.py file
• Jean-François Fabre: because that how python works. You cannot traverse directories with import without them being marked with __init__.py, it's not pointless. you still have the second solution: appending the path up to "modules".
• miro_x: ok but how on earth do I implement a code where I can import a module such that the program will work in whichever directory the user runs it without the init.py
• Jean-François Fabre: by appending the path up to the module directory. you can't do otherwise. or use importlib, see my edit

Answer 2

Although the answer can be found here, for convenience and completeness here is a quick solution:

import importlib
dirname, basename = os.path.split(pyfilepath) # pyfilepath: /my/path/mymodule.py
sys.path.append(dirname) # only directories should be added to PYTHONPATH
module_name = os.path.splitext(basename)[0] # /my/path/mymodule.py --> mymodule
module = importlib.import_module(module_name) # name space of defined module (otherwise we would literally look for "module_name")

Now you can directly use the namespace of the imported module, like this:

a = module.myvar
b = module.myfunc(a)