Add application root to path

Question

I am creating a package to be used in our company that adds the root of the project to the system path.

I will add a source_root.config file in our root directory.

add_root_path.py

import sys
import inspect
from pathlib import Path
from inspect import getsourcefile
CUR_FILE = Path(getsourcefile(lambda:0)).name
SOURCE_ROOT_DEFAULT_FILE = 'source_root.config'
def _get_dir_of_importer_file():
 """ the directory of the file importing the package"""
 frames_stack = inspect.stack()
 for frame in frames_stack:
 # first file that's not importlib._bootstrap and not current file
 if 'py' in frame.filename and Path(frame.filename).name != CUR_FILE: 
 file_name = frame.filename
 break
 return Path(file_name).resolve().parent
def find_source_root(source_root_file=SOURCE_ROOT_DEFAULT_FILE):
 importer_dir = _get_dir_of_importer_file()
 while True: # can go wrong if package is not used correctly
 files_in_dir = map(Path.resolve, importer_dir.glob('*'))
 for file in files_in_dir:
 if file.name == source_root_file:
 return importer_dir
 importer_dir = importer_dir.parent
def append_root_to_path(source_root_file=SOURCE_ROOT_DEFAULT_FILE):
 source_root = find_source_root(source_root_file)
 sys.path.append(source_root.resolve().as_posix())

This is done in order to properly import the files inside the project and because using PYTHONPATH in windows is not that convenient for pytest and alembic (meaning not running the files directly via python.exe).

• Peilonrayz: Can you give an example of how you call this? Do you just do append_root_to_path()?
• moshevi: @Peilonrayz yes exactly
• Peilonrayz: I just want to check what this is meant to do, if you run python my_project you want my_project to be in your path? Also if you run pytest for my_project to be in your path. Both these allow you to import my_project, allowing internal and external imports to work correctly? Or is this doing something else?
• moshevi: yes I want every file to be able to import from my_project dir, so i will import this package and run append_root_to_path
• 301_Moved_Permanently: What is the purpose of your source_root.config file? Is it just a file system maker or actual (eventually user editable) configuration? Do you rely on this code to find its location (such as open(f'{sys.path[-1]}/source_root.config'))?

Answer 1

Your project is setup wrong

Pytest

The simplest solution to most import problems is to just make your project a setuptools package. And install your package.

Whether or not your project is a library or application to fix pytest is really, really simple. You make your project a setuptools package (Which you should already have done if you're making a library). This is by configuring the setup.py file that explains to pip, setuptools, etc how to install your project.

From here you install your package and things just work.

$ pip install .
$ pytest

Applications

Now you might be saying that's cool and all it works for pytest. But now it's broken when I run my program, using python. There are two solutions to that.

• Execute your application as a module. Add a __main__.py to the top level directory which will be your application's entry point. Make sure you move your if __name__ == '__main__' code here. And then just use:

  $ python -m my_project
  

• Setup an entry point for the setuptools package.

  Once you get the above working, then all you need to do is ensure your main guard is only calling a function main. If this is the case, then you can say main is your entry point in your setup.py.

  entry_points={
   'console_scripts': [
   'my_project=my_project.__main__:main',
   ],
  },
  

  Usage is then just:

  $ my_project
  

  This is how the cool kids publish applications to PyPI.

But what about my code? Where's my code review?

Oh yeah, your code is an unneeded hack job. Seriously just make a setuptools package. You get some benefits from it like your project always being on the path, being able to install projects from a private PyPI repository, being able to use tox and nox, having a customizable entry point mapped to a cool name, and not having to use hacks to get your tests to work. I feel I'm biased here, but I really don't see any downsides. Heck I now only use pip to install to Apache.

From here you can just use either of the import strategies you want. I prefer relative imports, but absolute imports might be your jam.

1. Absolute imports.

   from my_project import subpackage
   subpackage.foo()
   

2. Cool relative imports.

   from . import subpackage
   subpackage.foo()
   

MVCE of all of the above

I remember when I was trying to convert to relative imports I Googled and all that I could find was hack jobs. People saying the names of some PEPs that talk about what I'm on about, but only talk about __name__, __files__ and '__main__'. Overall I feel the subjects a (削除) shit show (削除ここまで) mess.

And so if you're like I was and just trying to make some sense of all this nonsense, I have a small MVCE for you. If you copy the files and commands verbatim then you too can have a properly configured Python project.

./src/my_project/__init__.py

./src/my_project/__main__.py (Works with from my_project.foo import FOO too.)

from .foo import FOO
def main():
 print(FOO)
if __name__ == '__main__':
 main()

./src/my_project/foo.py

FOO = 'Bar'

./tests/test_foo.py

import my_project.foo
def test_foo():
 assert my_project.foo.FOO == 'Bar'

./setup.py

from setuptools import setup, find_packages
setup(
 name='my_project',
 packages=find_packages('src'),
 package_dir={'': 'src'},
 entry_points={
 'console_scripts': [
 'my_project=my_project.__main__:main',
 ],
 },
)

Running it

$ cd src
$ python -m my_project
Bar
$ cd ..
$ pip install .
$ pytest
===== 1 passed in 0.05s =====
$ my_project
Bar

In Addition

If you configure your project correctly then your problems should magically disappear. If you still have problems then you should double check you have installed your project.

You may want to use the -e flag when you install the project, so that you don't need to pip install . each time you change a file.

Better yet use this as a stepping stone to upgrade to modern Python development and use tox or nox to run you tests. Not only do these come with the benefit that they test your setuptools package is correctly configured. They also let you, and your coworkers, not fret over having to install a virtual environment and ensure pip install -e . has ran. Just setup a config file and then you all only need to run tox or nox for things to just work^TM.

• moshevi: My main concern with this approach is the fact that my repo contains several projects, with each one of them containing a different if __name__ == '__main__'. they however interact with a common module that contains stuff that are generally needed (aws scripts etc).
• Peilonrayz: @moshevi I don't see how that is a problem, you can have multiple packages in the src directory. If you pip install . then you can python -m my_package_1 and also python -m my_package_2. Also you can setup multiple entry points to allow my_package_1 and my_package_2 to be valid commands. The common module will still be import commonmodule.
• moshevi: But they under the same code base with each one of them requiring different dependencies, how would one setup.py handle this? . are you running python -m my_package_1 inside the my_project dir?
• Peilonrayz: @moshevi If they're wildly different projects, then they should be in their own projects then. No, my_package_1 would be in the src dir, if you're using subpackages then your entire structure is massively wonky. Also if you pip install . then you can python -m my_package_1 from anywhere, as long as you're using the correct python.