Setting up Python path during development

Question

I work on a couple of different programs and packages in Python. They are each developed in their own Git repository, but frequently need to import modules defined in other packages. For instance, during development, the directory structure looks something like:

src/
|-- project-a/
| |-- client.py
| |-- server.py
| |-- package-a
| |-- __init__.py
| |-- module.py
|-- project-b/
| |-- package-b
| | |-- __init__.py
| | |-- other_module.py
| |-- package-c
| |-- __init__.py
| |-- third_module.py
|-- project-c/
 |-- server1.py
 |-- server2.py
 |-- package-d/
 |-- package-e/
 |-- package-f/

When they are all installed, they work fine; they are all installed such that each package is in your Python path, and you can import from them as you need.

However, in development, I want the development version of each of these to be in my Python path, not the installed version. When making changes, I don't want to have to install each package that I'm changing to test it, I want the changes to take effect immediately. That means my Python path needs to include the directories project-a, project-b, etc.

Our current solution is just to have an environment.bash in the top level, which you source in your shell and it sets PYTHONPATH. That works OK, but I frequently forget to do so; since this is a client server application, with communications between servers, I need to have at least four windows open to different VMs to run this, and it happens pretty often that I forget to source environment.bash in at least one of those, leading me to try debugging strange behavior until I realize I'm importing the wrong things.

Another solution would be to set sys.path from within the top level client.py or server.py. This would work fine for launching them directly, but I would also need the path set up for running tools like Pylint or Sphinx, which that solution wouldn't cover. I'd also need a way to distinguish between running from source (when I want the path to include . and ../project-b) and running the installed version (which should use the standard path without modification).

Another choice would be to have a Makefile which sets up PYTHONPATH appropriately for various targets like make run-server, make lint, make doc, and so on. That's OK for those targets, which don't require any options, but would be inconvenient for running the client, which takes arguments. make run-client ARGS='foo bar' is a fairly cumbersome way to invoke it.

Is there any common way of setting up the Python path during development so that both my executables and tools like Pylint and Sphinx can pick it up appropriately, without interfering with how it will behave when installed?

Answer 1

A straightforward solution would be to simply symlink in the directories for each module in a separate folder, and run things from there. That way Python sees them all being in the same location, even though the actual sources are in different repositories.

src/
|-- project-a/
| |-- client.py
| |-- server.py
| |-- package-a
| |-- __init__.py
| |-- module.py
|-- project-b/
 |-- package-b
 | |-- __init__.py
 | |-- other_module.py
 |-- package-c
 |-- __init__.py
 |-- third_module.py
run/
|-- client.py --> ../src/project-a/client.py
|-- server.py --> ../src/project-a/server.py
|-- package-a/ --> ../src/project-a/package-a/
|-- package-b/ --> ../src/project-b/package-b/
|-- package-c/ --> ../src/project-b/package-c/

• Brian Campbell: Interesting thought. That is one potential solution, though that means that my working directory would need to be run, rather than project-a which is what I'm actually working on (and which contains my Git repo, Makefile, and so on).
• Amber: Yes, but if you "have 4 windows open" anyways, there's no reason the other windows can't be in the run/ directory while your editor windows are in the source directory.
• Brian Campbell: True. This is definitely a solution worth considering, but it feels a little hacky like the Makefile solution, and it's an extra setup step to give other developers (and the continuous integration system, and so on), as well as something they'd need to maintain and update when using new versions that depend on new packages. You could write a script that sets up the "run" directory, but you need to remember to run that, and re-run it any time the list of packages changes. Something that automatically got the path right as soon as you check out the code would be preferable.
• Amber: If you want something cohesive to give to other developers, then perhaps multiple repositories isn't the way to go? Alternatively, you could have a parent repository which contains all of the others as submodules, and includes the run/ folder as well.
• Brian Campbell: Multiple repositories is pretty deeply embedded in our workflow at this point, and is the right thing as they are independent projects. I've used submodules before, and find them to generally be more of a pain than they're worth.