Directory Structure for Importing Python Package using __init__.py

Question

I've got a python project (projectA), which I've included as a git submodule in a separate project (projectB) as a subfolder. The structure is laid out as

projectB/
 projectB_file.py
 projectA/ (repository)
 projectA/ (source code)
 module1/
 file1.py (contains Class1)
 file2.py (contains Class2)
 tests/
 test_file1.py

I'm trying to figure out how to layout __init__.py files so in projectB_file.py I can import Class1 and Class2 as

from projectA.module1 import Class1
from projectA import Class2

• Blckknght: I think the "right" answer is to put projectB/projectA into your module search path, then the imports as you've written them should just work. Without the path manipulation, you'll need projectA.projectA.module1 and so on.
• abnowack: Ah, hey that does help! I didn't consider that. That's much nicer, as I'm trying to figure out how to make projectA into a proper python package eventually, and wouldnt need that top level __init__.py. I just wish PyCharm would know what I'm doing and tab complete from projectA when using sys.path
• abnowack: Actually marking the top level projectA/ as a source directory in PyCharm also takes care of that issue, so I'm set! If you post your comment as an answer I'll mark it correct

Answer 1

I think having the top level projectA part of the import will be a mistake. That will require you to write your imports with projectA duplicated (e.g. import projectA.projectA.module1.file1).

Instead, you should add the top projectA folder to the module search path in some way (either as part of an install script for projectB, or with a setting in your IDE). That way, projectA as a top-level name will refer to the inner folder, which you actually intend to be the projectA package.

Answer 2

You'll need an __init__.py in each subdirectory you want to treat as a package. Which in your case means one in:

• projectA
• projectA/projectA
• projectA/projectA/module1
• projectA/projectA/tests

It would definitely be better you could lose that first projectA folder.

• abnowack: Not really sure how to do that, projectA is it's own separate repository. I have the subdirectory to place the actual project code apart from docs/ and tests/ etc. And including init.py everywhere doesn't quite do the trick. As it means including from projectA.projectA.module1.file1 import Class1
• Ethan Furman: @abnowack: If projectA is a separate project, why do you have it under projectB?
• abnowack: projectA is a generic library for interpreting a dataset I've written, projectB is a specific analysis I am doing for a paper.

Answer 3

This is an annoying issue. The main approach I use is to edit the PYTHONPATH. This is essentially how I do it.

ProjectB/ # Main Repository
 projectb/ # Package/library
 __init__.py
 projectB_file.py
 docs/
 tests/
 setup.py # Setup file to distribute the library
 freeze.py # CX_Freeze file to make executables
 ProjectA/ # Subrepository
 projecta/ # Package/library
 __init__.py
 projectA_file.py
 file2.py
 submodule/
 __init__.py
 file3.py
 pa_module1/ # Additional package/library in 1 project
 __init__.py
 file1.py
 docs/
 tests/
 setup.py

With this setup I have to edit the python path before running ProjectB's code. I use this structure, because it is easier for deployment and distribution. The subrepository helps track the version of ProjectA for each release of ProjectB.

cd ProjectB
export PYTHONPATH=${PWD}:${PWD}/ProjectA
python projectb/projectB_file.py

projectB_file.py

from projecta import projectA_file

This is for my development environment. For a release environment someone would install with the below code.

# Need a correct setup.py to handle 2 packages for pa_module1 and projecta
pip install ProjectA (install projecta to site-packages) 
cd ..
pip install ProjectB (install projectb to site-packages)

This means projectb and projecta are in site packages. From there any file can simply

from projectb import projectB_file
from projecta import projectA_file, file2
from pa_module1 import file1
# Personally I don't like imports that use project.sub.sub.sub ...
# I have seen it cause import confusion and problems, but it is useful in some situations.
from projecta.submodule import file3
#import projecta.projectA_file
#import projecta # if __init__.py has import code
file1.Class1()
file2.Class2()
file3.Class3()

Additionally with pip you can install a library as a developer environment which links to the real project directory instead of copying files to site-packages.

pip install -e ProjectA