Shared class and function code management in Python

Question

I'm new to Python for the last 3 months but have quite a bit of development experience. I'm trying to figure out a good way to manage a collection of functions and classes that are shared across several projects. I'm working on a Windows 10 machine that I don't have admin access to, so the PATH variable is not an option. I'm using VS Code with Python 3.10

I have several active projects and my current working directory structure is:

python

• Project A
• Project B
• Project C
• Common

•  __init__.py (empty)
  

•  ClassA.py
  

•  ClassB.py
  

•  functions.py
  

I've added a .pth file in AppData/Local/Programs/Python/Python310/Lib/site-packages which contains the path to the python root folder.

Right now I'm able to use this configuration by importing each file as a separate module:

from Common.ClassA import ClassA
from Common.ClassB import ClassB
from Common import functions as fn

I would like to do something like:

from Common import ClassA, ClassB, functions as fn

Just looking for some experienced advice on how to manage this situation. Thanks to any and all who have time to respond.

• paisanco: Some languages require a one to one mapping between classes and files, Python doesn't
• juanpa.arrivillaga: For starters, the simplest thing would eb to put all the class definitions in a common.py file. Otherwise, you have to import those classes into the __init__.py int he packages. But again, why are these classes in seperate files to begin with?

Answer 1

(disclaimer, I am an admin on my mac, but none of what I am doing here required sudo permissions).

One way to do that is to put your common code in a "package", say common and use pip to do an editable local install. via pip install -e common.

After installation, your Python path is modified so that it includes the directory where common lives and your project-side code can then use it like:

from common.classa import ClassA

Now, writing an installable package is (削除) not that trivial, but (削除ここまで) TRIVIAL nowadays and this is likely the more robust approach, over modifying pythonpath with .pth files - been there, done that myself.

Now, as far as what your imports can look like in your project A, B, C code you will find that many packages do an import of constituent files in their __init__.py.

common/__init__.py:

from .classa import ClassA
from .classb import ClassB
import .functions

which means ProjectB can use:

import common
a = common.ClassA()
res = common.functions.foobar(42)

You can look at sqlalchemy's init.py for that type of approach:

from .engine import AdaptedConnection as AdaptedConnection
from .engine import BaseRow as BaseRow
from .engine import BindTyping as BindTyping
from .engine import ChunkedIteratorResult as ChunkedIteratorResult
from .engine import Compiled as Compiled
from .engine import Connection as Connection

which my own code can then use as:

import sqlalchemy
...
if not isinstance(engine, sqlalchemy.engine.base.Engine):
 ... 

Note: none of this explanation should be taken as detracting from the comments and answers reminding you that Python can put any number of functions and classes into the same .py file. Python is not Java. But in practice, a Python file with over 400-500 lines of code is probably looking for a bit of refactoring. Not least because that facilitates git-based merging if those become relevant. And also because it facilitates code discovery: "Ah, a formatting question. Let's look in formatting.py"

OK, so how much work IS setting up a locally installed package?

TLDR: very little nowadays.

Let's take the package structure and Python files first

(this is under a directory called testpip)

├── common
│  ├── __init__.py
│  └── classa.py

common/__init__.py:

from .classa import A
from pathlib import Path
class B:
 def __repr__(self) -> str:
 return f"{self.__class__.__name__}"
 
 def info(self):
 pa = Path(__file__)
 print(f"{self} in {pa.relative_to(pa.parent)}")

common/classa.py:

class A:
 def __repr__(self) -> str:
 return f"{self.__class__.__name__}"
 
 def whoami(self):
 print(f"{self}")

Let's start with just that and try a pip install.

testpip % pip install -e .

Obtaining file:.../testpip
ERROR: file:.../testpip does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

OK, I know setup.py used to be complicated, but what about that pyproject.toml?

There's a write up about a minimal pyproject.toml here.

But that still seemed like a lot of stuff, so I ended up with.

echo "" > pyproject.toml. i.e. an empty pyproject.toml

(yes, a touch would do, but the OP is on Windows)

testpip % pip install -e .

 
Obtaining file:///.../testpip
 Installing build dependencies ... done
 Checking if build backend supports build_editable ... done
 Getting requirements to build editable ... done
 Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: common
 Building editable for common (pyproject.toml) ... done
 Created wheel for common: filename=common-0.0.0-0.editable-py3-none-any.whl size=2266 sha256=fe01aa92de3160527136d13a233bfd9ff92da973040981631a4bb8f372adbb0b
 Stored in directory: /private/var/folders/bk/_1cwm6dj3h1c0ptrhvr2v7dc0000gs/T/pip-ephem-wheel-cache-l3v8jbfr/wheels/1a/64/41/6ec6e2e75e362f2818c47a49356f82be33b0a6dba83b41354c
Successfully built common
Installing collected packages: common
Successfully installed common-0.0.0

And now, let's go to another directory and try it out.

src/testimporter.py:

from common import A, B
a = A()
a.whoami()
b = B()
b.info()

python testimporter.py:

A
B in __init__.py

The full project structure ended up as:

.
├── common 👈 your Python code
│  ├── __init__.py 
│  └── classa.py
├── common.egg-info 👈 generated by pip install -e .
│  ├── PKG-INFO
│  ├── SOURCES.txt
│  ├── dependency_links.txt
│  └── top_level.txt
├── pyproject.toml 👈 an EMPTY file to make pip install -e work

Answer 2

The easiest way would be to package everything in Common into a single .py file in the same folder as your projects.

The reasoning is that when you do

from Common.ClassA import ClassA

It looks in the Common folder, finds the ClassA file, and imports the ClassA class.

By organizing your directory structure like this:

• Project A
• Project B
• Project C
• Common.py

Then you can just run:

from Common import ClassA, ClassB, functions as fn

• juanpa.arrivillaga: "It looks in the Common folder, finds the ClassA file, and imports the ClassA class." Correction: it imports the Common.ClassA module, then puts the Common.ClassA.ClassA into the current namespace as ClassA You do not import objects like classes, you import modules