How to organize a Python Project?

Question

I'm new to Python and I'm starting a mini Project, but I have some doubts on how to organize the folders in the "Python Way".

I'm using PyDev in my Development Environment, and when I create a new project a folder is created called src

+ src

Now, in the PyDev, I can create Pydev Module and PyDev Package

I need to organize my Project in the following way:

+ Indicators
 - Moving_averages.py
 - Stochastics.py
+ Strategies
 - Moving_averages_cross.py
- example.py

How can I organize this in terms of Modules and Packages? What is the meaning of Modules and Packages?

• LiorK: I can recommend this link, out of the official Python Tutorial. Is it pretty much clear and straight forward.

Answer 1

A Package is basically a folder with __init__.py file under it and usually some Modules, where Module is a *.py file. It has to do with import mainly. If you add __init__.py to Indicators you can use:

from Indicators.Stochastics import *

or

from Indicators import Stochastics

By the way, I would recommend to keep module/package names lowercase. It does not affect functionality but it's more "pythonic".

Answer 2

From a file system perspective, a module is a file ending with .py and a package is a folder containing modules and (nested) packages again. Python recognizes a folder as a package if it contains a __init__.py file.

A file structure like that

some/
 __init__.py
 foofoo.py
 thing/
 __init__.py
 barbar.py

defines the package some, which has a module foofoo and a nested package thing, which again has a module barbar. However, when using packages and modules, you don't really distinguish these two types:

import some
some.dothis() # dothis is defined in 'some/__init__.py'
import some.foofoo # <- module
import some.thing # <- package

Please follow PEP8 when selecting naming your packages/modules (i.e. use lower-case names).

• Jay Wang: Can I call the function from foofoo.py if I only import some? I am confused about the __init__.py, what should I put there?
• Jawad Mehmood: @JayWong It's not compulsory to put something in __init__.py, it helps python accept the current folder as a module and *.py files as packages. Whenever you import a module, the __init__.py file runs and executes whatever's in it.

Answer 3

See python-package-template

Directory structure

 .
 |-- bin
 | `-- my_program
 |-- docs
 | `-- doc.txt
 |-- my_program
 | |-- data
 | | `-- some_data.html
 | |-- __init__.py
 | |-- submodule
 | | `-- __init__.py
 | |-- helpers.py
 |-- tests
 | |-- __init__.py
 | |-- test_helpers.py
 |-- Makefile
 |-- CHANGES.txt
 |-- LICENSE.txt
 |-- README.md
 |-- requirements-dev.txt
 |-- requirements.txt
 `-- setup.py

cat Makefile

 PYTHON=`which python`
 NAME=`python setup.py --name`
 all: check test source deb
 init:
 pip install -r requirements.txt --use-mirrors
 dist: source deb
 source:
 $(PYTHON) setup.py sdist
 deb:
 $(PYTHON) setup.py --command-packages=stdeb.command bdist_deb
 rpm:
 $(PYTHON) setup.py bdist_rpm --post-install=rpm/postinstall --pre-uninstall=rpm/preuninstall
 test:
 unit2 discover -s tests -t .
 python -mpytest weasyprint
 check:
 find . -name \*.py | grep -v "^test_" | xargs pylint --errors-only --reports=n
 # pep8
 # pyntch
 # pyflakes
 # pychecker
 # pymetrics
 clean:
 $(PYTHON) setup.py clean
 rm -rf build/ MANIFEST dist build my_program.egg-info deb_dist
 find . -name '*.pyc' -delete

• Elias Dorneles: hi, Vitaly! I've just come across your project. Are you still maintaining it? Thanks!
• Dimitrios Mistriotis: Ended up with a 404
• Toothpick Anemone: The link https://github.com/vital-fadeev/python-package-template is no longer valid. Where can this python-package-template be found?

Answer 4

You might want to check out the modern-package-template libary. It provides a way to setup a really nice basic layout for a project that walks you through a few questions and tries to help you get something that's able to be distributed fairly easily.

http://pypi.python.org/pypi/modern-package-template

Answer 5

Before deciding on a project structure, it's good to ask yourself what the purpose of the project is going to be. Is this going to be one off analysis? A toy concept you want to investigate? A full blown project you intend to distribute? The amount of effort you want to put into structuring your project will be different.

• If it's a one off analysis, I like to use ipython notebooks. The notebook will capture the flow of your thoughts, and you can add notes in markup to your code for later reference.
• If it's a toy concept you want to investigate, I find a simple, quick approach to work best. You want to be able to quickly implement your concept to discover if it's even feasible and thus worth spending more time on it. Part of Python's philosophy is 'Don’t try for perfection because "good enough" is often just that.' You can always come back later and structure your project in a way that follows best software engineering practices.

• If you want to structure your project so you can later distribute it, and so that it scales to many modules I recommend the following structure:

  projectname
   ├── MANIFEST.in
   ├── setup.py
   ├── README
   ├── .gitignore
   ├── .git
   ├── projectname_env
   └── projectname
   ├── __init__.py
   ├── subpackageone
   │ ├── __init__.py
   │ ├── second_module.py
   │ ├── tests
   │ │ └── test_second_module.py
   │ └── models
   │ └── model1
   ├── first_module.py 
   └── tests
   └── test_second_module.py
  

The detailed reasons why I like this structure are in my blog post, but the basic gist is that the hierarchically lower level projectname directory contains your actual project. Alongside it are all the tools that help manage (git) and package (setup.py, MANIFEST.in) it.

• buhtz: How would you run your project? In the root folder doing python3 -m projectname?

Answer 6

A package is a directory with a __init__.py in it. The difference from a directory is that you can import it.

There isn't a "Python way" per se, but you'll find that it's a good idea to put all your modules in one package with a name related to the project.

Also, to follow the Python style guide, PEP8, the package and module names should be all lowercase. So, if we assume the project is called "Botond Statistics" your structure would be something like this:

botondstats/
 indicators/
 moving_averages.py
 stochastics.py
 strategies/
 moving_averages_cross.py
 example.py

You would then find the Stochastics class by doing

from botondstats.indicators.stochastics.Stochastics

(There are various ways to keep the structure but make imports shorter, but that's another question).

You can put this structure under src/ if you want to, but it's not necessary. I never do. Instead I have a main directory:

BotondStatistics/
 docs/
 botonstats/ # the above structure
 setup.py # Distutils/distribute configuration for packaging.

In this directory I also typically have a virtualenv so I actually also have bin/ lib/ et al. Development is typically done by running

./bin/python setup.py tests

As I use the Distrubute test runner to run the tests.

That's how I do it. :-)

Answer 7

Try python_boilerplate_template:

https://pypi.python.org/pypi/python_boilerplate_template

Answer 8

The cookiecutter project by audreyr includes several Python project templates:

• https://github.com/audreyr/cookiecutter#python

The package uses a single ~/.cookiecutterrc file to create custom project templates in Python, Java, JS, and other languages.

For example, a Python template compatible with PyPI:

• https://github.com/audreyr/cookiecutter-pypackage

cookiecutter-pypackage