Packages "within" modules

Question

I have a growing number of scripts that make up a program I am writing and decided it was time to clean up my source tree and package them up correctly. I'm sure this is a simple question but I can't find out how to do it.

If I have a group of modules, that fit together, but one should be a top-level module and the others should be prefixed by the module name but actually go into lower level packages, how can I do it.

For example say I wanted to be able to import mystuff, and get all mystuff. but I also should be able to import mystuff.test.test1. I thought I would create a source tree like this,

myprogram/
 mystuff.py
 mystuff/
 __init__.py
 tests/
 __init__.py
 test1.py
 test2.py
 ...

But in this case, it seems mystuff/ always takes precedence over mystuff.py, so import mystuff does nothing (as long as mystuff/'s __init__.py is empty).

What would be the correct approach to get the desired behaviour? Or is this not possible and must I move mystuff.py into mystuff/ and have to access it as mystuff.mystuff ( seem's like unnecessary repetion).

Sorry, if I've just missed something obvious. I assume this must be documented somewhere, but I can't seem to find where somewhere is.

Update. While I believe Ignacio's method is the correct one, I don't really like it! If I have multiple files open in my editor and they're all called __init__.py things could get messy. So, I have decided to leave my structure as it is, and link mystuff.py to mystuff/__init__.py. If anyone has any opinions on why I shouldn't be doing this, I'd like to hear them.

Actually in the end, I am linking the other way around, since I couldn't find a way to make distutils dereference symlinks when creating a tar.gz, and I ended up with broken links in my output. This way has the same effect and keeps it happy.

• cwallenpoole: Yes. You should not do that because Ignatio's response is the correct one.
• tjm: @cwallenpoole. Agreed, but at the end of the day the way I have it setup has exactly the same effect. Even the sdist output will be exactly the same as Ignacio's, without my links. So it just makes it easier for me whilst developing. Is there some other reason why this is a bad thing to do?

Answer 1

Everything in mystuff.py should be placed into mystuff/__init__.py instead.

• tjm: I thought about that. But it seemed like a bit of a nasty way to do it. Is that the recommended method though? If so, then that's what I'll do, or is it just "a way that will work"? Thanks for your answer.
• Ignacio Vazquez-Abrams: That's how it's done. Anything in foo/__init__.py will become part of foo when it is imported.
• tjm: Ok. That's great then. As easy as that :) Thanks very much. (Will accept when allowed)
• tjm: Probably should be a seperate question, but since it may just be a quick "yes/no" answer, I wonder if I can ask. If mystuff.py is actually a compiled mystuff.so (I'm using cython) will this still work. That is, will __init__.so be read if it exists?
• Ignacio Vazquez-Abrams: I don't know for certain, but I wouldn't try it. Make the C module separate and import from it in __init__.py.

Answer 2

You cannot have both a mystuff.py and a mystuff/ package.

You have two choices:

• put the code in mystuff.py into mystuff/__init__.py
• rename mystuff.py to, for example, mystuff/_stuff.py, and then import that into mystuff/__init__.py.

The second option looks something like this:

myprogram/
 mystuff.py -------
 mystuff/ \
 __init__.py /
 _stuff.py <---
 tests/
 __init__.py
 test1.py
 test2.py
 ...

and mystuff/__init__.py looks like:

from mystuff._stuff import *