Python modules with submodules and functions

Question

I had a question on how libraries like numpy work. When I import numpy, I'm given access to a host of built in classes, functions, and constants such as numpy.array, numpy.sqrt etc.

But within numpy there are additional submodules such as numpy.testing.

How is this done? In my limited experience, modules with submodules are simply folders with a __init__.py file, while modules with functions/classes are actual python files. How does one create a module "folder" that also has functions/classes?

Answer 1

A folder with .py files and a __init__.py is called a package. One of those files containing classes and functions is a module. Folder nesting can give you subpackages.

So for example if I had the following structure:

 mypackage
 __init__.py
 module_a.py
 module_b.py
 mysubpackage
 __init__.py
 module_c.py
 module_d.py

I could import mypackage.module_a or mypackage.mysubpackage.module_c and so on.

You could also add functions to mypackage (like the numpy functions you mentioned) by placing that code in the __init__.py. Though this is usually considered to be ugly.

If you look at numpy's __init__.py you will see a lot of code in there - a lot of this is defining these top-level classes and functions. The __init__.py code is the first thing executed when the package is loaded.

• ImpGuard: Then how is it that a library like numpy or scipy can be imported, and contain both classes/functions (like a module) and other modules (like a package). So I can do numpy.array (a class) or numpy.testing.assert... (getting a module).
• Mike Vella: I've just answered that, it's all in the __init__.py. Any function in there will be a first-class member of the package when it is loaded by the interpreter.
• ImpGuard: Ah, so that's what I was wondering. I thought it would be considered ugly since a lot of misc. logic would go in it. I presume everything was separately coded and somehow all combined into init.py after? It seems quite useful to have something like this but I'm not sure how to replicate it without just dumping lots of code in one file.
• ImpGuard: Ah, I got it! So if I import something in the init.py file, those items become first class members of the package as well! However, just be really clear, that does mean that if I separated my classes into separate files (to be more organized) and just imported them all when the package is loaded, I still technically have those files as modules that are "hidden"? Meaning if the people who made numpy were organized and put np.matrix in some submodule under numpy and simply imported it when numpy is imported, there might be some hidden numpy.__matrix module that I don't know about.
• Mike Vella: nothing in python is ever really "hidden", but yes you are correct if I understand you. You should make some packages and play around,it's the easiest way to learn this sort of thing!