How do I install a Python extension module using distutils?

Question

I'm working on a Python package named "lehmer" that includes a bunch of extension modules written in C. Currently, I have a single extension module, "rng". I am using Python's Distutils to build and install the module. I can compile and install the module, but when I try to import the module using import lehmer.rng or from lehmer import rng, the Python interpreter throws an ImportError exception. I can import "lehmer" fine.

Here are the contents of my setup.py file:

from distutils.core import setup, Extension
exts = [Extension("rng", ["lehmer/rng.c"])]
setup(name="lehmer",
 version="0.1",
 description="A Lehmer random number generator",
 author="Steve Park, Dave Geyer, and Michael Dippery",
 maintainer="Michael Dippery",
 maintainer_email="[email protected]",
 packages=["lehmer"],
 ext_package="lehmer",
 ext_modules=exts)

When I list the contents of Python's site-packages directory, I see the following:

th107c-4 lehmer $ ls /scratch/usr/lib64/python2.5/site-packages/lehmer
__init__.py __init__.pyc rng.so*

My PYTHONPATH environment variable is set correctly, so that's not the problem (and as noted before, I can import lehmer just fine, so I know that PYTHONPATH is not the issue). Python uses the following search paths (as reported by sys.path):

['', '/scratch/usr/lib64/python2.5/site-packages', '/usr/lib/python25.zip', '/usr/lib64/python2.5', '/usr/lib64/python2.5/plat-linux2', '/usr/lib64/python2.5/lib-tk', '/usr/lib64/python2.5/lib-dynload', '/usr/lib64/python2.5/site-packages', '/usr/lib64/python2.5/site-packages/Numeric', '/usr/lib64/python2.5/site-packages/PIL', '/usr/lib64/python2.5/site-packages/SaX', '/usr/lib64/python2.5/site-packages/gtk-2.0', '/usr/lib64/python2.5/site-packages/wx-2.8-gtk2-unicode', '/usr/local/lib64/python2.5/site-packages']

Update

It works when used on an OpenSUSE 10 box, but the C extensions still fail to load when tested on Mac OS X. Here are the results from the Python interpreter:

>>> sys.path
['', '/usr/local/lib/python2.5/site-packages', '/opt/local/lib/python25.zip', '/opt/local/lib/python2.5', '/opt/local/lib/python2.5/plat-darwin', '/opt/local/lib/python2.5/plat-mac', '/opt/local/lib/python2.5/plat-mac/lib-scriptpackages', '/opt/local/lib/python2.5/lib-tk', '/opt/local/lib/python2.5/lib-dynload', '/opt/local/lib/python2.5/site-packages']
>>> from lehmer import rng
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: cannot import name rng
>>> import lehmer.rngs
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: No module named rngs
>>> import lehmer.rng 
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: No module named rng
>>> from lehmer import rngs
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: cannot import name rngs

• S.Lott: Actually, you don't know that PYTHONPATH is set correctly. Your local directory is always first on the sys.path list. Could you run Python and run 'import sys; print sys.path` and add that to your question?
• TFKyle: sorry for the stupid question, but what exactly is the ImportError's message? iirc ImportErrors are thrown for problems when loading extension modules so it could be caused by forgetting to link a library fex.
• mipadi: Whoa! When I tried to import it again, it magically worked. I swear I didn't touch anything, or do anything differently than what I had been doing...although I guess I must've done something differently. Cookie to anyone who might offer an explanation as to why. :)
• mipadi: Isn't "" always in the path first?
• S.Lott: @mipadi: Yes, '' is usually first (not always, but usually) and that means that the local directory is examined for your module before any other directory is checked. Developers often test their imports in the same place they're developing, masking PATH problems.

Answer 1

For the record (and because I am tired of seeing this marked as unanswered), here were the problems:

1. Since the current directory is automatically added to the Python packages path, the interpreter was first looking in the current directory for packages; since some C modules were not compiled in the current directory, the interpreter couldn't find them. Solution: Don't launch the interpreter from the same directory in which your working copy of the code is stored.
2. Distutils did not install the module with the correct permissions on OS X. Solution: Fix the permissions.

• Setjmp: Is there a way to make sure the installation proceeds with proper permissions instead of changing them after the fact?
• merwok: Yes, please define what the correct permissions should be and what distutils does. If there is a distutils bug, it should be reported to bugs.python.org.