Distribute pre-compiled python extension module with distutils

Question

Quick one today: I'm learning the in's and out's of Pythons distutils library, and I would like to include a python extension module (.pyd) with my package. I know of course that the recommended way is to have distutils compile the extension at the time the package is created, but this is a fairly complex extension spanning many source files and referencing several external libs so it's going to take some significant playing to get everything working right.

In the meantime I have a known working build of the extension coming out of Visual Studio, and would like to use it in the installer as a temporary solution to allow me to focus on other issues. I can't specify it as a module, however, since those apparently must have an explicit .py extension. How could I indicate in my setup.py that I want to include a pre-compiled extension module?

(Python 3.1, if it matters)

Answer 1

Try a manifest template:

http://docs.python.org/distutils/sourcedist.html#specifying-the-files-to-distribute

• j-m: As of writing, a working URL for this is docs.python.org/3.11/distutils/…

Answer 2

I solved this by overriding Extension.build_extension:

setup_args = { ... }
if platform.system() == 'Windows':
 class my_build_ext(build_ext):
 def build_extension(self, ext):
 ''' Copies the already-compiled pyd
 '''
 import shutil
 import os.path
 try:
 os.makedirs(os.path.dirname(self.get_ext_fullpath(ext.name)))
 except WindowsError, e:
 if e.winerror != 183: # already exists
 raise
 shutil.copyfile(os.path.join(this_dir, r'..\..\bin\Python%d%d\my.pyd' % sys.version_info[0:2]), self.get_ext_fullpath(ext.name))
 setup_args['cmdclass'] = {'build_ext': my_build_ext }
setup(**setup_args)

• letmaik: Very nice! The answer should also mention the extension entry that goes into setup(..): ext_modules=[Extension("_my_extension", sources=[])]

Answer 3

I ran into this same issue while building an extension library using Python 3.7, CMake 3.15.3 and Swig 4.0.1 with Visual Studio 2017. The build system generates three files: mymodule.py, _mymodule.lib and _mymodule.pyd. After a bunch of trial and error, I found the following combination to work:

1. Create a 'setup.cfg' file that specifies that you are installing a package; e.g:

 [metadata]
 name = mymodule
 version = 1.0
 [options]
 include_package_data = True
 package_dir=
 =src
 packages=mymodule
 python_requires '>=3.7'
 [options.package_data]
 * = *.pyd

2. Change CMake to create a distribution directory structure as follows:

 setup.py
 setup.cfg
 src/
 mymodule/
 __init__.py
 _mymodule.pyd

3. The 'setup.py' file is then trivial:

 setup()

This requires having CMake rename the 'mymodule.py' output file to 'init.py'. I did this with the 'install' command in CMake:

 install (TARGETS ${SWIG_MODULE_${PROJECT_NAME}_REAL_NAME} DESTINATION "${CMAKE_BINARY_DIR}/dist/src/${PROJECT_NAME}")
 install (FILES 
 setup.py
 setup.cfg
 DESTINATION "${CMAKE_BINARY_DIR}/dist"
 )
 install (FILES
 ${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}.py
 DESTINATION "${CMAKE_BINARY_DIR}/dist/src/${PROJECT_NAME}"
 RENAME "__init__.py"

I believe that the key secret to having this work was to restructure the build output as a Python package rather than trying to have setup use the default build output of as a Python script.

Answer 4

I had the same issue. This is what has worked out for me. I have changed the name of the module, just for the sake of a simple example.

My setup is: Visual Studio 2017 Project which builds an extension with a final file name myextension.pyd

I then locally created the stubs for this module using stubgen from mypy module.

This is my file tree

myextension/__init__.pyi
myextension/submodule.pyi
setup.py
myextension.pyd

And this is the contents of setup.py

from setuptools import setup, Distribution
class BinaryDistribution(Distribution):
 def has_ext_modules(foo):
 return True
setup(
 name='myextension',
 version='1.0.0',
 description='myextension Wrapper',
 packages=['', 'myextension'],
 package_data={
 'myextension': ['*.pyi'],
 '': ['myextension.pyd'],
 },
 distclass=BinaryDistribution
)

After running pip wheel . I get a pretty nice wheel containing the extensions, along with the required stubs.

Answer 5

Try using package_data: http://docs.python.org/distutils/setupscript#installing-package-data

• Dilawar: package_data does not honor directory structure of precompiled modules.