[Python-Dev] Make extension module initialisation more like Python module initialisation

Thu Nov 8 14:51:54 CET 2012

Stefan Behnel, 08.11.2012 14:20:
> M.-A. Lemburg, 08.11.2012 14:01:
>> On 08.11.2012 13:47, Stefan Behnel wrote:
>>> I suspect that this will be put into a proper PEP at some point, but I'd
>>> like to bring this up for discussion first. This came out of issues 13429
>>> and 16392.
>>>>>> http://bugs.python.org/issue13429
>>>>>> http://bugs.python.org/issue16392
>>>>>> Stefan
>>>>>>>>> The problem
>>> ===========
>>>>>> Python modules and extension modules are not being set up in the same way.
>>> For Python modules, the module is created and set up first, then the module
>>> code is being executed. For extensions, i.e. shared libraries, the module
>>> init function is executed straight away and does both the creation and
>>> initialisation. This means that it knows neither the __file__ it is being
>>> loaded from nor its package (i.e. its FQMN). This hinders relative imports
>>> and resource loading. In Py3, it's also not being added to sys.modules,
>>> which means that a (potentially transitive) re-import of the module will
>>> really try to reimport it and thus run into an infinite loop when it
>>> executes the module init function again. And without the FQMN, it's not
>>> trivial to correctly add the module to sys.modules either.
>>>>>> We specifically run into this for Cython generated modules, for which it's
>>> not uncommon that the module init code has the same level of complexity as
>>> that of any 'regular' Python module. Also, the lack of a FQMN and correct
>>> file path hinders the compilation of __init__.py modules, i.e. packages,
>>> especially when relative imports are being used at module init time.
>>>>>> The proposal
>>> ============
>>>>>> ... [callbacks] ...
>>>>>> Alternatives
>>> ============
>>> ...
>>> 3) The callback could be registered statically in the PyModuleDef struct by
>>> adding a new field. This is not trivial to do in a backwards compatible way
>>> because the struct would grow longer without explicit initialisation by
>>> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but
>>> would still break at least binary compatibility.
>>>> I think the above is the cleaner approach than the callback mechanism.
>> Oh, definitely.
>>>> There's no problem in adding new slots to the end of the PyModuleDef struct
>> - we've been doing that for years in many other structs :-)
>>>> All you have to do is bump the Python API version number.
>>>> (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details)
>> The difference is that this specific struct is provided by user code and
> (typically) initialised statically. There is no guarantee that user code
> that does not expect the additional field will initialise it to 0. Failing
> that, I don't see how we could trust its value in any way.

Hmm - you're actually right. In C, uninitialised fields in a static struct
are set to 0 automatically. Same case as the type structs. That makes your
objection perfectly valid. I'll rewrite and shorten the proposal.
Thanks!
Stefan