homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [C API] Prepare PyTypeObject structure for a stable ABI: avoid accessing members in the public API
Type: Stage: resolved
Components: C API Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: corona10, erlendaasland, miss-islington, rhettinger, ronaldoussoren, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020年04月03日 11:57 by vstinner, last changed 2022年04月11日 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19375 closed vstinner, 2020年04月04日 21:58
PR 19376 merged vstinner, 2020年04月04日 22:02
PR 19377 merged vstinner, 2020年04月04日 22:13
PR 19378 merged vstinner, 2020年04月04日 22:23
PR 19379 merged vstinner, 2020年04月04日 22:38
PR 19426 merged vstinner, 2020年04月07日 23:39
PR 19428 merged vstinner, 2020年04月08日 00:04
PR 19464 merged shihai1991, 2020年04月10日 22:35
PR 19541 closed shihai1991, 2020年04月15日 15:35
PR 21390 merged vstinner, 2020年07月08日 08:22
PR 21391 merged miss-islington, 2020年07月08日 09:02
PR 22375 merged vstinner, 2020年09月23日 10:52
PR 23153 closed vstinner, 2020年11月04日 19:42
PR 23235 merged shihai1991, 2020年11月11日 04:36
PR 23236 merged vstinner, 2020年11月11日 12:41
PR 24535 merged erlendaasland, 2021年02月15日 13:26
PR 24548 merged erlendaasland, 2021年02月16日 10:01
PR 24553 merged erlendaasland, 2021年02月16日 21:33
PR 24555 merged erlendaasland, 2021年02月17日 18:14
PR 29470 merged vstinner, 2021年11月08日 14:59
PR 29471 merged miss-islington, 2021年11月08日 16:42
PR 29472 merged miss-islington, 2021年11月08日 16:42
PR 30938 merged vstinner, 2022年01月27日 01:26
PR 30940 merged vstinner, 2022年01月27日 01:37
PR 30942 merged vstinner, 2022年01月27日 01:55
PR 30943 merged vstinner, 2022年01月27日 02:42
Messages (59)
msg365689 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月03日 11:57
Leaking the PyTypeObject structure in the C API indirectly causes ABI issue (especially for statically allocated types), cause practical issues when old fields are removed and new fields are added (ex: tp_vectorcall addition and tp_print removal caused a lot of troubles with C code generated by Cython: see bpo-37250), prevents us to add feature and experiment optimization.
I don't expect that we will be able to make PyTypeObject opaque soon. The purpose of this issue is to track the work done towards this goal.
I propose to slowly prepare the Python code base, the C API and third party code (especially Cython) to make PyTypeObject structure opaque.
We have to identify most common code patterns which access directly PyTypeObject fields, provide helper functions, and ease the migration to solutions which don't access directly PyTypeObject.
See also bpo-39573: "Make PyObject an opaque structure in the limited C API" and bpo-39947 "Make the PyThreadState structure opaque (move it to the internal C API)".
Longer rationale about making structures of the C API opaque:
* https://pythoncapi.readthedocs.io/
* https://pythoncapi.readthedocs.io/bad_api.html
* https://pythoncapi.readthedocs.io/optimization_ideas.html
--
Multiple practical issues are preventing us to make PyTypeObject opaque right now.
(*) Many C extension modules are still using statically allocated types: there is an on-going effort in bpo-40077 to convert C extension modules one by one to PyType_FromSpec().
(*) Py_TYPE(obj)->tp_name is commonly accessed to format an error message. Example:
 PyErr_Format(PyExc_TypeError, "exec() globals must be a dict, not %.100s",
 Py_TYPE(globals)->tp_name);
I worked on bpo-34595 and started a discussion on python-dev to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():
https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53
Sadly, we failed to reach a consensus and I gave up on this idea. We should reconsider this idea.
We need to agree on how types should be formatted:
* just the name without any dot "type_name",
* qualified name "something.type_name",
* fully qualified name "module.something.type_name"
There is also the question of breaking applications which rely on the current exact error message. And the question of removing legacy "%.100s" which was used before Python was able to allocate a buffer large enough to arbitrary string length. When an error is formatted in pure Python, names are never truncated.
(*) Call the function of the parent type when a method is overriden in a subclass. Example with PyTypeObject.tp_free called in a deallocator:
static void
abc_data_dealloc(_abc_data *self)
{
 PyTypeObject *tp = Py_TYPE(self);
 ...
 tp->tp_free(self);
 Py_DECREF(tp);
}
(*) The PEP 384 provides the most generic PyType_GetSlot() but it's not convenient to use: need to handle error (NULL), need to cast the void* into the expected type (error prone cast), etc.
We should slowly add more and more helper functions for most common use cases. We can try to convert a few C extension modules of the Python stdlib to see which use cases are the most common.
Hopefully, many use cases are already abstracted by widely used functions like PyNumber_Add(), PySequence_Size(), etc.
(*) Likely other issues that I forgot.
msg365795 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月04日 21:45
Macros and static inline functions of the public C API which access directly PyTypeObject fields. There may be more.
#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )
static inline vectorcallfunc
PyVectorcall_Function(PyObject *callable)
{
 ...
 tp = Py_TYPE(callable);
 offset = tp->tp_vectorcall_offset;
 ...
}
#define PyObject_CheckBuffer(obj) \
 ((Py_TYPE(obj)->tp_as_buffer != NULL) && \
 (Py_TYPE(obj)->tp_as_buffer->bf_getbuffer != NULL))
#define PyIndex_Check(obj) \
 (Py_TYPE(obj)->tp_as_number != NULL && \
 Py_TYPE(obj)->tp_as_number->nb_index != NULL)
#define PyObject_GET_WEAKREFS_LISTPTR(o) \
 ((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))
static inline int
PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
#ifdef Py_LIMITED_API
 return ((PyType_GetFlags(type) & feature) != 0);
#else
 return ((type->tp_flags & feature) != 0);
#endif
}
#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )
msg365848 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月06日 12:07
New changeset 38aefc585f60a77d66f4fbe5a37594a488b53474 by Victor Stinner in branch 'master':
bpo-40170: PyObject_GET_WEAKREFS_LISTPTR() becomes a function (GH-19377)
https://github.com/python/cpython/commit/38aefc585f60a77d66f4fbe5a37594a488b53474
msg365850 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020年04月06日 12:55
Just manually verified that PR19377, when compiled against xlc - crashes during make:
	rm -f libpython3.9d.a
	ar rcs libpython3.9d.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/hamt.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/_abc.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/signalmodule.o Modules/_stat.o Modules/timemodule.o Modules/_threadmodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o
	./Modules/makexp_aix Modules/python.exp . libpython3.9d.a; xlc_r -Wl,-bE:Modules/python.exp -lld -o python Programs/python.o libpython3.9d.a -lintl -ldl -lm -lm 
	 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; then echo "generate-posix-vars failed" ; rm -f ./pybuilddir.txt ; exit 1 ; fi
Objects/genobject.c:127: _PyObject_GC_TRACK: Assertion "!(((PyGC_Head *)(op)-1)->_gc_next != 0)" failed: object already tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback
object address : 30084150
object refcount : 0
object type : 20013aa8
object type name: generator
object repr : <refcnt 0 at 30084150>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: core initialized
Current thread 0x00000001 (most recent call first):
 File "<frozen importlib._bootstrap_external>", line 1593 in _setup
 File "<frozen importlib._bootstrap_external>", line 1634 in _install
 File "<frozen importlib._bootstrap>", line 1189 in _install_external_importers
/bin/sh: 24117648 IOT/Abort trap(coredump)
make: 1254-004 The error code from the last command is 134.
Stop.
FYI: about two hours ago I verified that xlc and 08050e959e6c40839cd2c9e5f6a4fd1513e3d605 : bpo-40147: Fix a compiler warning on Windows in Python/compile.c (GH-19389)
all was green.
msg365852 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020年04月06日 13:22
Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 have any issues building.
msg365862 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月06日 15:24
Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc:
#define Py_TRASHCAN_BEGIN(op, dealloc) \
 Py_TRASHCAN_BEGIN_CONDITION(op, \
 Py_TYPE(op)->tp_dealloc == (destructor)(dealloc))
It should use PyType_GetSlot() or a new getter function (to read PyTypeObject.tp_dealloc) should be added.
msg365863 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月06日 15:25
> It should use PyType_GetSlot()
Oh. It seems like currently, PyType_GetSlot() can only be used on a heap allocated types :-( The function starts with:
 if (!PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE) || slot < 0) {
 PyErr_BadInternalCall();
 return NULL;
 }
msg365873 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月06日 16:52
> Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 have any issues building.
That sounds like an AIX specific issue. Please open a separated issue.
msg365956 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月07日 22:38
New changeset 9205520d8c43488696d66cbdd9aefbb21871c508 by Victor Stinner in branch 'master':
bpo-40170: PyObject_NEW() becomes an alias to PyObject_New() (GH-19379)
https://github.com/python/cpython/commit/9205520d8c43488696d66cbdd9aefbb21871c508
msg365960 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月07日 23:13
New changeset ef5c615f5ae72c4f6979159c94da46afefbfab9a by Victor Stinner in branch 'master':
bpo-40170: Convert PyObject_CheckBuffer() macro to a function (GH-19376)
https://github.com/python/cpython/commit/ef5c615f5ae72c4f6979159c94da46afefbfab9a
msg365961 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月07日 23:42
New changeset 45ec5b99aefa54552947049086e87ec01bc2fc9a by Victor Stinner in branch 'master':
bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)
https://github.com/python/cpython/commit/45ec5b99aefa54552947049086e87ec01bc2fc9a
msg365962 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月08日 00:02
New changeset a15e260b708a98edaba86a2aa663c3f6b2abc964 by Victor Stinner in branch 'master':
bpo-40170: Add _PyIndex_Check() internal function (GH-19426)
https://github.com/python/cpython/commit/a15e260b708a98edaba86a2aa663c3f6b2abc964
msg365964 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月08日 00:03
Hum, once most changes will land, maybe it would be worth it to document them at:
https://docs.python.org/dev/whatsnew/3.9.html#build-and-c-api-changes 
msg365966 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月08日 00:26
New changeset 307b9d0144e719b016a47fcc43397c070615e01e by Victor Stinner in branch 'master':
bpo-40170: Remove PyIndex_Check() macro (GH-19428)
https://github.com/python/cpython/commit/307b9d0144e719b016a47fcc43397c070615e01e
msg366152 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月10日 20:41
/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
 (PyType_IS_GC(Py_TYPE(o)) \
 && (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))
This macro should be converted to an opaque function.
msg366155 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020年04月10日 21:09
> This macro should be converted to an opaque function
Can I try it?
msg366417 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月14日 18:11
New changeset 675d9a3d7afc767a2818c84da7ba4bf4181dcf26 by Hai Shi in branch 'master':
bpo-40170: Convert PyObject_IS_GC() macro to a function (GH-19464)
https://github.com/python/cpython/commit/675d9a3d7afc767a2818c84da7ba4bf4181dcf26
msg366474 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月15日 01:38
On python-dev, Ronald Oussoren asked to add support for the buffer protocol in PyType_FromSpec() and PyTypeSpec API:
Ronald:
> BTW. This will require growing the PyTypeSpec ABI a little, there are features you cannot implement using that API for example the buffer protocol.
https://mail.python.org/archives/list/python-dev@python.org/message/PGKRW7S2IUOWVRX6F7RT6VAWD3ZPUDYS/
See also PyType_FromSpec() issue with opaque PyObject:
https://bugs.python.org/issue39573#msg366473 
msg366494 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020年04月15日 08:36
Something else that probably needs attention with the TypeSpec API is subclassing type in an extension when that subclass adds fields to the type object.
I use this in PyObjC to (dynamically) create types that have some additional data. I could probably work around this issue by adding a level of indirection (basically storing the extra data in a WeakKeyDictionary), but haven't looked into this yet.
msg366521 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020年04月15日 14:30
> Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc
Looks like this macro not recorded in docs. Do we need using function to replace this macro?
msg366522 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年04月15日 14:34
> Looks like this macro not recorded in docs.
It never prevented anyone to use a function of the C API :-)
msg366529 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020年04月15日 16:23
> It never prevented anyone to use a function of the C API :-)
Got it. If possible someone uses it, I will try to add a function to repalce it(MAYBE udpate this docs too;) )
msg372050 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年06月22日 08:39
> Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc (...) currently, PyType_GetSlot() can only be used on a heap allocated types
I created bpo-41073: [C API] PyType_GetSlot() should accept static types.
msg372054 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年06月22日 09:04
PyIter_Check() and PySequence_ITEM() macros access directly PyTypeObject members and must be converted to opaque functions:
#define PyIter_Check(obj) \
 (Py_TYPE(obj)->tp_iternext != NULL && \
 Py_TYPE(obj)->tp_iternext != &_PyObject_NextNotImplemented)
#define PySequence_ITEM(o, i)\
 ( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )
msg372056 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年06月22日 09:10
> PyIter_Check() and PySequence_ITEM() macros access directly PyTypeObject members and must be converted to opaque functions: (...)
PyIter_Check() and PySequence_ITEM() are declared as functions in the limited C API, but overriden with macros in the CPython C API.
I suggest to simply remove the macros to always declare them as functions.
See bpo-33738 "PyIndex_Check conflicts with PEP 384" which added the functions.
See also Tools/scripts/pep384_macrocheck.py script.
msg373293 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年07月08日 09:02
New changeset b26a0db8ea2de3a8a8e4b40e69fc8642c7d7cb68 by Victor Stinner in branch 'master':
Revert "bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)" (GH-21390)
https://github.com/python/cpython/commit/b26a0db8ea2de3a8a8e4b40e69fc8642c7d7cb68
msg373295 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年07月08日 09:06
> New changeset 45ec5b99aefa54552947049086e87ec01bc2fc9a by Victor Stinner in branch 'master':
> bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)
This change causes performance issues on macOS, see discussion starting at:
https://bugs.python.org/issue39542#msg372962
So I reverted the change. I will wait until my PEP 620 is accepted before considering to reapply it.
If it's reapplied, we have to make sure that Python internals currently using PyTuple_Check() still access directly PyTypeObject.tp_flags member. For example, a new _PyTuple_Check() function could be added and uses the internal _PyType_HasFeature() function.
msg373298 - (view) Author: miss-islington (miss-islington) Date: 2020年07月08日 09:19
New changeset a0a6f1167834c87f12e2eca11dd77143103e7691 by Miss Islington (bot) in branch '3.9':
Revert "bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)" (GH-21390)
https://github.com/python/cpython/commit/a0a6f1167834c87f12e2eca11dd77143103e7691
msg373338 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020年07月08日 19:32
Thanks for doing this. I can confirm the performance regression is fixed and that clean code is being generated for PyTuple_Check().
BTW, I support your efforts — just wanted to make sure we didn't unintentionally take a step backwards.
msg373346 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年07月08日 20:46
> Thanks for doing this. I can confirm the performance regression is fixed and that clean code is being generated for PyTuple_Check().
Thanks for checking!
> BTW, I support your efforts — just wanted to make sure we didn't unintentionally take a step backwards.
I tried to avoid changes which could affect performances.
I wrote PEP 620 for such changes.
msg377372 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年09月23日 12:08
New changeset 97d15ae1d8411b49b1fcdc0c67c51849dccce9c9 by Victor Stinner in branch 'master':
bpo-40170: Use inline _PyType_HasFeature() function (GH-22375)
https://github.com/python/cpython/commit/97d15ae1d8411b49b1fcdc0c67c51849dccce9c9
msg380757 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年11月11日 13:28
New changeset ba2958ed40d284228836735cbed4a155190e0998 by Victor Stinner in branch 'master':
bpo-40170: Fix PyType_Ready() refleak on static type (GH-23236)
https://github.com/python/cpython/commit/ba2958ed40d284228836735cbed4a155190e0998
msg381775 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020年11月24日 22:03
New changeset ed1a5a5baca8f61e9a99c5be3adc16b1801514fe by Hai Shi in branch 'master':
bpo-40170: Hide impl detail of Py_TRASHCAN_BEGIN macro (GH-23235)
https://github.com/python/cpython/commit/ed1a5a5baca8f61e9a99c5be3adc16b1801514fe
msg385176 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年01月18日 08:18
See also bpo-41618: [C API] How many slots of static types should be exposed in PyType_GetSlot().
msg387057 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021年02月15日 20:33
After GH-24533 and GH-24535, there's (AFAICS) only a handful of cases left.
$ grep -r "\->tp_" Include/:
Include/internal/pycore_abstract.h: PyNumberMethods *tp_as_number = Py_TYPE(obj)->tp_as_number;
Include/internal/pycore_interp.h: unsigned int version; // initialized from type->tp_version_tag
Include/internal/pycore_object.h: return ((type->tp_flags & feature) != 0);
Include/internal/pycore_object.h: Py_ssize_t offset = Py_TYPE(op)->tp_weaklistoffset;
Include/internal/pycore_object.h: && (Py_TYPE(obj)->tp_is_gc == NULL
Include/internal/pycore_object.h: || Py_TYPE(obj)->tp_is_gc(obj)));
Include/cpython/pyerrors.h:#define PyExceptionClass_Name(x) (((PyTypeObject*)(x))->tp_name)
Include/cpython/abstract.h: offset = tp->tp_vectorcall_offset;
Include/cpython/abstract.h: ( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )
Include/cpython/object.h: ((PyMemberDef *)(((char *)etype) + Py_TYPE(etype)->tp_basicsize))
Include/cpython/objimpl.h:#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )
Include/cpython/objimpl.h: _Py_SIZE_ROUND_UP((typeobj)->tp_basicsize + \
Include/cpython/objimpl.h: (nitems)*(typeobj)->tp_itemsize, \
Include/cpython/objimpl.h:#define PyType_SUPPORTS_WEAKREFS(t) ((t)->tp_weaklistoffset > 0)
Include/object.h: flags = type->tp_flags;
Should we strive to fix the cases in Include/internal as well?
msg387059 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月15日 20:42
> Should we strive to fix the cases in Include/internal as well?
No. The internal C API access directly to structure members on purpose, for best performances.
msg387092 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月16日 07:50
New changeset 871eb4237b9be95263ca13ba8856e78344eb9eba by Erlend Egeberg Aasland in branch 'master':
bpo-40170: Convert PyDescr_IsData() to static inline function (GH-24535)
https://github.com/python/cpython/commit/871eb4237b9be95263ca13ba8856e78344eb9eba
msg387113 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月16日 15:06
New changeset cc54001c2eb3b14320c1667b22602d69c90d5865 by Erlend Egeberg Aasland in branch 'master':
bpo-40170: Always define PyIter_Check() as a function (GH-24548)
https://github.com/python/cpython/commit/cc54001c2eb3b14320c1667b22602d69c90d5865
msg387115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月16日 15:06
> New changeset cc54001c2eb3b14320c1667b22602d69c90d5865 by Erlend Egeberg Aasland in branch 'master':
> bpo-40170: Always define PyIter_Check() as a function (GH-24548)
For macOS which doesn't use LTO compiler optimization, we added private static inline functions of some "Check" functions. But I don't think that it's worth it here (I don't think that the function is commonly called in "hot code").
msg387129 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021年02月16日 21:03
For PyExceptionClass_Name: Is it ok to just remove the macro version (like with GH-24548)?
msg387131 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月16日 21:28
> For PyExceptionClass_Name: Is it ok to just remove the macro version (like with GH-24548)?
Yes, I think so.
msg387133 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021年02月16日 22:33
Thanks, Victor.
For PySequence_ITEM, I guess adding a private C version (for example _PySequence_Item) and redirecting the macro to the C version would be acceptable.
Ditto for PyHeapType_GET_MEMBERS and PyType_SUPPORTS_WEAKREFS.
msg387140 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月17日 10:51
New changeset cd80f430daa7dfe7feeb431ed34f88db5f64aa30 by Erlend Egeberg Aasland in branch 'master':
bpo-40170: Always define PyExceptionClass_Name() as a function (GH-24553)
https://github.com/python/cpython/commit/cd80f430daa7dfe7feeb431ed34f88db5f64aa30
msg387142 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月17日 10:55
> For PySequence_ITEM, I guess adding a private C version (for example _PySequence_Item) and redirecting the macro to the C version would be acceptable.
This can introduce a performance slowdown and so should wait until the PEP 620 is accepted.
msg387146 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021年02月17日 11:32
> This can introduce a performance slowdown and so should wait until the PEP 620 is accepted.
Noted.
msg387177 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年02月17日 18:15
New changeset 630264a152115f9671d6b793455ef5c2cea09a97 by Erlend Egeberg Aasland in branch 'master':
 bpo-40170: Move 3 NEWS entries to the C API section (GH-24555)
https://github.com/python/cpython/commit/630264a152115f9671d6b793455ef5c2cea09a97
msg405956 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年11月08日 16:42
New changeset 99c7e9853fa13af414168f179213e3d2fae03a45 by Victor Stinner in branch 'main':
bpo-40170: Update What's New in Python 3.9 (GH-29470)
https://github.com/python/cpython/commit/99c7e9853fa13af414168f179213e3d2fae03a45
msg405964 - (view) Author: miss-islington (miss-islington) Date: 2021年11月08日 17:05
New changeset 69b3de65adc691cc5ad66e70e5c7caf61c202c79 by Miss Islington (bot) in branch '3.10':
bpo-40170: Update What's New in Python 3.9 (GH-29470)
https://github.com/python/cpython/commit/69b3de65adc691cc5ad66e70e5c7caf61c202c79
msg405966 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021年11月08日 17:10
New changeset 80580f5ab85e3c45c8c5e8999963a891598d10bf by Miss Islington (bot) in branch '3.9':
bpo-40170: Update What's New in Python 3.9 (GH-29470) (GH-29472)
https://github.com/python/cpython/commit/80580f5ab85e3c45c8c5e8999963a891598d10bf
msg411816 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 00:46
TODO:
* Macros still accessing directly PyTypeObject members:
 * PyHeapType_GET_MEMBERS()
 * PySequence_ITEM()
 * _PyObject_SIZE()
 * _PyObject_VAR_SIZE()
 * PyType_SUPPORTS_WEAKREFS()
* Try again to apply "bpo-40170: PyType_HasFeature() now always calls PyType_GetFlags() (GH-19378)" which has been reverted?
* Py_TYPE(obj)->tp_name is still commonly used to format error messages: see rejected bpo-34595
Other TODO tasks which can be addressed in follow-up issues:
* Add buffer protocol to PyType_FromSpec()
* Stdlib C extensions still define static types: see bpo-40077
* 3rd party C extensions still define static types: PEP 630 and others propose heap types
msg411817 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 01:07
I searched for "_PyGC_FINALIZED" in top 5000 PyPI projects. It seems like only Cython is impacted.
ddtrace and guppy3 use directly the internal C API.
== Cython 0.29.26 ==
* Cython/Compiler/ModuleNode.py: finalised_check = '!_PyGC_FINALIZED(o)'
* Cython/Compiler/ModuleNode.py: '(!PyType_IS_GC(Py_TYPE(o)) || !_PyGC_FINALIZED(o))')
== ddtrace 0.57.3 ==
In ddtrace/profiling/collector/stack.pyx:
 IF PY_MINOR_VERSION >= 9:
 # Needed for accessing _PyGC_FINALIZED when we build with -DPy_BUILD_CORE
 cdef extern from "<internal/pycore_gc.h>":
 pass
== guppy3-3.1.2 ==
In src/heapy/hv.c:
#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 9
# define Py_BUILD_CORE
/* PyGC_Head */
# undef _PyGC_FINALIZED
# include <internal/pycore_gc.h>
# undef Py_BUILD_CORE
#endif
msg411818 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 01:10
I searched for "_PyObject_DebugMallocStats" in top 5000 PyPI projects. There is a single project using it: guppy3.
Extract of guppy3 src/heapy/xmemstats.c:
...
 dlptr__PyObject_DebugMallocStats = addr_of_symbol("_PyObject_DebugMallocStats");
...
static PyObject *
hp_xmemstats(PyObject *self, PyObject *args)
{
 if (dlptr__PyObject_DebugMallocStats) {
 fprintf(stderr, "======================================================================\n");
 fprintf(stderr, "Output from _PyObject_DebugMallocStats()\n\n");
 dlptr__PyObject_DebugMallocStats(stderr);
 }
 ...
}
addr_of_symbol() is implemented with dlsym() or GetModuleHandle(NULL)+GetProcAddress(): it searchs for the symbol in the current process.
msg411819 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 01:16
In the top 5000 PyPI projects, the _PyObject_SIZE() and _PyObject_VAR_SIZE() functions are used by 7 projects:
* Cython-0.29.26
* frozendict-2.2.0: implement "sizeof" function, found in copies of Objects/dictobject.c file
* JPype1-1.3.0
* numpy-1.22.1: gentype_alloc() in numpy/core/src/multiarray/scalartypes.c.src, used as type tp_alloc functions
* pickle5-0.0.12
* pyobjc-core-8.2
* recordclass-0.17.1
msg411822 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 02:01
New changeset af32b3ef1fbad3c2242627a14398320960a0cb45 by Victor Stinner in branch 'main':
bpo-40170: PyType_SUPPORTS_WEAKREFS() becomes a regular function (GH-30938)
https://github.com/python/cpython/commit/af32b3ef1fbad3c2242627a14398320960a0cb45
msg411824 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 02:35
New changeset 6b491b9dc0b0fdfd1f07ea4e2151236186d8e7e6 by Victor Stinner in branch 'main':
bpo-40170: Remove _Py_GetAllocatedBlocks() function (GH-30940)
https://github.com/python/cpython/commit/6b491b9dc0b0fdfd1f07ea4e2151236186d8e7e6
msg411919 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 20:23
New changeset 0575551f69ba9c999835bfb176a543d468083c03 by Victor Stinner in branch 'main':
bpo-40170: Move _Py_GetAllocatedBlocks() to pycore_pymem.h (GH-30943)
https://github.com/python/cpython/commit/0575551f69ba9c999835bfb176a543d468083c03
msg411939 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月27日 23:40
New changeset 18ea973c21ee4a6adc26be41027881043fa498eb by Victor Stinner in branch 'main':
bpo-40170: Remove PyHeapType_GET_MEMBERS() macro (GH-30942)
https://github.com/python/cpython/commit/18ea973c21ee4a6adc26be41027881043fa498eb
msg411998 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月28日 14:19
Changes already done:
* Macros converted to regular functions:
 
 * PyObject_GET_WEAKREFS_LISTPTR(); add internal inline _PyObject_GET_WEAKREFS_LISTPTR()
 * PyType_SUPPORTS_WEAKREFS(); add internal inline _PyType_SUPPORTS_WEAKREFS()
 * PyObject_CheckBuffer(); no fast internal API
 * PyObject_IS_GC(); no fast internal API
 * PyDescr_IsData(); no fast internal API
* Always implemented as a function, remove macro optimization in the non-limited API:
 * PyIter_Check(); no fast internal API
 * PyIndex_Check(); add internal inline _PyIndex_Check()
 * PyExceptionClass_Name(); no fast internal API
* Py_TRASHCAN_BEGIN() macro now calls _PyTrash_cond() function: no longer read directly tp_dealloc member
* PyObject_NEW() macro becomes an alias to PyObject_New()
* Remove PyHeapType_GET_MEMBERS() (move it to the internal C API)
PyType_HasFeature() is left unchanged since the change caused performance issue on macOS, whereas Python is not built with LTO.
msg411999 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022年01月28日 14:25
I close the issue. While this issue is not fully fixed, it's a milestone of my "stable ABI" goal. I prefer to address remaining issues in new separted issues.
This issues is not fully fixed, there are are still a 4 macros which access directly PyTypeObject members:
* PySequence_ITEM()
* _PyObject_SIZE()
* _PyObject_VAR_SIZE()
* PyType_HasFeature() (if Py_LIMITED_API is not defined)
PySequence_ITEM() and PyType_HasFeature() are important for performance, I prefer to have a PEP before changing these two functions.
_PyObject_SIZE() and _PyObject_VAR_SIZE() should be made public with a better name like PyObject_SizeOf() and PyVarObject_SizeOf(). But I prefer to do that in a separated issue.
IMO it would be interesting to merge the PyHeapTypeObject structure into the PyTypeObject structure:
https://bugs.python.org/issue46433#msg411167
And make the PyTypeObject opaque: deprecate static types and promote the usage of heap types in C extensions. That's a big project which may be splitted into multiple issues and the final change may need its own PEP.
History
Date User Action Args
2022年04月11日 14:59:29adminsetgithub: 84351
2022年01月28日 14:25:55vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg411999

stage: patch review -> resolved
2022年01月28日 14:22:41Michael.Feltsetnosy: - Michael.Felt
2022年01月28日 14:19:51vstinnersettitle: [C API] Make PyTypeObject structure an opaque structure in the public C API -> [C API] Prepare PyTypeObject structure for a stable ABI: avoid accessing members in the public API
2022年01月28日 14:19:20vstinnersetmessages: + msg411998
2022年01月27日 23:40:08vstinnersetmessages: + msg411939
2022年01月27日 20:23:37vstinnersetmessages: + msg411919
2022年01月27日 02:42:32vstinnersetpull_requests: + pull_request29122
2022年01月27日 02:35:55vstinnersetmessages: + msg411824
2022年01月27日 02:01:06vstinnersetmessages: + msg411822
2022年01月27日 01:55:27vstinnersetpull_requests: + pull_request29121
2022年01月27日 01:37:45vstinnersetpull_requests: + pull_request29119
2022年01月27日 01:26:57vstinnersetpull_requests: + pull_request29117
2022年01月27日 01:16:56vstinnersetmessages: + msg411819
2022年01月27日 01:10:54vstinnersetmessages: + msg411818
2022年01月27日 01:07:40vstinnersetmessages: + msg411817
2022年01月27日 00:46:39vstinnersetmessages: + msg411816
2021年11月08日 17:10:49vstinnersetmessages: + msg405966
2021年11月08日 17:05:22miss-islingtonsetmessages: + msg405964
2021年11月08日 16:42:17miss-islingtonsetpull_requests: + pull_request27725
2021年11月08日 16:42:09miss-islingtonsetpull_requests: + pull_request27724
2021年11月08日 16:42:08vstinnersetmessages: + msg405956
2021年11月08日 14:59:54vstinnersetpull_requests: + pull_request27723
2021年02月17日 18:15:46vstinnersetmessages: + msg387177
2021年02月17日 18:14:45erlendaaslandsetpull_requests: + pull_request23341
2021年02月17日 11:32:42erlendaaslandsetmessages: + msg387146
2021年02月17日 10:55:54vstinnersetmessages: + msg387142
2021年02月17日 10:51:33vstinnersetmessages: + msg387140
2021年02月16日 22:33:21erlendaaslandsetmessages: + msg387133
2021年02月16日 21:33:56erlendaaslandsetpull_requests: + pull_request23335
2021年02月16日 21:28:43vstinnersetmessages: + msg387131
2021年02月16日 21:03:33erlendaaslandsetmessages: + msg387129
2021年02月16日 15:06:46vstinnersetmessages: + msg387115
2021年02月16日 15:06:02vstinnersetmessages: + msg387113
2021年02月16日 10:01:45erlendaaslandsetpull_requests: + pull_request23331
2021年02月16日 07:50:11vstinnersetmessages: + msg387092
2021年02月15日 20:42:54vstinnersetmessages: + msg387059
2021年02月15日 20:33:52erlendaaslandsetmessages: + msg387057
2021年02月15日 13:26:07erlendaaslandsetnosy: + erlendaasland
pull_requests: + pull_request23322
2021年01月18日 08:18:32vstinnersetmessages: + msg385176
2020年11月24日 22:03:35vstinnersetmessages: + msg381775
2020年11月11日 13:28:00vstinnersetmessages: + msg380757
2020年11月11日 12:41:32vstinnersetpull_requests: + pull_request22133
2020年11月11日 04:36:58shihai1991setpull_requests: + pull_request22132
2020年11月04日 19:42:29vstinnersetpull_requests: + pull_request22065
2020年09月23日 12:08:41vstinnersetmessages: + msg377372
2020年09月23日 10:52:37vstinnersetpull_requests: + pull_request21414
2020年07月08日 22:21:48petdancesetnosy: - petdance
2020年07月08日 20:46:37vstinnersetmessages: + msg373346
2020年07月08日 19:32:51rhettingersetnosy: + rhettinger
messages: + msg373338
2020年07月08日 09:19:41miss-islingtonsetmessages: + msg373298
2020年07月08日 09:06:12vstinnersetmessages: + msg373295
2020年07月08日 09:02:41miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request20537
2020年07月08日 09:02:36vstinnersetmessages: + msg373293
2020年07月08日 08:22:56vstinnersetpull_requests: + pull_request20536
2020年06月22日 09:10:40vstinnersetmessages: + msg372056
2020年06月22日 09:04:46vstinnersetmessages: + msg372054
2020年06月22日 08:39:12vstinnersetmessages: + msg372050
versions: + Python 3.10, - Python 3.9
2020年04月15日 16:23:03shihai1991setmessages: + msg366529
2020年04月15日 15:35:16shihai1991setpull_requests: + pull_request18889
2020年04月15日 14:34:39vstinnersetmessages: + msg366522
2020年04月15日 14:30:16shihai1991setmessages: + msg366521
2020年04月15日 08:36:15ronaldoussorensetnosy: + ronaldoussoren
messages: + msg366494
2020年04月15日 01:38:49vstinnersetmessages: + msg366474
2020年04月14日 18:11:24vstinnersetmessages: + msg366417
2020年04月10日 22:35:49shihai1991setpull_requests: + pull_request18819
2020年04月10日 21:09:19shihai1991setmessages: + msg366155
2020年04月10日 20:41:48vstinnersetmessages: + msg366152
2020年04月08日 00:26:48vstinnersetmessages: + msg365966
2020年04月08日 00:04:05vstinnersetpull_requests: + pull_request18786
2020年04月08日 00:03:59vstinnersetmessages: + msg365964
2020年04月08日 00:02:04vstinnersetmessages: + msg365962
2020年04月07日 23:42:30vstinnersetmessages: + msg365961
2020年04月07日 23:39:40vstinnersetpull_requests: + pull_request18784
2020年04月07日 23:13:57vstinnersetmessages: + msg365960
2020年04月07日 22:38:19vstinnersetmessages: + msg365956
2020年04月06日 16:52:08vstinnersetmessages: + msg365873
2020年04月06日 15:25:55vstinnersetmessages: + msg365863
2020年04月06日 15:24:18vstinnersetmessages: + msg365862
2020年04月06日 13:22:48Michael.Feltsetmessages: + msg365852
2020年04月06日 12:55:56Michael.Feltsetnosy: + Michael.Felt
messages: + msg365850
2020年04月06日 12:07:10vstinnersetmessages: + msg365848
2020年04月04日 22:38:49vstinnersetpull_requests: + pull_request18741
2020年04月04日 22:23:21vstinnersetpull_requests: + pull_request18740
2020年04月04日 22:13:13vstinnersetpull_requests: + pull_request18739
2020年04月04日 22:02:59vstinnersetpull_requests: + pull_request18738
2020年04月04日 21:58:13vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request18737
2020年04月04日 21:45:49vstinnersetmessages: + msg365795
2020年04月04日 14:12:15shihai1991setnosy: + shihai1991
2020年04月03日 18:58:02petdancesetnosy: + petdance
2020年04月03日 11:58:24corona10setnosy: + corona10
2020年04月03日 11:57:03vstinnercreate

AltStyle によって変換されたページ (->オリジナル) /