homepage

Python 3.11 made many changes in PyFrameObject and PyThreadState structures. Code which accessed directly structure members doesn't build anymore. For example, "frame->f_code" is now fails because PyFrameObject.f_code member is done. I propose to document these changes and explain how to port code.
I will write a documentation PR.
== PyFrameObject changes ==
PyFrameObject now only creates a Python frame object on demand.
* f_code: removed, use PyFrame_GetCode() instead, warning: it returns a strong reference (Py_DECREF is needed)
* f_lineno: changed, use PyFrame_GetLineNumber()
* f_back: changed, use PyFrame_GetBack()
* f_builtins: removed, get the "f_builtins" attribute in Python
* f_globals: removed, get the "f_globals" attribute in Python
* f_locals: removed, get the "f_locals" attribute in Python
* f_valuesstack: removed
* f_stackdepth: removed
* f_gen: removed
* f_lasti: removed, get the "f_lasti" attribute in Python?
* f_iblock: removed
* f_state: removed
* f_blockstack: removed
* f_localsplus: removed
Accessing f_lineno and f_back doesn't fail with a compiler error, but these members are filled lazily. If PyFrame_GetLineNumber() is not called, it can return 0 even if the frame is running and has a line number. If PyFrame_GetBack() is not called, f_back is NULL even if the frame has a next outer frame.
== PyThreadState changes ==
* frame: removed, use PyThreadState_GetFrame(), warning: it returns a strong reference (Py_DECREF is needed)
* recursion_depth: removed, use (tstate->recursion_limit - tstate->recursion_remaining) instead
* stackcheck_counter: removed
* tracing: changed, use PyThreadState_EnterTracing() and PyThreadState_LeaveTracing(), added by bpo-43760
== Notes ==
We should also explain how to get new C API functions, like PyFrame_GetCode(), on older Python, and maybe suggest to use pythoncapi_compat to get them:
 https://github.com/pythoncapi/pythoncapi_compat
See also:
* bpo-39947: "[C API] Make the PyThreadState structure opaque (move it to the internal C API)"
* bpo-40421: "[C API] Add getter functions for PyFrameObject and maybe move PyFrameObject to the internal C API"
* bpo-43760: "The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)" -- add PyThreadState_EnterTracing()

> f_lineno: changed, use PyFrame_GetLineNumber()
Oh I was wrong, PyFrame_GetLineNumber() code is the same in Python 3.10. It was already a bad idea to read directly the f_lineno member in Python 3.10: PyFrame_GetLineNumber() should always be called.

Are there any C programming guides we can link to, that explain API use?
I would hope that competent C programmers would know not to read or write to undocumented fields. But if they come from a Python background, that might not be obvious.
If the advice for something is "use Python", we should probably add an API function.

New changeset 0885999a8e5ffad3fae0302675ad0030e33a15af by Victor Stinner in branch 'main':
 bpo-46355: Document PyFrameObject and PyThreadState changes (GH-30558)
https://github.com/python/cpython/commit/0885999a8e5ffad3fae0302675ad0030e33a15af

I merged my PR so the doc can be reviewed online at https://docs.python.org/dev/whatsnew/3.11.html once it will be rendered ;-)

> I would hope that competent C programmers would know not to read or write to undocumented fields. But if they come from a Python background, that might not be obvious.
Tread carefully when changing decades-old API, documented or not. The docs weren't always as useful and complete as they are now.
> If the advice for something is "use Python", we should probably add an API function.
Not really. If PyObject_GetAttrString doesn't add too much overhead, IMO it's fine to use it.

New changeset a4cb31927a1f0ee31025ea1ca82fcbfad44755dc by Petr Viktorin in branch 'main':
bpo-46355: What's New: Note that PyFrameObject are private (GH-31032)
https://github.com/python/cpython/commit/a4cb31927a1f0ee31025ea1ca82fcbfad44755dc

New changeset ba5725171d9c411fc4764349205eff5cfc028797 by Erlend Egeberg Aasland in branch 'main':
bpo-46355: Amend What's New in Python 3.11 C API wording (GH-31288)
https://github.com/python/cpython/commit/ba5725171d9c411fc4764349205eff5cfc028797

I created bpo-46836: "[C API] Move PyFrameObject to the internal C API".

New changeset ec4a580f7cada002441ae5611b909d56e3b5b613 by Victor Stinner in branch 'main':
bpo-46355: Update pythoncapi_compat project URL (GH-31670)
https://github.com/python/cpython/commit/ec4a580f7cada002441ae5611b909d56e3b5b613