On 22. 04. 22 14:47, Fabio Zadrozny wrote:
Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin <[email protected]
<mailto:[email protected]>> escreveu:
Hello Fabio,
Let's talk a bit about which API should, exactly, be guaranteed to not
change across minor releases.
So far it looks like:
- PyEval_RequestCodeExtraIndex
- PyCode_GetExtra
- PyCode_SetExtra
- PyFrameEvalFunction
- PyInterpreterState_GetEvalFrameFunc
- PyInterpreterState_SetEvalFrameFunc
Do any more come to mind?
The issue with this set is that in 3.11, _PyFrameEvalFunction changes
its signature to take _PyInterpreterFrame rather than PyFrameObject.
Exposing _PyInterpreterFrame would be quite problematic. For example,
since it's not a PyObject, it has its own lifetime management that's
controlled by the interpreter itself,. And it includes several
pointers whose lifetime and semantics also isn't guaranteed (they
might be borrowed, cached or filled on demand). I don't think we can
make any guarantees on these, so the info needs to be accessed using
getter functions.
There is the function _PyFrame_GetFrameObject, which returns a
PyFrameObject.
I think it would be best to only expose _PyInterpreterFrame as an
opaque structure, and expose PyFrame_GetFrameObject so debuggers can
get a PyFrameObject from it.
Does that sound reasonable?
Humm, now I'm a bit worried... the approach the debugger is using gets
the PyFrameObject that's about to be executed and changes the
PyFrameObject.f_code just before the execution so that the new code is
executed instead.
From what you're saying the PyFrameObject isn't really used anymore
(apparently it's substituted by a _PyInterpreterFrame?)... in this case,
will this approach still let the debugger patch the code object in the
frame before it's actually executed?
PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it
adds PyObject metadata (type & refcount), and not much else. It's
allocated at most once for each _PyInterpreterFrame -- once it's created
it stays attached to the frame.
So, for the most heavily optimized code paths a PyFrameObject is not
allocated, but it's trivial to get it whenever it's needed.
-- i.e.: the debugger changes the state.interp.eval_frame to its own
custom evaluation function, but _PyEval_EvalFrameDefault is still what
ends up being called afterwards (it works more as a hook to change the
PyFrameObject.f_code prior to execution than as an alternate interpreter).
Ah, you also need PyEval_EvalFrameDefault exposed. The public version
would take PyFrameObject and pass its _PyInterpreterFrame to the
internal _PyEval_EvalFrameDefault.
On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny <[email protected]
<mailto:[email protected]>> wrote:
>
>
> Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny
<[email protected] <mailto:[email protected]>> escreveu:
>>>
>>> PEP 523 API added more private functions for code objects:
>>>
>>> * _PyEval_RequestCodeExtraIndex()
>>> * _PyCode_GetExtra()
>>> * _PyCode_SetExtra()
>>>
>>> The _PyEval_RequestCodeExtraIndex() function seems to be used
by the
>>> pydevd debugger. The two others seem to be unused in the wild.
I'm not
>>> sure if these ones should be moved to the internal C API. They
can be
>>> left unchanged, since they don't use a type only defined by the
>>> internal C API.
>>
>> Just to note, the pydevd/debugpy debuggers actually uses all of
those APIs.
>>
>> i.e.:
>>
>>
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
<https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187>
>>
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
<https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232>
>>
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
<https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311>
>>
>> The debugger already has workarounds because of changes to
evaluation api over time (see:
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491
<https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491>)
and I know 3.11 won't be different.
>>
>> I'm ok with changes as I understand that this is a special API
-- as long as there's still a way to use it and get the information
needed (the debugger already goes through many hops because it needs
to use many internals of CPython -- in every new release it's a
**really** big task to update to the latest version as almost
everything that the debugger relies to make debugging fast changes
across versions and I never really know if it'll be possible to
support it until I really try to do the port -- I appreciate having
less things in a public API so it's easier to have extensions work
in other interpreters/not recompiling on newer versions, but please
keep it possible to use private APIs which provides the same access
that CPython has to access things internally for special cases such
as the debugger).
>>
>> Maybe later on that PEP from mark which allows a better debugger
API could alleviate that (but until then, if possible I appreciate
it if there's some effort not to break things unless really needed
-- ideally with instructions on how to port).
>>
>> Anyways, to wrap up, the debugger already needs to be built with
`Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private
API (as long as it's still accessible in that case) is probably not
a big issue for the debugger and having setters/getters to set it
instead of relying on `state.interp.eval_frame` seems good to me.
>>
>> Cheers,
>>
>> Fabio
>>
>
>
> I think the main issue here is the compatibility across the same
version though... is it possible to have some kind of guarantee on
private APIs that something won't change across micro-releases?
>
> I.e.: having the frame evaluation function change across major
releases and having them be reworked seems reasonable, but then
having the frame evaluation be changed across micro-releases
wouldn't be.
>
> So, I'm ok in pushing things to the internal API, but then I
still would like guarantees about the compatibility of that API in
the same major release (otherwise those setters/getters/frame
evaluation should probably remain on the public API if the related
structure was moved to the internal API).
>
> Cheers,
>
> Fabio
> _______________________________________________
> Python-Dev mailing list -- [email protected]
<mailto:[email protected]>
> To unsubscribe send an email to [email protected]
<mailto:[email protected]>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
<https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
https://mail.python.org/archives/list/[email protected]/message/DHKE7LVN4R7NQFTBJJHGXI3AJOK6OYIV/
<https://mail.python.org/archives/list/[email protected]/message/DHKE7LVN4R7NQFTBJJHGXI3AJOK6OYIV/>
> Code of Conduct: http://python.org/psf/codeofconduct/
<http://python.org/psf/codeofconduct/>
_______________________________________________
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/7XVEU2Y4LGFT6LNR2KYNGVENOI4N5EQU/
Code of Conduct: http://python.org/psf/codeofconduct/