On 20.06.2015 09:30, Victor Stinner wrote: > Hi, > > I didn't get much feedback on this PEP. Since the Python 3.6 branch is > open (default), it's probably better to push such change in the > beginning of the 3.6 cycle, to catch issues earlier. > > Are you ok to chain exceptions at C level by default?
I think it's a good idea to make C APIs available to simplify chaining exceptions at the C level, but don't believe that always doing this by default is a good idea. It should really be a case-by-case decision, IMO. Note that Python exceptions are cheap to raise in C (very much unlike in Python), so making this more expensive by default would introduce a significant overhead - without much proven benefit. More below... > PEP: 490 > Title: Chain exceptions at C level > Version: $Revision$ > Last-Modified: $Date$ > Author: Victor Stinner <[email protected]> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 25-March-2015 > Python-Version: 3.6 > > > Abstract > ======== > > Chain exceptions at C level, as already done at Python level. > > > Rationale > ========= > > Python 3 introduced a new killer feature: exceptions are chained by > default, PEP 3134. > > Example:: > > try: > raise TypeError("err1") > except TypeError: > raise ValueError("err2") > > Output:: > > Traceback (most recent call last): > File "test.py", line 2, in <module> > raise TypeError("err1") > TypeError: err1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "test.py", line 4, in <module> > raise ValueError("err2") > ValueError: err2 > > Exceptions are chained by default in Python code, but not in > extensions written in C. > > A new private ``_PyErr_ChainExceptions()`` function was introduced in > Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called > explicitly to chain exceptions and its usage is not trivial. > > Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport`` > module to chain the previous ``OSError`` to a new ``ZipImportError`` > exception:: > > PyObject *exc, *val, *tb; > PyErr_Fetch(&exc, &val, &tb); > PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); > _PyErr_ChainExceptions(exc, val, tb); > > This PEP proposes to also chain exceptions automatically at C level to > stay consistent and give more information on failures to help > debugging. The previous example becomes simply:: > > PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); > > > Proposal > ======== > > Modify PyErr_*() functions to chain exceptions > ---------------------------------------------- > > Modify C functions raising exceptions of the Python C API to > automatically chain exceptions: modify ``PyErr_SetString()``, > ``PyErr_Format()``, ``PyErr_SetNone()``, etc. > > > Modify functions to not chain exceptions > ---------------------------------------- > > Keeping the previous exception is not always interesting when the new > exception contains information of the previous exception or even more > information, especially when the two exceptions have the same type. > > Example of an useless exception chain with ``int(str)``:: > > TypeError: a bytes-like object is required, not 'type' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: int() argument must be a string, a bytes-like object or > a number, not 'type' > > The new ``TypeError`` exception contains more information than the > previous exception. The previous exception should be hidden. > > The ``PyErr_Clear()`` function can be called to clear the current > exception before raising a new exception, to not chain the current > exception with a new exception. > > > Modify functions to chain exceptions > ------------------------------------ > > Some functions save and then restore the current exception. If a new > exception is raised, the exception is currently displayed into > sys.stderr or ignored depending on the function. Some of these > functions should be modified to chain exceptions instead. > > Examples of function ignoring the new exception(s): > > * ``ptrace_enter_call()``: ignore exception > * ``subprocess_fork_exec()``: ignore exception raised by enable_gc() > * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised > by trying to display the bootstrap function to ``sys.stderr`` > * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore > exception raised by looking for a key in the dictionary > * ``_PyErr_TrySetFromCause()``: ignore exception > * ``PyFrame_LocalsToFast()``: ignore exception raised by > ``dict_to_map()`` > * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used > to debug, to inspect a running process, it should not modify the > Python state. > * ``Py_ReprLeave()``: ignore exception "because there is no way to > report them" > * ``type_dealloc()``: ignore exception raised by > ``remove_all_subclasses()`` > * ``PyObject_ClearWeakRefs()``: ignore exception? > * ``call_exc_trace()``, ``call_trace_protected()``: ignore exception > * ``remove_importlib_frames()``: ignore exception > * ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example: > ignore exception? > * ``flush_io()``: ignore exception > * ``sys_write()``, ``sys_format()``: ignore exception > * ``_PyTraceback_Add()``: ignore exception > * ``PyTraceBack_Print()``: ignore exception Which of these do you think would benefit from chaining exceptions ? > Examples of function displaying the new exception to ``sys.stderr``: > > * ``atexit_callfuncs()``: display exceptions with > ``PyErr_Display()`` and return the latest exception, the function > calls multiple callbacks and only returns the latest exception > * ``sock_dealloc()``: log the ``ResourceWarning`` exception with > ``PyErr_WriteUnraisable()`` > * ``slot_tp_del()``: display exception with > ``PyErr_WriteUnraisable()`` > * ``_PyGen_Finalize()``: display ``gen_close()`` exception with > ``PyErr_WriteUnraisable()`` > * ``slot_tp_finalize()``: display exception raised by the > ``__del__()`` method with ``PyErr_WriteUnraisable()`` > * ``PyErr_GivenExceptionMatches()``: display exception raised by > ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()`` Same here. In many cases, there's no way to report exceptions at all, so chaining them make things better :-) > Backward compatibility > ====================== > > A side effect of chaining exceptions is that exceptions store > traceback objects which store frame objects which store local > variables. Local variables are kept alive by exceptions. A common > issue is a reference cycle between local variables and exceptions: an > exception is stored in a local variable and the frame indirectly > stored in the exception. The cycle only impacts applications storing > exceptions. It's not only about reference cycles. Tracebacks can also keep files, sockets and other external resources open as well as prevent large memory areas from being freed. > The reference cycle can now be fixed with the new > ``traceback.TracebackException`` object introduced in Python 3.5. It > stores informations required to format a full textual traceback without > storing local variables. > > The ``asyncio`` is impacted by the reference cycle issue. This module > is also maintained outside Python standard library to release a > version for Python 3.3. ``traceback.TracebackException`` will maybe > be backported in a private ``asyncio`` module to fix reference cycle > issues. > > > Alternatives > ============ > > No change > --------- > > A new private ``_PyErr_ChainExceptions()`` function is enough to chain > manually exceptions. > > Exceptions will only be chained explicitly where it makes sense. > > > New helpers to chain exceptions > ------------------------------- > > Functions like ``PyErr_SetString()`` don't chain automatically > exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier, > new private functions are added: > > * ``_PyErr_SetStringChain(exc_type, message)`` > * ``_PyErr_FormatChain(exc_type, format, ...)`` > * ``_PyErr_SetNoneChain(exc_type)`` > * ``_PyErr_SetObjectChain(exc_type, exc_value)`` > > Helper functions to raise specific exceptions like > ``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name, > path)`` don't chain exceptions. The generic > ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used > to chain exceptions with these helper functions. > > > Appendix > ======== > > PEPs > ---- > > * `PEP 3134 -- Exception Chaining and Embedded Tracebacks > <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0): > new ``__context__`` and ``__cause__`` attributes for exceptions > * `PEP 415 - Implement context suppression with exception attributes > <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3): > ``raise exc from None`` > * `PEP 409 - Suppressing exception context > <https://www.python.org/dev/peps/pep-0409/>`_ > (superseded by the PEP 415) > > > Python C API > ------------ > > The header file ``Include/pyerror.h`` declares functions related to > exceptions. > > Functions raising exceptions: > > * ``PyErr_SetNone(exc_type)`` > * ``PyErr_SetObject(exc_type, exc_value)`` > * ``PyErr_SetString(exc_type, message)`` > * ``PyErr_Format(exc, format, ...)`` > > Helpers to raise specific exceptions: > > * ``PyErr_BadArgument()`` > * ``PyErr_BadInternalCall()`` > * ``PyErr_NoMemory()`` > * ``PyErr_SetFromErrno(exc)`` > * ``PyErr_SetFromWindowsErr(err)`` > * ``PyErr_SetImportError(message, name, path)`` > * ``_PyErr_SetKeyError(key)`` > * ``_PyErr_TrySetFromCause(prefix_format, ...)`` > > Manage the current exception: > > * ``PyErr_Clear()``: clear the current exception, > like ``except: pass`` > * ``PyErr_Fetch(exc_type, exc_value, exc_tb)`` > * ``PyErr_Restore(exc_type, exc_value, exc_tb)`` > * ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)`` > * ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)`` > > Others function to handle exceptions: > > * ``PyErr_ExceptionMatches(exc)``: check to implement > ``except exc: ...`` > * ``PyErr_GivenExceptionMatches(exc1, exc2)`` > * ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)`` > * ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` > > > Python Issues > ------------- > > Chain exceptions: > > * `Issue #23763: Chain exceptions in C > <http://bugs.python.org/issue23763>`_ > * `Issue #23696: zipimport: chain ImportError to OSError > <http://bugs.python.org/issue23696>`_ > * `Issue #21715: Chaining exceptions at C level > <http://bugs.python.org/issue21715>`_: added > ``_PyErr_ChainExceptions()`` > * `Issue #18488: sqlite: finalize() method of user function may be > called with an exception set if a call to step() method failed > <http://bugs.python.org/issue18488>`_ > * `Issue #23781: Add private _PyErr_ReplaceException() in 2.7 > <http://bugs.python.org/issue23781>`_ > * `Issue #23782: Leak in _PyTraceback_Add > <http://bugs.python.org/issue23782>`_ > > Changes preventing to loose exceptions: > > * `Issue #23571: Raise SystemError if a function returns a result with an > exception set <http://bugs.python.org/issue23571>`_ > * `Issue #18408: Fixes crashes found by pyfailmalloc > <http://bugs.python.org/issue18408>`_ > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2015年06月16日: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015年06月10日: Released mxODBC Plone/Zope DA 2.2.2 http://egenix.com/go76 2015年07月20日: EuroPython 2015, Bilbao, Spain ... 30 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com