homepage

I created a first toy based on the PEP 445 to test of Python handles MemoryError: in short, Python handles them badly...
Home page of the project:
https://pypi.python.org/pypi/pyfailmalloc
I will use this issue to track crashes found by this tool.

New changeset 111c2a070f28 by Victor Stinner in branch 'default':
Issue #18408: PyObject_GC_NewVar() now raises SystemError exception if nitems
http://hg.python.org/cpython/rev/111c2a070f28
New changeset ba766323b53a by Victor Stinner in branch 'default':
Issue #18408: Fix PyDict_New() to handle correctly new_keys_object() failure
http://hg.python.org/cpython/rev/ba766323b53a
New changeset 68887e177dd4 by Victor Stinner in branch 'default':
Issue #18408: Fix list.pop() to handle list_resize() failure (MemoryError).
http://hg.python.org/cpython/rev/68887e177dd4
New changeset 697d722d97f9 by Victor Stinner in branch 'default':
Issue #18408: Fix marshal reader for Unicode strings: handle
http://hg.python.org/cpython/rev/697d722d97f9
New changeset de1473f4503b by Victor Stinner in branch 'default':
Issue #18408: Fix PyType_Ready() and type.__bases__ setter to handle
http://hg.python.org/cpython/rev/de1473f4503b
New changeset f4311870e329 by Victor Stinner in branch 'default':
Issue #18408: Fix call_function() of ceval.c to handle PyTuple_New() failure
http://hg.python.org/cpython/rev/f4311870e329
New changeset df8b40593a08 by Victor Stinner in branch 'default':
Issue #18408: Fix _PyUnicodeWriter_Finish(): clear writer->buffer,
http://hg.python.org/cpython/rev/df8b40593a08 

"I created a first toy based on the PEP 445 to test of Python handles MemoryError: in short, Python handles them badly..."
Ooops: to test *how* Python handles..

New changeset ba79f6a86300 by Victor Stinner in branch 'default':
Issue #18408: Fix ConvParam() of the ctypes module to handle paramfunc failure
http://hg.python.org/cpython/rev/ba79f6a86300
New changeset 2ef2edfd1a4c by Victor Stinner in branch 'default':
Issue #18408: Fix zlib.compressobj() to handle PyThread_allocate_lock() failure
http://hg.python.org/cpython/rev/2ef2edfd1a4c
New changeset a45407fa4a5b by Victor Stinner in branch 'default':
Issue #18408: Fix PyCode_Optimize(): raise a MemoryError on memory allocation
http://hg.python.org/cpython/rev/a45407fa4a5b
New changeset ed0c9d77e179 by Victor Stinner in branch 'default':
Issue #18408: Fix usage of _PyBytes_Resize()
http://hg.python.org/cpython/rev/ed0c9d77e179
New changeset d605c6b8095c by Victor Stinner in branch 'default':
Issue #18408: _PyUnicodeWriter_Finish() now clears its buffer attribute in all
http://hg.python.org/cpython/rev/d605c6b8095c
New changeset 549d8d3297f2 by Victor Stinner in branch 'default':
Issue #18408: Fix _PyMem_DebugRealloc()
http://hg.python.org/cpython/rev/549d8d3297f2
New changeset c91e7f707562 by Victor Stinner in branch 'default':
Issue #18408: Fix select.select() to handle PyList_New() failure (MemoryError)
http://hg.python.org/cpython/rev/c91e7f707562 

use_failmalloc.patch: Patch to use pyfailmalloc on the Python test suite. Command to run test suite: "gdb -args ./python -m test -F".

call_exc_trace.patch: Fix for the call_exc_trace() function when the traceback is not set.

type_slot_tp_str.patch: Fix for the __str__ slot of types (slot_tp_str() in typeobject.c).

misc_fixes.patch: Misc bugfixes related to MemoryError, this patch should be read carefully. The changes in unicodeobject.c may be written differently. I wrote them to fix a failing assertion ("assert(data != NULL);") in _PyUnicode_CheckConsistency(), called when the creation of a new string failed.

New changeset 4f730c045f5f by Victor Stinner in branch 'default':
Issue #18408: Fix call_exc_trace(): if the traceback is NULL, use None when
http://hg.python.org/cpython/rev/4f730c045f5f 

New changeset 01a46dc00fc8 by Victor Stinner in branch 'default':
Issue #18408: slot_tp_str() must not fallback on slot_tp_repr() on error
http://hg.python.org/cpython/rev/01a46dc00fc8 

New changeset 39eb1ce5f377 by Victor Stinner in branch 'default':
Issue #18408: ste_new() initialize all attributes before handling error
http://hg.python.org/cpython/rev/39eb1ce5f377
New changeset aaa6e8b8a5c9 by Victor Stinner in branch 'default':
Issue #18408: Fix compiler_import() to handle PyUnicode_Substring() failure properly
http://hg.python.org/cpython/rev/aaa6e8b8a5c9
New changeset ff93930a53c0 by Victor Stinner in branch 'default':
Issue #18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/ff93930a53c0
New changeset 1a1869baec4c by Victor Stinner in branch 'default':
Issue #18408: Fix _Pickler_New() and _Unpickler_New(): initialize all
http://hg.python.org/cpython/rev/1a1869baec4c
New changeset e11121b9bd09 by Victor Stinner in branch 'default':
Issue #18408: _elementtree.c now handles create_extra() failure
http://hg.python.org/cpython/rev/e11121b9bd09
New changeset 0877e17fa25e by Victor Stinner in branch 'default':
Issue #18408: Different fixes in _elementtree.c to handle correctly MemoryError
http://hg.python.org/cpython/rev/0877e17fa25e 

New changeset 51eddca9dd6f by Victor Stinner in branch 'default':
Issue #18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/51eddca9dd6f
New changeset 5e50f1a0c985 by Victor Stinner in branch 'default':
Issue #18408: In debug mode, PyCFunction_Call() now checks if an exception was
http://hg.python.org/cpython/rev/5e50f1a0c985 

New changeset 4975bcd67aa7 by Victor Stinner in branch 'default':
Issue #18408: normalizestring() now raises MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/4975bcd67aa7
New changeset 1eac89af9acf by Victor Stinner in branch 'default':
Issue #18408: Oh, I was wrong: Pickler_New() must call Py_DECREF() to destroy
http://hg.python.org/cpython/rev/1eac89af9acf
New changeset d9446c2a2fd4 by Victor Stinner in branch 'default':
Issue #18408: ceval.c: in debug mode, convert the PyErr_Occurred() check on
http://hg.python.org/cpython/rev/d9446c2a2fd4
New changeset 2f7c4df5cc46 by Victor Stinner in branch 'default':
Issue #18408: errors.c: in debug mode, calling PyErr_BadInternalCall() now
http://hg.python.org/cpython/rev/2f7c4df5cc46
New changeset affb87b1d7ec by Victor Stinner in branch 'default':
Issue #18408: pmerge() help of mro_implementation() now raises MemoryError on
http://hg.python.org/cpython/rev/affb87b1d7ec
New changeset f85fcbbbe8de by Victor Stinner in branch 'default':
Issue #18408: _PyMemoTable_ResizeTable() now restores the old table if
http://hg.python.org/cpython/rev/f85fcbbbe8de
New changeset bb5da6e795a1 by Victor Stinner in branch 'default':
Issue #18408: _pickle.c: Add missing PyErr_NoMemory() on memory allocation failures
http://hg.python.org/cpython/rev/bb5da6e795a1 

New changeset 89c6495d1ff2 by Victor Stinner in branch 'default':
Issue #18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
http://hg.python.org/cpython/rev/89c6495d1ff2
New changeset c80a9705803a by Victor Stinner in branch 'default':
Issue #18408: parser module: fix error handling in node2tuple()
http://hg.python.org/cpython/rev/c80a9705803a 

New changeset 60b1d7967ef8 by Victor Stinner in branch 'default':
Issue #18408: Fix create_extra() of _elementtree.c, raise MemoryError on memory
http://hg.python.org/cpython/rev/60b1d7967ef8
New changeset 1e0afd558ba3 by Victor Stinner in branch 'default':
Issue #18408: Fix constructors of _elementtree.c
http://hg.python.org/cpython/rev/1e0afd558ba3 

listpop.patch: list_ass_slice() now handles listresize() failure when items are deleted.

frame_fasttolocals.patch: Add a new PyFrame_FastToLocalsWithError() function, same than PyFrame_FastToLocals(), except that errors are reported.
This function is not atomic: if an error occurs in the middle of map_to_dict(), the frame can enter an inconsistent state. Is it worse than not reporting the failure?

New changeset 735e3c43e61c by Victor Stinner in branch 'default':
Issue #18408: Fix pyexpat.ParserCreate()
http://hg.python.org/cpython/rev/735e3c43e61c
New changeset 27f8ccce687d by Victor Stinner in branch 'default':
Issue #18408: Fix CJK decoders, raise MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/27f8ccce687d
New changeset 82b2ee140994 by Victor Stinner in branch 'default':
Issue #18408: PyObject_Call() now fails with an assertion error in debug mode
http://hg.python.org/cpython/rev/82b2ee140994
New changeset ef0dbe00760e by Victor Stinner in branch 'default':
Issue #18408: Don't check unicode consistency in _PyUnicode_HAS_UTF8_MEMORY()
http://hg.python.org/cpython/rev/ef0dbe00760e
New changeset c676fc74d779 by Victor Stinner in branch 'default':
Issue #18408: type_new() and PyType_FromSpecWithBases() now raise MemoryError
http://hg.python.org/cpython/rev/c676fc74d779
New changeset 9244e79c6e14 by Victor Stinner in branch 'default':
Issue #18408: random_seed() now raises a MemoryError on memory allocation
http://hg.python.org/cpython/rev/9244e79c6e14
New changeset 71a572a516f9 by Victor Stinner in branch 'default':
Issue #18408: Fix PyEval_EvalFrameEx() for MemoryError
http://hg.python.org/cpython/rev/71a572a516f9 

New changeset 2e42c0a39321 by Victor Stinner in branch 'default':
Issue #18408: Fix _Py_DisplaySourceLine()
http://hg.python.org/cpython/rev/2e42c0a39321
New changeset 92a9ccb2a521 by Victor Stinner in branch 'default':
Issue #18408: add more assertions on PyErr_Occurred() in ceval.c to detect bugs
http://hg.python.org/cpython/rev/92a9ccb2a521
New changeset 9213313b1a6f by Victor Stinner in branch 'default':
Issue #18408: Fix show_warning(), clear also the exception raised by
http://hg.python.org/cpython/rev/9213313b1a6f 

I created the issue #18469: "PyStructSequence does not handle exceptions correctly".

import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure.

New issue #18470: "tokenizer.c does not handle new_string() failure".

New changeset 533eb9ab895a by Victor Stinner in branch 'default':
Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
http://hg.python.org/cpython/rev/533eb9ab895a
New changeset f0efd7ea1627 by Victor Stinner in branch 'default':
Issue #18408: Fix cjkcodecs decoders, add a new MBERR_EXCEPTION constant to
http://hg.python.org/cpython/rev/f0efd7ea1627
New changeset 395e67f348e7 by Victor Stinner in branch 'default':
Issue #18408: Fix list.extend(), handle list_resize() failure
http://hg.python.org/cpython/rev/395e67f348e7
New changeset ac4e8e6a7436 by Victor Stinner in branch 'default':
Issue #18408: Fix PyDict_GetItemString(), suppress PyUnicode_FromString() error
http://hg.python.org/cpython/rev/ac4e8e6a7436
New changeset b7c6d203c9fb by Victor Stinner in branch 'default':
Issue #18408: dict_new() now fails on new_keys_object() error
http://hg.python.org/cpython/rev/b7c6d203c9fb
New changeset 28ff7ac91477 by Victor Stinner in branch 'default':
Issue #18408: Py_ReprLeave() now saves/restores the current exception,
http://hg.python.org/cpython/rev/28ff7ac91477
New changeset 6bd01a59762a by Victor Stinner in branch 'default':
Issue #18408: handle PySys_GetObject() failure, raise a RuntimeError
http://hg.python.org/cpython/rev/6bd01a59762a 

You are great!

New changeset c8696ee49291 by Victor Stinner in branch 'default':
Issue #18408: Fix typo in build_node_tree() of the parser module
http://hg.python.org/cpython/rev/c8696ee49291
New changeset cf8f42eadbd0 by Victor Stinner in branch 'default':
Issue #18408: Fix Python-ast.c: handle init_types() failure (ex: MemoryError)
http://hg.python.org/cpython/rev/cf8f42eadbd0
New changeset a685f4c6e0b6 by Victor Stinner in branch 'default':
Issue #18408: Fix PyErr_NormalizeException(), handle PyObject_IsSubclass() failure
http://hg.python.org/cpython/rev/a685f4c6e0b6
New changeset 7fe4a0c0e905 by Victor Stinner in branch 'default':
Issue #18408: Fix locale.localeconv(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/7fe4a0c0e905
New changeset 8c1ca6720246 by Victor Stinner in branch 'default':
Issue #18408: Handle PyArena_AddPyObject() failure in ast.c
http://hg.python.org/cpython/rev/8c1ca6720246 

New changeset 1872555a2f4e by Victor Stinner in branch 'default':
Issue #18408: Mention changes in Misc/NEWS
http://hg.python.org/cpython/rev/1872555a2f4e 

@Serhiy: Would like to review attached patches listpop.patch (msg193130) and frame_fasttolocals.patch (msg193131)? (Other are already commited)

Oops.
@Serhiy: Would *you* like to review attached patches ...

listpop.patch:
list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.

Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).

> Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
Seconded.

2013年7月17日 Serhiy Storchaka <report@bugs.python.org>:
> listpop.patch:
>
> list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.
Resizing a buffer to make it smaller "should" not fail, and if it
fails, the case is very unlikely. But I prefer to handle the case,
rather than hoping that the case cannot occur in the libc
implementation.
For example, _PyObject_Realloc() of pymalloc allocates a *new* buffer
to shrink an existing buffer if the size is reduced by more than 25%:
 if (nbytes <= size) {
 /* The block is staying the same or shrinking. If
 * it's shrinking, there's a tradeoff: it costs
 * cycles to copy the block to a smaller size class,
 * but it wastes memory not to copy it. The
 * compromise here is to copy on shrink only if at
 * least 25% of size can be shaved off.
 */
 if (4 * nbytes > 3 * size) {
 /* It's the same,
 * or shrinking and new/old > 3/4.
 */
 return p;
 }
 size = nbytes;
 }
 bp = _PyObject_Malloc(ctx, nbytes);
I plan also to test Python with PyMem_Malloc using pymalloc to check
if it's faster or not. So PyMem_Realloc() may fail when shrinking a
buffer ;-)
By the way, I already fixed similar bugs (handle PyMem_Realloc failure
when shrinking the buffer) in this issue. Examples:
http://hg.python.org/cpython/rev/68887e177dd4
http://hg.python.org/cpython/rev/549d8d3297f2
listpop.patch only changes the error handling to make it more
resilient when the "very unlikely" case occurs ;-)

2013年7月17日 Charles-François Natali <report@bugs.python.org>:
> Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
> This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).
I plan to write a new module and propose it for inclusion in Python
3.4. It will have the features of pytracemalloc *and* pyfailmalloc,
and also statistics (number of calls to each allocator function, with
a function to reset counters).
The blocker pointer is to get the size of a buffer on free(). In
pytracemalloc, I'm using a glib hash lib: pointer => structure {size
of the buffer, filename, line number}. The dependency to glib should
be removed to integrate the module in Python :-)
The simpler solution is to allocate a wider buffer and store the
metadata directly in the buffer before the data. The second problem is
how this feature would be enabled. Enabling the feature at runtime
would require a new environment variable to set the right memory
allocator at startup. The choice of the memory allocator must be done
before the first memory allocation, so in main().
Integrating pytracemalloc into CPython was my main motivation to write
the PEP 445.
If you would like to use pyfailmalloc right now, you have to:
* build pyfailmalloc with Python 3.4
* apply debug_cpython.patch on Python which is the pyfailmalloc repository
* run gdb -x debug_cpython.gdb (which can also be found in the
pyfailmalloc repository)
* https://bitbucket.org/haypo/pyfailmalloc 

New changeset 7b81a535ad14 by Victor Stinner in branch 'default':
Issue #18408: Fix structseq_reduce(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/7b81a535ad14 

Open issues (TODO list):
- review and apply listpop.patch and frame_fasttolocals.patch
- fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION
- fix Modules/pyexpat.c: handlers do no detect failures
- fix sqlite: Modules/_sqlite/connection.c calls the finalize method on the aggregate whereas calling step() failed
- fix tests hang when an exception occurs in a thread
- fix new bugs found by pyfailmalloc :-)

New changeset 4d3bb9763bf9 by Victor Stinner in branch 'default':
Issue #18408: Fix time.tzset(), detect exception when calling PyInit_timezone()
http://hg.python.org/cpython/rev/4d3bb9763bf9
New changeset 563b27bef79f by Victor Stinner in branch 'default':
Issue #18408: Fix heapq.heappop(), handle PyList_SetSlice() failure
http://hg.python.org/cpython/rev/563b27bef79f
New changeset 6fb3414710ab by Victor Stinner in branch 'default':
Issue #18408: Fix ast_for_atom(), PyErr_Fetch(&type, &value, &tback) can set value to NULL
http://hg.python.org/cpython/rev/6fb3414710ab
New changeset 479af6ca3a3c by Victor Stinner in branch 'default':
Issue #18408: Fix _PyImport_LoadDynamicModule(), handle PyUnicode_FromFormat() failure
http://hg.python.org/cpython/rev/479af6ca3a3c
New changeset 573f7d485ce8 by Victor Stinner in branch 'default':
Issue #18408: Fix PyErr_SetImportError(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/573f7d485ce8
New changeset 97bb3bdf1443 by Victor Stinner in branch 'default':
Issue #18408: Fix listpop(), handle list_ass_slice() failure
http://hg.python.org/cpython/rev/97bb3bdf1443
New changeset 2f5f1db8eb88 by Victor Stinner in branch 'default':
Issue #18408: Fix Py_ReprEnter(), handle PyList_Append() failure
http://hg.python.org/cpython/rev/2f5f1db8eb88
New changeset c2d90ff0780c by Victor Stinner in branch 'default':
Issue #18408: Fix PyType_Ready(), handle _PyDict_SetItemId() failure
http://hg.python.org/cpython/rev/c2d90ff0780c 

New changeset 736514fb7b52 by Victor Stinner in branch 'default':
Issue #18408: Rewrite NEGATE() macro in longobject.c to handle PyLong_FromLong() failure
http://hg.python.org/cpython/rev/736514fb7b52 

New changeset d85b0c355052 by Victor Stinner in branch 'default':
Issue #18408: Fix array_index(), handle getarrayitem() failure
http://hg.python.org/cpython/rev/d85b0c355052
New changeset b05a6a6eb525 by Victor Stinner in branch 'default':
Issue #18408: Fix array_tolist(), handle PyList_SetItem() failure
http://hg.python.org/cpython/rev/b05a6a6eb525
New changeset 161de66cae49 by Victor Stinner in branch 'default':
Issue #18408: Fix dict_repr(), don't call PyObject_Repr() with an exception set
http://hg.python.org/cpython/rev/161de66cae49
New changeset 48a869a39e2d by Victor Stinner in branch 'default':
Issue #18408: PyEval_EvalFrameEx() and PyEval_CallObjectWithKeywords() now fail
http://hg.python.org/cpython/rev/48a869a39e2d 

New changeset 9bae7696951f by Victor Stinner in branch 'default':
Issue #18408: Fix _pysqlite_fetch_one_row(), in debug mode, don't call
http://hg.python.org/cpython/rev/9bae7696951f
New changeset 5bd9db528aed by Victor Stinner in branch 'default':
Issue #18408: PyObject_Str(), PyObject_Repr() and type_call() now fail with an
http://hg.python.org/cpython/rev/5bd9db528aed 

New changeset ddff866d820d by Victor Stinner in branch 'default':
Issue #18408: Fix PyInit__curses_panel(), handle import_curses() failure
http://hg.python.org/cpython/rev/ddff866d820d 

Changesets 48a869a39e2d and 5bd9db528aed adding "assert(!PyErr_Occurred());" broke some buildbots. I fixed the issue on OpenSolaris, I'm unable to reproduce the 2 remaining issues (I tested on Fedora 18 and FreeBSD 9).
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/8236/steps/test/logs/stdio
python: Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted
Current thread 0x404df8c0:
 File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 441 in authorizer_cb
 File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 406 in test_table_access
 File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 496 in run
 ...
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/98/steps/test/logs/stdio
FAIL: test_finalize_runnning_thread (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threading.py", line 298, in test_finalize_runnning_thread
 self.assertEqual(rc, 42)
AssertionError: -6 != 42

Another buildbot failure:
[150/375/4] test_ctypes
Assertion failed: __EX, file Python/ceval.c, line 4064
Fatal Python error: Aborted
Current thread 0x00000001:
 File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 44 in capture_stderr
 File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 63 in test_FloatDivisionError

http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/310/steps/test/logs/stdio 

This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree:
 def test_parser_target_error_in_start(self):
 assertEqual = self.assertEqual
 events = []
 class Target(object):
 def start(self, tag, attrib):
 events.append("start")
 assertEqual("TAG", tag)
 raise ValueError("TEST")
 def end(self, tag):
 events.append("end")
 assertEqual("TAG", tag)
 def close(self):
 return "DONE"
 parser = ET.XMLParser(target=Target())
 try:
 parser.feed("<TAG/>")
 except ValueError:
 self.assertTrue('TEST' in str(sys.exc_info()[1]))
 else:
 self.assertTrue(False)
 self.assertTrue("start" in events)
Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
(note: this is not the exact code, I removed a couple of conditions that are not used by ET)

New changeset 6ec0e9347dd4 by Victor Stinner in branch 'default':
Issue #18408: Fix _elementtree.c, don't call Python function from an expat
http://hg.python.org/cpython/rev/6ec0e9347dd4 

> This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree: (...)
Thanks for the quick report! Are you testing automatically lxml with the Python HEAD?
The issue should be fixed by the changeset 6ec0e9347dd4. I also opened the issue #18501 to discuss how ElementTree should handle Python exceptions.

New changeset 5a6cdc0d7de1 by Victor Stinner in branch 'default':
Issue #18501, #18408: Fix expat handlers in pyexpat, don't call Python
http://hg.python.org/cpython/rev/5a6cdc0d7de1 

I created the issue #18488: "sqlite: finalize() method of user function may be called with an exception set if a call to step() method failed".

@Victor: yes, the Cython project has CI tests running against debug builds of all CPython branches since 2.4, updated daily. lxml is part of an extended set of tests for Cython, and the test suite of lxml includes several compatibility tests for ElementTree.
https://sage.math.washington.edu:8091/hudson/
https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk/
I noticed the failures when those tests started failing. Thanks for the quick fix, and actually for all the work you're doing here.

New changeset 3f25a7dd8346 by Victor Stinner in branch 'default':
Issue #18408: Fix list_ass_slice(), handle list_resize() failure
http://hg.python.org/cpython/rev/3f25a7dd8346 

"import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure."
I created the issue #18507 for that.

"fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION"
I created the issue #18509 for that.

New changeset 026593cbc006 by Victor Stinner in branch 'default':
Issue #18519, #18408: Fix sqlite authorizer callback
http://hg.python.org/cpython/rev/026593cbc006 

"""
python: Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted
Current thread 0x404df8c0:
 File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 441 in authorizer_cb
"""
Ned Deily created the issue #18519 for that. It should be fixed by the changeset 026593cbc006.

New changeset de1ea7f84e08 by Victor Stinner in branch 'default':
Issue #18609, #18408: _ctypes_add_traceback() now clears the current exception
http://hg.python.org/cpython/rev/de1ea7f84e08 

"""
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/98/steps/test/logs/stdio
FAIL: test_finalize_runnning_thread (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threading.py", line 298, in test_finalize_runnning_thread
 self.assertEqual(rc, 42)
AssertionError: -6 != 42
"""
Oh, this error is not a regression introduced by this issue: it was already present in the oldest build of this buildbot:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/5/steps/test/logs/stdio 

TODO:
- review and integrate frame_fasttolocals.patch
- #18507: import_init() should not use Py_FatalError() but return an error
- #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error
- fix tests hang when an exception occurs in a thread
Most bugs have been fixed.

New changeset 42aecc601a41 by Antoine Pitrou in branch 'default':
Back out 5bd9db528aed (issue #18408). It caused unsolved buildbot failures.
http://hg.python.org/cpython/rev/42aecc601a41 

Victor, I'll let you find out how to improve your debug checks, but in the meantime I've reverted the commit that randomly broke some Windows buildbots.

I didn't know that buildbots are failing. Do you have at least an example?
An error message, anything?

It's reported in issue18664 (you're nosied there :-)).

New changeset 79ce25c70795 by Victor Stinner in branch 'default':
Issue #18408: _PyObject_Dump() now saves/restores the current exception
http://hg.python.org/cpython/rev/79ce25c70795
New changeset e63f19d0a651 by Victor Stinner in branch 'default':
Issue #18664, #18408: Rewrite PyErr_WriteUnraisable() to handle errors
http://hg.python.org/cpython/rev/e63f19d0a651
New changeset 8fb3a6f9b0a4 by Victor Stinner in branch 'default':
Restore changeset 5bd9db528aed (issue #18408)
http://hg.python.org/cpython/rev/8fb3a6f9b0a4 

New changeset 4ef4578db38a by Victor Stinner in branch 'default':
Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
http://hg.python.org/cpython/rev/4ef4578db38a 

Ok, most critical bugs have been fixed. If I find new bugs, I will open new issues.

New changeset e1d51c42e5a1 by Victor Stinner in branch 'default':
Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
http://hg.python.org/cpython/rev/e1d51c42e5a1
New changeset 7f4a976829f1 by Victor Stinner in branch 'default':
Issue #18408: Fix zipimport, handle PyUnicode_Substring() and get_subname() failures
http://hg.python.org/cpython/rev/7f4a976829f1 

New changeset 4749c3ac0654 by Victor Stinner in branch 'default':
Issue #18408: Fix iobase_readline(), handle PyByteArray_Resize() failure
http://hg.python.org/cpython/rev/4749c3ac0654 

New changeset 549cd815bf8d by Victor Stinner in branch 'default':
Issue #18408: Fix error handling in PyBytes_FromObject()
http://hg.python.org/cpython/rev/549cd815bf8d
New changeset 9d9371d4ff7b by Victor Stinner in branch 'default':
Issue #18408: Fix bytearrayiter.partition()/rpartition(), handle
http://hg.python.org/cpython/rev/9d9371d4ff7b
New changeset 20cd15a28ad3 by Victor Stinner in branch 'default':
Issue #18408: Fix _ctypes_alloc_format_string(), raise MemoryError on memory
http://hg.python.org/cpython/rev/20cd15a28ad3
New changeset 1386fb31c0ed by Victor Stinner in branch 'default':
Issue #18408: Fix PyCStructUnionType_update_stgdict(), handle
http://hg.python.org/cpython/rev/1386fb31c0ed 

New changeset 8f556ee0f6ba by Antoine Pitrou in branch 'default':
Fix refleak introduced by 4f730c045f5f (issue #18408) and unveiled by 95eea8624d05 (issue #16596).
http://hg.python.org/cpython/rev/8f556ee0f6ba 

New changeset b742c1c5c0bf by Victor Stinner in branch 'default':
PEP 490: add issue 18408
https://hg.python.org/peps/rev/b742c1c5c0bf