homepage

Python implements debug hooks on Python memory allocators:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer.
Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks.
test.py used in below example:
----
import _testcapi
def f():
 _testcapi.pymem_buffer_overflow()
f()
----
Example:
---
haypo@selma$ ./python -X tracemalloc=5 test.py 
Debug memory block at address p=0x22e8be0: API 'm'
 16 bytes originally requested
 The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
 The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb):
 at tail+0: 0x78 *** OUCH
 ...
 The block was made by call #37240 to debug malloc/realloc.
 Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb
Memory block traceback (most recent call first):
 File "test.py", line 4
 File "test.py", line 6
Fatal Python error: bad trailing pad byte
Current thread 0x00007f4a93d9c700 (most recent call first):
 File "test.py", line 4 in f
 File "test.py", line 6 in <module>
Abandon (core dumped)
---
The patch adds the "Memory block traceback" section.

See also the issue #26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted".

I reviewed my own patch on Rietveld :-)

New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
Enhance and rewrite traceback dump C functions
https://hg.python.org/cpython/rev/18a19e62bac5
New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
_tracemalloc: store lineno as unsigned int
https://hg.python.org/cpython/rev/fea3c6e9a38e 

New changeset cef6a32d805f by Victor Stinner in branch 'default':
On memory error, dump the memory block traceback
https://hg.python.org/cpython/rev/cef6a32d805f 

New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
Oops, revert unwanted change used to create an example
https://hg.python.org/cpython/rev/8215dae7ec3c 

New changeset 769dfcb701ee by Victor Stinner in branch 'default':
Issue #26564: Fix test_capi
https://hg.python.org/cpython/rev/769dfcb701ee 

FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:
..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea3円.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea3円.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea3円.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]

New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default':
Fix compilation error of traceback.c on Windows
https://hg.python.org/cpython/rev/5f2284ecf9c6 

Martin Panter added the comment:
> FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:
Thank you, sometimes I miss buildbot failures.
I pushed changes which had 2 tests, so I knew that all buildbots were
red, and I planned to come back the day after to check.
For Windows, it's a pity, the expression is constant. Stupid compiler
:-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the
code. I just fixed it.
I will check again if buildbots are now happy with the current code ;-)

Ok, buildbots are green again. I close the issue.

New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774 

I'm reopening this to commit a fix for a comment which became obsolete following this change: https://hg.python.org/cpython/rev/18a19e62bac5.

New changeset 40125ab3252453bf205ed906e46bf9741c27bf9d by Irit Katriel in branch 'master':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
https://github.com/python/cpython/commit/40125ab3252453bf205ed906e46bf9741c27bf9d

New changeset cecbaa3a80d5ae28cdd4129d6d2c4395c12a89e4 by Miss Islington (bot) in branch '3.8':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
https://github.com/python/cpython/commit/cecbaa3a80d5ae28cdd4129d6d2c4395c12a89e4

New changeset 1c70d40e94741578ce28b8851fb65372ac2e7942 by Miss Islington (bot) in branch '3.9':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
https://github.com/python/cpython/commit/1c70d40e94741578ce28b8851fb65372ac2e7942

I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.
I do not propose to change anything. It is just curiosity.

> I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.
53/22 is closer to math.log(256)/math.log(10) than 5/2, and 8*53 should not overflow :-)