<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:11pt;">
<div> </div>
<div> </div>
<div>> -----Original Message-----</div>
<div>> Take a look at the benchmark suite at</div>
<div>> <a href="http://hg.python.org/benchmarks/">http://hg.python.org/benchmarks/</a></div>
<div>> The test runner has an -m option that profiles memory usage, you could take</div>
<div>> a look at how that is implemented</div>
<div>> </div>
<div> </div>
<div>Yes, out of process monitoring of memory as reported by the OS. We do gather those counters as well on clients and servers.</div>
<div>But they don't give you the granularity you want when checking for memory leaks and memory usage by certain algorithms.</div>
<div>In the same way that the unittests have reference leak reports, they could just have memory usage reports, if the underlying allocator supported that.</div>
<div> </div>
<div>FYI the current state of affairs of the cPython 2.7 branch we use is as follows:</div>
<div>1) We allow the API user to specify the base allocator python uses, both for regular allocs and allocating blocks for the obmalloc one, using:</div>
<div> </div>
<div>/* Support for custom allocators */</div>
<div>typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int line, const char *msg);</div>
<div>typedef void *(*PyCCP_Realloc_t)(void *ptr, size_t size, void *arg, const char *file, int line, const char *msg);</div>
<div>typedef void (*PyCCP_Free_t)(void *ptr, void *arg, const char *file, int line, const char *msg);</div>
<div>typedef size_t (*PyCCP_Msize_t)(void *ptr, void *arg);</div>
<div>typedef struct PyCCP_CustomAllocator_t</div>
<div>{</div>
<div> PyCCP_Malloc_t pMalloc;</div>
<div> PyCCP_Realloc_t pRealloc;</div>
<div> PyCCP_Free_t pFree;</div>
<div> PyCCP_Msize_t pMsize; /* can be NULL, or return -1 if no size info is avail. */</div>
<div> void *arg; /* opaque argument for the functions */</div>
<div>} PyCCP_CustomAllocator_t;</div>
<div> </div>
<div>/* To set an allocator! use 0 for the regular allocator, 1 for the block allocator.</div>
<div> * pass a null pointer to reset to internal default</div>
<div> */</div>
<div>PyAPI_FUNC(void) PyCCP_SetAllocator(int which, const PyCCP_CustomAllocator_t *); /* for BLUE to set the current context */</div>
<div> </div>
<div>/* internal data member */</div>
<div>extern PyCCP_CustomAllocator_t _PyCCP_CustomAllocator[];</div>
<div> </div>
<div>2) using ifdefs, the macros will delegate all final allocations through these allocators. This includes all the "naked" malloc calls scattered about, they are patched up using #defines.</div>
<div> </div>
<div>3) Additionally, there is an internal layer of management, before delegating to the external allocators. This internal manager provides statistics, exposed through the "sys" module.</div>
<div> </div>
<div>The layering is something like this, all more or less definable by pre-processor macros. (raw malloc() is turned into something else via pre-processor magic and a special “patch_malloc.h” file added to the modules which uses raw malloc())</div>
<div> </div>
<div><font face="Courier New"> PyMem_Malloc() PyObject_Malloc()</font></div>
<div><font face="Courier New"> | |</font></div>
<div><font face="Courier New"> v v</font></div>
<div><font face="Courier New"> Mem bookkeeping obj bookkeeping</font></div>
<div><font face="Courier New"> | |</font></div>
<div><font face="Courier New"> | v</font></div>
<div><font face="Courier New"> malloc() | obmallocator</font></div>
<div><font face="Courier New"> | | |</font></div>
<div><font face="Courier New"> v v v</font></div>
<div><font face="Courier New"> PyMem_MALLOC_RAW() PyObject_MALLOC_RAW</font></div>
<div><font face="Courier New"> | |</font></div>
<div><font face="Courier New"> v v</font></div>
<div><font face="Courier New"> malloc() or vectored allocator specified through API function</font></div>
<div> </div>
<div> </div>
<div>Cheers,</div>
<div> </div>
<div>K</div>
<div> </div>
</span></font>
</body>
</html>