homepage

With the PEP 445 and the issue #3329, Python will get an API to setup custom memory allocators. To be able to configure how memory is handled in external libraries, some libraries allow to setup a custom allocator too. New functions PyMem_RawMalloc(), PyMem_GetRawAllocators() PyMem_SetRawAllocators() can be used for that.
The safest option is to only reuse custom allocators if a library allows to setup them for a specfic function call or a specific object, and not replace the memory allocators globally. For example, the lzma library allows to set memory allocators only for one compressor object: LzmaEnc_Create(&SzAllocForLzma);
We might change the global allocators of a library if Python is not embedded, but Python *is* the application (the standard "python" program).
I don't know yet if it is safe to reuse custom memory allocators.
Windows has for example a special behaviour: each DLL (dynamic library) has its own heap, memory allocator in a DLL cannot be released from another DLL. Would this issue introduce such bug?

See also the issue #18203: "Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()".

Uncomplete(?) list of external libraries used by Python:
- libffi (_ctypes): has its own memory allocator (dlmalloc), is it used? see also issue #18178
- libmpdec (_decimal): already configured to reuse the PyMem_Malloc() family (see Modules/_decimal/_decimal.c: "Init libpdec")
- _sqlite (sqlite3: see http://www.sqlite.org/malloc.html
- expat (expact): ?
- zlib (zlib): http://www.zlib.net/manual.html#Usage
- OpenSSL (_ssl, hashlib): CRYPTO_set_mem_functions, http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124
- Tcl/Tk (_tkinter): http://tmml.sourceforge.net/doc/tcl/Alloc.html
- bz2: http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html
- libncurses (curses): ?
- dbm libraries: ndbm, gdbm, db, ... (dbm): ?
- lzma: http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html
- Windows API (_winapi, nt): ?
- readline (readline): ?

It looks like CRYPTO_set_mem_functions() of OpenSSL 1.0.1e-4.fc18 does not work: CRYPTO_set_mem_functions() calls indirectly CRYPTO_malloc() which sets "allow_customize = 0;" and so CRYPTO_set_mem_functions() does nothing (just return 0, instead of 1).
Gdb trace with a modified _ssl module:
#0 0x0000003803463100 in CRYPTO_malloc () from /lib64/libcrypto.so.10
#1 0x0000003803542fae in FIPS_drbg_new () from /lib64/libcrypto.so.10
#2 0x00000038035448e1 in FIPS_drbg_health_check () from /lib64/libcrypto.so.10
#3 0x0000003803542e88 in FIPS_drbg_init () from /lib64/libcrypto.so.10
#4 0x00000038034cf9d1 in RAND_init_fips () from /lib64/libcrypto.so.10
#5 0x0000003803465764 in OPENSSL_init_library () from /lib64/libcrypto.so.10
#6 0x0000003803462c61 in CRYPTO_set_mem_functions () from /lib64/libcrypto.so.10
#7 0x00007ffff135bc6c in PyInit__ssl () at /home/haypo/prog/python/default/Modules/_ssl.c:3180
See the code:
http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124 

Here is an initial attempt: set a custom allocator for bz2, lzma and zlib modules. The allocator is only replaced for an instance of a compressor or decompress, the change does not affect the library globally.
PyMem_RawMalloc() is used instead of PyMem_Malloc() because the GIL is always released.

New changeset a876d9d2e4fc by Victor Stinner in branch 'default':
Issue #18227: Use PyMem_RawAlloc() in bz2, lzma and zlib modules
http://hg.python.org/cpython/rev/a876d9d2e4fc 

New changeset 12f26c356611 by Victor Stinner in branch 'default':
Issue #18227: "Free" function of bz2, lzma and zlib modules has no return value (void)
http://hg.python.org/cpython/rev/12f26c356611 

New changeset 7f17c67b5bf6 by Christian Heimes in branch 'default':
Issue #18227: pyexpat now uses a static XML_Memory_Handling_Suite. cElementTree uses the same approach since at least Python 2.6
http://hg.python.org/cpython/rev/7f17c67b5bf6 

I modified modules when it was possible and easy to do. More modules should be modified, but it's more tricky. If you are interested, please open new issues.