[Python-Dev] Problems with the Python Memory Manager

Thu Nov 24 10:54:13 CET 2005

Martin v. Löwis wrote:
> Travis Oliphant wrote:
>>> So, I now believe that his code (plus the array scalar extension 
>> type) was actually exposing a real bug in the memory manager itself. 
>> In theory, the Python memory manager should have been able to re-use 
>> the memory for the array-scalar instances because they are always the 
>> same size. In practice, the memory was apparently not being re-used 
>> but instead new blocks were being allocated to handle the load.
>>> That is really very hard to believe. Most people on this list would
> probably agree that obmalloc certain *will* reuse deallocated memory
> if the next request is for the very same size (number of bytes) that
> the previously-release object had.

Yes, I see that it does. This became more clear as all the simple tests 
I tried failed to reproduce the problem (and I spent some time looking 
at the code and reading its comments). I just can't figure out another 
explanation for why the problem went away when I went to using the 
system malloc other than some kind of corner-case in the Python memory 
allocator.
>>> His code is quite complicated and it is difficult to replicate the 
>> problem. 
>>> That the code is complex would not so much be a problem: we often
> analyse complex code here. It is a problem that the code is not
> available, and it would be a problem if the problem was not
> reproducable even if you had the code (i.e. if the problem would
> sometimes occur, but not the next day when you ran it again).
>The problem was definitely reproducible. On his machine, and on the two 
machines I tried to run it on. It without fail rapidly consumed all 
available memory.
> So if you can, please post the code somewhere, and add a bugreport
> on sf.net/projects/python.
>I'll try to do this at some point. 
I'll have to get permission from him for the actual Python code. The 
extension modules he used are all publically available (PyMC). I 
changed the memory allocator in scipy --- which eliminated the problem 
--- so you'd have to check out an older version of the code from SVN to 
see the problem.
Thanks for the tips.
-Travis