homepage

On Mon, Mar 17, 2008 at 11:55 AM, Alec Thomas <report@bugs.python.org> wrote:
>
> Alec Thomas <alec@swapoff.org> added the comment:
>
> Hi Neal,
>
> This seems to be a more general problem than just unicode.
Kinda, sorta. The general issue is the pattern of memory
allocation/deallocation. In the case of
>>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]
The memory that is not returned is in the integer free list. If this
code is changed to:
>>> for x in ((1, 2, 3, 4, i) for i in xrange(800000)): pass
That doesn't hold on to any extra memory. The problem is that holes
are left in memory and a lot of it can't always be returned to the
O/S. It can still be re-used by python.
> Both exhibit the same behaviour. Naively to me it seems like using the
> custom allocator uniformly would fix this problem.
In general, we should find places (like unicode) that use PyMem_* (ie,
system malloc) and replace them with PyObject_* (pymalloc). That
should improve the behaviour, but there will always be some allocation
patterns that will be suboptimal. Note that using pymalloc will only
help for objects < 256 bytes. Larger objects are delegated to the
system malloc and will still exhibit some of the bad problems.
Alec, can you find places that are using the PyMem_* interface and
create a patch to fix them. That would be great!