homepage

Similar to issue #11849, this patch proposes to use VirtualAlloc/VirtualFree to allocate the Python allocator's memory arenas (rather than malloc() / free()). It might help release more memory if there is some fragmentation, although I don't know how Microsoft's malloc() works.

The patch looks good to me.
To study Microsoft's malloc, see VC\crt\src\malloc.c. Typically, it uses HeapAlloc from the CRT heap, unless it's in 32-bit mode, and __active_heap is either __V6_HEAP or __V5_HEAP. This is determined at startup by __heap_select, inspecting an environment variable __MSVCRT_HEAP_SELECT. If that's not set, the CRT heap is used.
The CRT heap, in turn, is created with HeapCreate (no flags).
As an alternative approach, Python could consider completely dropping obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH) instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the GIL).
If we take the route proposed by this patch, I recommend also dropping all other CRT malloc() calls in Python, and make allocations from the process heap instead (that's a separate issue, though).

> The patch looks good to me.
> 
> To study Microsoft's malloc, see VC\crt\src\malloc.c. Typically, it
> uses HeapAlloc from the CRT heap, unless it's in 32-bit mode, and
> __active_heap is either __V6_HEAP or __V5_HEAP. This is determined at
> startup by __heap_select, inspecting an environment variable
> __MSVCRT_HEAP_SELECT. If that's not set, the CRT heap is used.
Ah, right, I guessed it was using HeapAlloc indeed. What would be more
interesting is how HeapAlloc works :)
I think it would be nice to know whether the patch has a chance of being
useful before committing it. I did it as a thought experiment after the
similar change was committed for Unix, but I'm not an expert in Windows
internals. Perhaps HeapAlloc deals fine with fragmentation? Tim, Brian,
do you know anything about this?
> As an alternative approach, Python could consider completely dropping
> obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH)
> instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the
> GIL).
I'm not sure that would serve the same purpose as obmalloc, which
(AFAIU) is very fast at the expense of compacity.

'fraid not. I've never had to dig into the allocation stuff at this level.

> I think it would be nice to know whether the patch has a chance of being
> useful before committing it. I did it as a thought experiment after the
> similar change was committed for Unix, but I'm not an expert in Windows
> internals. Perhaps HeapAlloc deals fine with fragmentation?
Unfortunately, the implementation of HeapAlloc isn't really documented.
If Reactos is right, it looks like this: http://bit.ly/t2NPHh
Blocks < 1024 bytes are allocated from per-size free lists.
Blocks < Heap->VirtualMemoryThreshold are allocated through the free
list for variable-sized blocks of the heap.
Other blocks are allocated through ZwAllocateVirtualMemory, adding
sizeof(HEAP_VIRTUAL_ALLOC_ENTRY) in the beginning. I think this header
will cause malloc() to allocate one extra page in front of an arena.
>> As an alternative approach, Python could consider completely dropping
>> obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH)
>> instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the
>> GIL).
> 
> I'm not sure that would serve the same purpose as obmalloc, which
> (AFAIU) is very fast at the expense of compacity.
I'd expect that LFH heaps are also very fast. The major difference I can
see is that blocks in the LFH heap still have an 8-byte header (possibly
more on a 64-bit system). So I wouldn't expect any speed savings, but
(possibly relevant) memory savings from obmalloc.

> Tim, Brian, do you know anything about this?
Unfortunately, no. It's on my todo list of things to understand but I don't see that happening in the near future.
I'm willing to run tests or benchmarks for this issue, but that's likely the most I can provide.

Le mardi 29 novembre 2011 à 22:39 +0000, Brian Curtin a écrit :
> Brian Curtin <brian@python.org> added the comment:
> 
> > Tim, Brian, do you know anything about this?
> 
> Unfortunately, no. It's on my todo list of things to understand but I
> don't see that happening in the near future.
> 
> I'm willing to run tests or benchmarks for this issue, but that's
> likely the most I can provide.
Benchmarks would be nice indeed.

Here is a benchmark. Based on my assumption that this patch may reduce allocation overheads due to minimizing padding+fragmentation, it allocates a lot of memory, and then waits 20s so you can check in the process explorer what the "Commit Size" of the process is.
For the current 3.3 tree, in 32-bit mode, on a 64-bit Windows 7 installation, I get 464,756K for the unpatched version, and 450,436K for the patched version.
This is a 3% saving, which seems good enough for me.

Here is an updated patch.

Martin, do you think your latest patch can be committed?

Antoine's request for benchmarks still stands. I continue to think that it should be applied even in absence of benchmarks. In the absence of third opinions on this specific aspect, I don't think it can be applied.

I can't speak for Antoine, but I guess that the result of pybench
would be enough to make sure it doesn't introduce any regression
(which would be *really* suprising).
As for the memory savings, the benchmark you posted earlier is
conclusive enough IMO (especially since the it can be difficult to
come up with a scheme leading to heap fragmentation).

I asked for benchmarks because I don't know anything about Windows virtual memory management, but if other people think this patch should go in then it's fine.
The main point of using VirtualAlloc/VirtualFree was, in my mind, to allow *releasing* memory in more cases than when relying on free() (assuming Windows uses some sbrk() equivalent). But perhaps Windows is already tuned to release memory on most free() calls.

Ah ok. I guess tuples.py then indeed demonstrates a saving. I'll apply the patch.

Set also issue #3329 which proposes an API to define memory allocators.

I tested VirtualAlloc/VirtualFree versus malloc/free on Windows 7 SP1 64-bit. On my small test, when using VirtualAlloc/VirtualFree, the memory peak is lower (ex: 58.1 MB vs 59.0), and the memory usage is the same or sometimes lower. The difference is small, malloc() implementation on Windows 7 is efficient! But I am in favor of using VirtualAlloc/VirtualFree because it is the native API and the gain may be bigger on a real application.
--
I used the following script for my test:
https://bitbucket.org/haypo/misc/raw/98eb42a3ed2144141d62c75e3d07933839fe2a0c/python/python_memleak.py
I reused get_process_mem_info() code from psutil to get current and peak memory usage (I failed to install psutil, I don't understand why).
I also replace func() of my script with tuples.py to create many tuples.
--
Python < 3.3 wastes a lot of memory with python_memleak.py. Python 3.3 behaves much better thanks to the usage of mmap() on Linux, and the fixed threshold on 64-bit (min=512 bytes, instead of 256).

Martin von Loewis: "If we take the route proposed by this patch, I recommend also dropping all other CRT malloc() calls in Python, and make allocations from the process heap instead (that's a separate issue, though)."
=> see issue #18203 

haypo> I tested VirtualAlloc/VirtualFree versus malloc/free
haypo> on Windows 7 SP1 64-bit. On my small test, ...
I realized that I was no precise: I tried attached va.diff patch. I didn't try to replace completly malloc().

> Ah ok. I guess tuples.py then indeed demonstrates a saving. I'll apply the patch.
According to my test, the memory usage is a little bit better with the patch. So Martin:,do you plan to commit the patch?
Or is a benchmark required? Or should check first check the Low Fragmentation Allocator?
I plan to test the Low Fragmentation Allocator, at least on Windows 7. But I prefer to do it later, I'm working on the PEP 445 right now.

> I plan to test the Low Fragmentation Allocator, at least on Windows 7.
I don't think it can be any better than raw mmap() / VirtualAlloc()...

>> I plan to test the Low Fragmentation Allocator, at least on Windows 7.
> I don't think it can be any better than raw mmap() / VirtualAlloc()...
I mean using the Low Fragmentation Allocator for PyObject_Malloc()
instead of pymalloc.
Martin wrote (msg148605):
"As an alternative approach, Python could consider completely dropping
obmalloc on Windows, and using a Windows Low Fragementation Heap (LFH)
instead, with HEAP_NO_SERIALIZE (as the heap would be protected by the
GIL)."

Ok, I'm going to commit this patch. Any further revisions (including reversions) can be done then.

New changeset 44f455e6163d by Martin v. Löwis in branch 'default':
Issue #13483: Use VirtualAlloc in obmalloc on Windows.
http://hg.python.org/cpython/rev/44f455e6163d