[Python-ideas] Preventing out of memory conditions

Tue Jan 1 04:22:29 CET 2013

Within CPython the way the C API is today it is too late by the time the
code to raise a MemoryError has been called so capturing all places that
could occur is not easy. Implementing this at the C level malloc later
makes more sense. Have it dip into a reserved low memory pool to satisfy
the current request and send the process a signal indicating it is running
low. This approach would also work with C extension modules or an embedded
Python.
I'd expect this already exists but I haven't looked for one.
Having a thread polling memory use it not generally wise as that is polling
rather than event driven and could easily miss low memory situations before
it is too late and a failure has already happened (allocation demand can
come in large spikes depending on the application).
OSes running processes in constrained environments or ones where the
resources available can be reduced by the OS later may already send their
own warning signals prior to outright killing the process but that should
not preclude an application being able to monitor and constrain itself on
its own without needing the OS to do it.
-gps
On Mon, Dec 31, 2012 at 4:11 PM, Ben Hoyt <benhoyt at gmail.com> wrote:
> Interesting idea, though I don't think it's something that should be a
> Python language extension. For instance, iOS (typically more
> resource-constrained) sends the application signals when system memory is
> getting low so it can free stuff -- this is done at the OS level. And I
> think that's the right place, because this will almost certainly be setup-
> and system-dependent. For instance, it would depend hugely on whether
> there's a virtual memory manager, and how it's configured.
>> I'd say your best bet is to write a little library that does the
> appropriate thing for your needs (your system or setup). Say starts a
> thread that checks the system's free memory every so often and sends your
> application a signal/callback saying "we're getting low, free some of your
> caches". It could even send a "level flag" to your callback saying "fairly
> low", "very low", or "critically low" -- I think iOS does this.
>> -Ben
>>>> On Tue, Jan 1, 2013 at 11:16 AM, Max Moroz <maxmoroz at gmail.com> wrote:
>>> Sometimes, I have the flexibility to reduce the memory used by my
>> program (e.g., by destroying large cached objects, etc.). It would be
>> great if I could ask Python interpreter to notify me when memory is
>> running out, so I can take such actions.
>>>> Of course, it's nearly impossible for Python to know in advance if the
>> OS would run out of memory with the next malloc call. Furthermore,
>> Python shouldn't guess which memory (physical, virtual, etc.) is
>> relevant in the particular situation (for instance, in my case, I only
>> care about physical memory, since swapping to disk makes my
>> application as good as frozen). So the problem as stated above is
>> unsolvable.
>>>> But let's say I am willing to do some work to estimate the maximum
>> amount of memory my application can be allowed to use. If I provide
>> that number to Python interpreter, it may be possible for it to notify
>> me when the next memory allocation would exceed this limit by calling
>> a function I provide it (hopefully passing as arguments the amount of
>> memory being requested, as well as the amount currently in use). My
>> callback function could then destroy some objects, and return True to
>> indicate that some objects were destroyed. At that point, the
>> intepreter could run its standard garbage collection routines to
>> release the memory that corresponded to those objects - before
>> proceeding with whatever it was trying to do originally. (If I
>> returned False, or if I didn't provide a callback function at all, the
>> interpreter would simply behave as it does today.) Any memory
>> allocations that happen while the callback function itself is
>> executing, would not trigger further calls to it. The whole mechanism
>> would be disabled for the rest of the session if the memory freed by
>> the callback function was insufficient to prevent going over the
>> memory limit.
>>>> Would this be worth considering for a future language extension? How
>> hard would it be to implement?
>>>> Max
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>>>> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>>-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121231/2c0c7ae8/attachment.html>