[Python-Dev] Bug in generator if the generator in created in a C thread

Thu Mar 29 23:26:42 CEST 2012

On 29/03/2012 21:35, Tim Lesher wrote:
> From a theoretical standpoint, I can't quite decide what the real error is:
>> 1) the fact that PyGILState_Release() destroys a temporary thread
> state that may still be referenced by some objects, or
> 2) the fact that some code is trying to keep frame objects after the
> creating thread state no longer exists.
>> This week I've been leaning toward 2), but then I realized that
> keeping frames post-thread-death is not that uncommon (for example,
> debuggers and other diagnostic techniques like
> http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html).

The problem is not the frame, but the Python thread state referenced by 
the frame. It's a private attribute. My patch just updates this 
reference before running the generator (and it clears the reference when 
the generator excution is stopped or finished).
> Locally we added some (unfortunate) code to our 3.1.2 port to wrap
> PyGILState_Ensure(), which I thought had sidestepped the issue for us:
>> void takeGIL()
> {
> PyGILState_Ensure();
> // This has the side effect of keeping such thread states alive until
> // the interpreter is finalized; however, all thread state objects get
> // unconditionally deleted during Py_Finalize, so they won't leak.
> PyThreadState* pThreadState = PyGILState_GetThisThreadState();
> if (pThreadState->gilstate_counter == 1)
> {
> ++pThreadState->gilstate_counter;
> }
> }
>> But clearly that can't be a correct answer (and it may not even be a
> functioning one, given that I'm seeing a similar issue again).

You may leak memory if your threads have a short lifetime and you create 
many threads. For example if one thread is only used to process one 
request and then is destroyed.
Victor