Python threads in C

Question

I'am writing a multi-threaded program in C. Before creating the threads, a global python environment is initialized by calling Py_Initialize(). Then, in every created thread, the global python environment is shared, and each thread calls a python method with the parameters converted in C. Everything works well until here.

When I use time.sleep() in loaded python modules, the C program raises a Segmentation Fault. Furthermore, the loaded python module is supposed to load another C lib to continue the work. I've written the following stupid counter lib to test it:

# python part, call the counter function
lib = ctypes.cdll.LoadLibrary(libpycount.so)
for i in xrange(10):
 lib.count()

// C part, dummy countings
#include <stdio.h>
int counter = 1;
void
count() {
 printf("counter:%d \n", counter);
 counter++;
}

I guess that it might be because I didn't manage the complex thread creation in the right way. And I've found Non-Python created threads in the python doc.

Any ideas or suggestions?

• Thomas Wouters: Did you actually follow the advice in that document you found? How are you grabbing the GIL in the code that calls into Python?
• alwinlin: @Thomas Wouters I've used PyGILState_STATE gstate;, gstate = PyGILState_Ensure(); and PyGILState_Release(gstate); as they are mentioned in the doc.

Answer 1

My problem has been solved. You may have your problems more particular, so I'm trying to write my solution in a more generic way here. Hope it helps.

- In main C thread

• initialize the Python environment at the very begining:

/*define a global variable to store the main python thread state*/
PyThreadState * mainThreadState = NULL;
if(!Py_IsInitialized())
 Py_Initialize();
mainThreadState = = PyThreadState_Get();

• Then start the C threads:

pthread_create(pthread_id, NULL, thread_entrance, NULL);

- In every thread, or we can say in the body of thread_entrance function

• prepare the environment:

/*get the lock and create new python thread state*/
PyEval_AcquireLock();
PyInterpreterState * mainInterpreterState = mainThreadState->interp;
PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
PyEval_ReleaseLock(); /*don't forget to release the lock*/
/*
 * some C manipulations here
 */

• put embeded Python code here:

/*get the lock and put your C-Python code here*/
PyEval_AcquireLock();
PyThreadState_Swap(myThreadState); /*swap your python thread state*/
PyEval_CallObject(py_function, py_arguments);
/*or just something like PyRun_SimpleString("print \"hello world\""); for test*/
PyThreadState_Swap(NULL); /*clean the thread state before leaving*/
PyEval_ReleaseLock();

- back to main C thread

• when every thread finishes their works, finalize the python environment

pthread_join(pthread_id, NULL);
PyEval_RestoreThread(mainThreadState);
Py_Finalize();

Answer 2

Question is whether the Python interpreter is thread safe -- This is what the documentation says about running multiple interpreters in the same process space;

Bugs and caveats: Because sub-interpreters (and the main interpreter) are part of the same process, the insulation between them isn't perfect -- for example, using low-level file operations like os.close() they can (accidentally or maliciously) affect each other's open files. Because of the way extensions are shared between (sub-)interpreters, some extensions may not work properly; this is especially likely when the extension makes use of (static) global variables, or when the extension manipulates its module's dictionary after its initialization. It is possible to insert objects created in one sub-interpreter into a namespace of another sub-interpreter; this should be done with great care to avoid sharing user-defined functions, methods, instances or classes between sub-interpreters, since import operations executed by such objects may affect the wrong (sub-)interpreter's dictionary of loaded modules. (XXX This is a hard-to-fix bug that will be addressed in a future release.)

...and I don't think that Python threads are the same thing as native threads such as found in C/C++