Python interpreter as a c++ class

Question

I am working on embedding python in to c++. In some peculiar case I require two separate instances of the interpreter in same thread.

Can I wrap Python interpreter in to a c++ class and get services from two or more class instances?

Answer 1

I have used Py_NewInterpreter for different interpreters in different threads, but this should also work for several interpreters within one thread:

In the main thread:

Py_Initialize();
PyEval_InitThreads();
mainThreadState = PyEval_SaveThread();

For each interpreter instance (in any thread):

// initialize interpreter
PyEval_AcquireLock(); // get the GIL
myThreadState = Py_NewInterpreter();
... // call python code
PyEval_ReleaseThread(myThreadState); // swap out thread state + release the GIL
... // any other code
// continue with interpreter
PyEval_AcquireThread(myThreadState); // get GIL + swap in thread state
... // call python code
PyEval_ReleaseThread(myThreadState);
... // any other code
// finish with interpreter
PyEval_AcquireThread(myThreadState);
... // call python code
Py_EndInterpreter(myThreadState);
PyEval_ReleaseLock(); // release the GIL

Note that you need a variable myThreadState for each interpreter instance!

Finally the finish in the main thread:

PyEval_RestoreThread(mainThreadState);
Py_Finalize();

There are some restrictions with using several interpreter instances (they seem not to be totally independent), but in most cases this does not seem to cause problems.

Answer 2

Callin Py_Initialize() twice won't work well, however Py_NewInterpreter can work, depending on what you're trying to do. Read the docs carefully, you have to hold the GIL when calling this.

• Amol Gawai: I guess I will not get a straightforward answer to my question. Your answer has given me some inputs on which I can start work. Py_NewInterpreter seems to be the correct option to start exploring the scenario I have described. Based on this I am accepting your answer.

Answer 3

You can, but I'd recommend you not to re-implement a Python interpreter when there is a standard implementation. Use boost::python to interface with Python.

• Amol Gawai: boost python uses python c apis. is it possible to start the interpreter twice by calling Py_Initialize()?

Answer 4

mosaik's answer did not work in my situation where my module is a plugin to a host application that already initializes python. I was able to get it to work with the following code.

// initialize interpreter
::PyEval_InitThreads();
::PyThreadState *mainThread = ::PyThreadState_Get();
myState = ::Py_NewInterpreter();
... // call python code
::PyThreadState_Swap(mainThread);
... // any other code
mainThread = ::PyThreadState_Swap(myState)
... // call python code
::PyThreadState_Swap(mainThread)
... // any other code
// finished with interpreter
mainThread = ::PyThreadState_Swap(myState)
::Py_EndInterpreter(myState);
::PyThreadState_Swap(mainThread)

When I called PyEval_AcquireLock() the program blocked and the function did not return. Further, calling PyEval_ReleaseThread(myState) seemed to invalidate the interpreter also.

Answer 5

I don't think you are the first person to want to do this, unfortunately I believe it is not possible. Are you able to run the python interperters as separate processes and use RPC?

Answer 6

• You can let the python interpreter live outside of your application memory space. Just embed the interpreter in a DLL.
• You can set up & save python contexts to simulate two different interpreters.