Embedded python: multiprocessing not working

Question

I'm using Python 3.1.4 that is embedded as a scripting environment in an application(x64). So far I have encountered a lot of limitations with the embedded python. I don't know if it is normal or if the programmers of the application have blocked some functionalities.

For example the following code isn't working:

from multiprocessing import Process
def f(name):
 print('hello', name)
if __name__ == '__main__':
 p = Process(target=f, args=('bob',))
 p.start()
 p.join()
# --> error in forking.py: 'module' object has no attribute 'argv'
# print(sys.argv) gives the same error

sys.executable return the path to the application.

I've tried this as wel:

multiprocessing.forking.set_executable('C:\Python31\python.exe')
multiprocessing.set_executable('C:\Python31\python.exe')

Without success.

Is there a workaround possible ? It is very unlikely that I would have the leverage to make the developers of the application change something in their code.

Thanks

EDIT

I got it to work by adding the following:

sys.argv = ['c:/pathToScript/scipt.py']

I needed this line as well:

multiprocessing.set_executable('C:/Python31/python.exe')

Otherwise an other instance of the application open instead of running the code.

The only problem I have left is that I can't use the methods that control the application itself (like: create_project(), add_report(),..). My primary goal was to be able to call multiple methods without the need to wait for the first one to finish completion. But I think this is just not possible.

• Ionut Hulub: are you sure you don't mean to use threading instead of multiprocessing?
• F. Justin: @ Ionut Hulub: I've problems as well with threading, something with the GIL I presume. That's why I tried with multiprocessing @ NPE: Windows 7, 64bit

Answer 1

By default, sys.argv is not available in embedded code:

Embedding Python

The basic initialization function is Py_Initialize(). This initializes the table of loaded modules, and creates the fundamental modules builtins, __main__, and sys. It also initializes the module search path (sys.path).

Py_Initialize() does not set the "script argument list" (sys.argv). If this variable is needed by Python code that will be executed later, it must be set explicitly with a call to PySys_SetArgvEx(argc, argv, updatepath) after the call to Py_Initialize()

On Windows, multiprocessing must spawn new processes from scratch. It uses a command line switch --multiprocessing-fork to distinguish child processes, and also transmits the original argv from parent to child.

Assigning sys.argv = ['c:/pathToScript/scipt.py'] before creating subprocesses, like you discovered, would seem to be a good workaround.

A second relevant piece of documentation is that of multiprocessing.set_executable():

Sets the path of the Python interpreter to use when starting a child process. (By default sys.executable is used). Embedders will probably need to do some thing like

set_executable(os.path.join(sys.exec_prefix, 'pythonw.exe'))
before they can create child processes. (Windows only)

Answer 2

If it does not work, try to set sys._base_executable = os.path.join(sys.exec_prefix, 'pythonw.exe')

PyObject *msys = PyImport_ImportModule("sys");
PyObject *pyString = PyUnicode_FromWideChar(L"path/to/pythonw.exe", -1);
PyObject_SetAttrString(msys, "executable", pyString);
PyObject_SetAttrString(msys, "_base_executable", pyString);
Py_XDECREF(pyString);