Embedding Python in Linux program

Question

I am currently trying to embed Python in my C++ application to give the user advanced scripting possibilities. My program is working great on Windows so far (it is fully working), and now I'm trying to do the same on GNU/Linux (Debian 7 for now) but this is giving me much more trouble than I expected. First, I download the python.tar.gz and compile it from source with enable-shared option to get the fPIC option:
./configure --enable-shared --prefix=/opt/python
make && make altinstall
Then, I install numpy thanks to pip : python3.4 -m pip install numpy. Easy.

Last, I copy the installation to another location (yes, it should be deployed anywhere), in my home directory to be precise, and name it python_install. This copy seems to give me much pain.

On the code side, I call Py_Initialize which is working good. Here is what I have done so far :

#include "Python.h"
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "numpy/arrayobject.h"
#include <iostream>
int InitNumpy()
{
 import_array();
}
int main()
{
 std::string python_home = "/home/xxxx/dev/test-python/python_install";
 setenv("PYTHONHOME", python_home.c_str(),1 );
 Py_Initialize();
 std::cout << "Importing Numpy... ";
 int cr = InitNumpy();
 std::cout << cr << std::endl;
 return 0;
}

ImportError: numpy.core.multiarray failed to import
I think the error is pretty clear, Python can't find any libraries including numpy. But I have tried everything from setting PYTHONPATH to playing with -Wl,-rpath to set additional directories.... even PySys_SetPath... What is working on Windows is failing here on Linux. Any ideas would be welcome ! Thanks.

EDIT : Here is the makefile I used (corrected):

CC=g++
CFLAGS= -Ipython_install/include/python3.4m -Ipython_install/lib/python3.4/site-packages/numpy/core/include -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall
#Wrong
#LDFLAGS= -Lpython_install/lib/python3.4/config-3.4m -lpython3.4m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-rpath,\$${ORIGIN}/python_install/lib
#Right
LDFLAGS= -Lpython_install/lib/ -lpython3.4m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-rpath,\$${ORIGIN}/python_install/lib
EXEC=test-python
all: $(EXEC)
test-python: test-python.o
 $(CC) -o $@ $^ $(LDFLAGS)
%.o: %.cpp
 $(CC) -o $@ -c $< $(CFLAGS)
.PHONY: clean mrproper
clean:
 rm -rf *.o
mrproper: clean
 rm -rf $(EXEC)

Options are coming directly from the python3.4m-config executable...

• Dunes: I can't reproduce your problem. Followed all the steps you gave and it worked. All I can think of is that numpy doesn't actually exist under the python home you've given. Have you tried running an interactive interpreter of your installation and importing numpy (making sure PYTHONPATH is not set)?
• poukill: Yes, me too. Long story short : I made a first installation of Python and copy the installation in my small project. I don't know why but numpy was referring to the first install. 'strace' gave me the clue. I deleted this first installation and now it works... But I hope I will be able to integrate nicely with other Python installation in the future otherwise some users will definitely complain...
• Dunes: What I did find that pip was hard coded to use the original install path to find the python executable. That is, it wouldn't try to use the python executable that is shared a directory with. It may be that pip's install path is also hard coded -- meaning you wouldn't be able to move the python installation about.
• poukill: @Dunes : Yes, you may be right. When I try to use pip on the moved folder, I get a /usr/bin/ld: cannot find -lpython3.4m. Real problem is : I am supposed to deploy my application and my python wrappers in .deb archive via apt-get... So I might force the python installation somewhere. I have to think about it.
• poukill: @Dunes I finally found what was my problem. The makefile was not correct. I linked statically on libpython.a but additional installations like numpy used libpython3.4.so, and it seems to provoke some weird conflicts I don't understand. Well, now that I use dynamic linking (shared objects) on both sides it is working beautifully. 5 days of trouble for one weird option...

Answer 1

The short answer is : don't link statically on Python. All Python modules link dynamically on libpython3.4.so, so the C++ program has to do the same. To sum up Python embedding :

1. Link dynamically on Python (cf corrected Makefile). Use rpath to locate the python .so.
2. In the C++ code, set PYTHONHOME environment variable to the Python installation or use Py_SetPythonHome() function.
3. Call Py_Initialize();

4. If you want to import custom module, add the module path to Python path.

   PyObject *sys = PyImport_ImportModule("sys");
   PyObject *path = PyObject_GetAttrString(sys, "path");
   PyList_Append(path, PyUnicode_FromString(sys_path.toUtf8().data()));
   

• sAguinaga: Would you mind adding an example of how you set PYTHONHOME in the C++ code?
• poukill: I think using Py_SetPythonHome do the same and should be used. cf docs.python.org/3.6/c-api/init.html

Answer 2

You say "python can't find any libraries", have you tried starting python and importing them? If it's an issue with importing directly in python you are going to have to recheck PYTHOPATH is correct, or perhaps your installed files weren't installed with permission settings to execute. (I've had that happen twice).

Also, have you checked out the tips for compiling on unix like systems from the manual?.

It is not necessarily trivial to find the right flags to pass to your compiler (and linker) in order to embed the Python interpreter into your application, particularly because Python needs to load library modules implemented as C dynamic extensions (.so files) linked against it.

To find out the required compiler and linker flags, you can execute the pythonX.Y-config script which is generated as part of the installation process (a python3-config script may also be available)

How do you compile your code?

EDIT

I've tried my best to reproduce your error but can't do it on my machine. The farthest I got is that I think that your analysis of the error is wrong.

I think the error is pretty clear, Python can't find any libraries including numpy.

The reason behind having an import_array() is to check if there's a numpy and if the numpy version you have matches the installed version of python. In the source it's defined as a macro (i.e. for np for py2):

#if PY_VERSION_HEX >= 0x03000000
#define NUMPY_IMPORT_ARRAY_RETVAL NULL
#else
#define NUMPY_IMPORT_ARRAY_RETVAL
#endif
#define import_array() {if (_import_array() < 0) {PyErr_Print();
PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to
import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }

Since that's the error you see, I've reason to suspect that your pip installed the "wrong" numpy for your python version. But then again based on your makefile there is a numpy inside your python3.4 folders, if it hadn't it would throw an error No such file or directory. Additionally you say you even had to use the LD_LIBRARY_PATH (not recommended btw) to start python with numpy, that just means that numpy really is not properly pointed to by any path that is currently defined for your python, including the site-packages folder you have in your compiled python (which is searched through by default).

I suspect that you have a preexisting python installed by default, and your system wide environment paths are set to it. But that's a really weird statement considering you explicitly link the numpy in python3 folder AND you setenv PYTHONHOME.

I'm sorry I couldn't be of more help, but I can't reproduce the issue without actually doing the python build from scratch (also not currently on debian). Good luck, keep me posted?

• poukill: I edited my first post to show full code. If I launch the Python executable itself (with LD_LIBRARY_PATH to help him find the .so), it's working great, numpy is importing correctly. On the other hand, when I'm calling import_array from the C++ code, I always get an import error !
• poukill: It finally worked using rpath and strace on my very small example. But I could not transfer that in my big project. Have to find out why. I will keep the community inform !
• ljetibo: @poukill I tried my best, but can't get python to behave like yours, even when I unset the paths linking seems to work for me.
• poukill: I started from scratch again, and it is also working for me.... even if I move my program. That is good news. But, if I install another Python elsewhere with the exact same method (but without numpy), the primary installation won't work anymore, and I get my "ImportError: numpy.core.multiarray failed to import" back... If I install numpy on the last Python install, then, both applications will work. To sum up : 1- Install Python and numpy with test-python.cpp. Compile and launch. Working! 2- Move to another folder. Still working. 3- Reinstall Python on the old folder. Broke the new one.
• ljetibo: Looks like I was in the neigbourhood at least. Hopefully I helped a little. This is bad news for you though, this means it's not, as is, deployable as a stand-alone. Not outside of clean install, or at least virtualenv. Building numpy inplace is an overkill. Additionally I was scouring the boost library installs to try and catch a hint of a solution when I ran into this which you might find useful as well.