Embedded Python fails to compile

Question

I installed python 3.9.1 following the instructions here https://www.ramoonus.nl/2020/10/06/how-to-install-python-3-9-on-raspberry-pi/ and set it as default python interpreter. I got my compiling and linking parameters for embedded Python following the instructions here https://docs.python.org/3.9/extending/embedding.html#compiling-and-linking-under-unix-like-systems which I have added in my makefile:

COMPILE_FLAGS += -I/usr/local/opt/python-3.9.1/include/python3.9 -I/usr/local/opt/python-3.9.1/include/python3.9 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall

and

LINK_LIBS += -L/usr/local/opt/python-3.9.1/lib/python3.9/config-3.9-arm-linux-gnueabihf -L/usr/local/opt/python-3.9.1/lib -lcrypt -lpthread -ldl -lutil -lm

Everything seems OK but when I try to compile I get:

/usr/bin/g++ coremath/UKF_AoA_TEV2A.o coremath/inv.o coremath/mrdivide.o coremath/SPpredict.o coremath/SPmeas.o coremath/asin.o coremath/sqrt.o coremath/norm.o coremath/quatprod.o coremath/rtGetInf.o coremath/rtGetNaN.o coremath/rt_nonfinite.o coremath/compass.o coremath/u_filter.o coremath/u_mat.o coremath/u_math.o include/gyrobias.o include/sensor_gps.o include/sensor_imu_bmx.o include/sensor_imu_st.o include/sensor_p.o include/sensor_bmp280_static.o include/sensor_bmp280_dynamic.o include/sensor_temp.o include/sensormaster.o include/sensormaster_compass.o include/sensormaster_debug.o include/sensormaster_gps.o include/sensormaster_imu.o include/sensormaster_pressure.o include/sensormaster_temp.o include/sensormaster_tracker.o include/talosudpclient.o include/talosudpserver.o include/talos_usb_server.o include/talos_iphone_server.o include/talos_adsb.o include/u_consts.o include/bargraph.o include/u_utils.o aeolus_server.o include/u_geomag.o libraries/GeomagnetismLibrary.o -o ../bin/aeolus_server -lwiringPi -L/usr/local/opt/python-3.9.1/lib/python3.9/config-3.9-arm-linux-gnueabihf -L/usr/local/opt/python-3.9.1/lib -lcrypt -lpthread -ldl -lutil -lm -lm -I. -I./include -I./coremath -I./libraries -pthread -Wuninitialized -O2 -O3 -D_GNU_SOURCE -DSIMULATOR -std=gnu++11 -Wall -Wextra -I/usr/local/opt/python-3.9.1/include/python3.9 -I/usr/local/opt/python-3.9.1/include/python3.9 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 coremath/compass.o: In function compassCheckCalibrate': /home/pi/aeolus/src/coremath/compass.c:321: undefined reference to PyUnicode_DecodeFSDefault' /home/pi/aeolus/src/coremath/compass.c:326: undefined reference to PyImport_Import' /home/pi/aeolus/src/coremath/compass.c:338: undefined reference to PyObject_GetAttrString' /home/pi/aeolus/src/coremath/compass.c:340: undefined reference to PyCallable_Check' /home/pi/aeolus/src/coremath/compass.c:341: undefined reference to PyTuple_New' /home/pi/aeolus/src/coremath/compass.c:346: undefined reference to PyLong_FromLong' /home/pi/aeolus/src/coremath/compass.c:347: undefined reference to PyTuple_SetItem' /home/pi/aeolus/src/coremath/compass.c:348: undefined reference to PyLong_FromLong' /home/pi/aeolus/src/coremath/compass.c:349: undefined reference to PyTuple_SetItem' /home/pi/aeolus/src/coremath/compass.c:350: undefined reference to PyObject_CallObject' /home/pi/aeolus/src/coremath/compass.c:352: undefined reference to PyLong_AsLong' coremath/compass.o: In function _Py_DECREF': /usr/local/opt/python-3.9.1/include/python3.9/object.h:430: undefined reference to _Py_Dealloc' /usr/local/opt/python-3.9.1/include/python3.9/object.h:430: undefined reference to _Py_Dealloc' /usr/local/opt/python-3.9.1/include/python3.9/object.h:430: undefined reference to _Py_Dealloc' /usr/local/opt/python-3.9.1/include/python3.9/object.h:430: undefined reference to _Py_Dealloc' /usr/local/opt/python-3.9.1/include/python3.9/object.h:430: undefined reference to _Py_Dealloc' coremath/compass.o:/usr/local/opt/python-3.9.1/include/python3.9/object.h:430: more undefined references to _Py_Dealloc' follow include/u_utils.o: In function python_init(char*)': /home/pi/aeolus/src/include/u_utils.c:344: undefined reference to Py_DecodeLocale' /home/pi/aeolus/src/include/u_utils.c:349: undefined reference to Py_SetProgramName' /home/pi/aeolus/src/include/u_utils.c:350: undefined reference to `Py_Initialize' collect2: error: ld returned 1 exit status makefile:83: recipe for target 'aeolus_server' failed make: *** [aeolus_server] Error 1

I tried a simpler test with the following code (test.c) :

int
main(int argc, char *argv[])
{
 wchar_t *program = Py_DecodeLocale(argv[0], NULL);
 if (program == NULL) {
 fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
 exit(1);
 }
 Py_SetProgramName(program); /* optional but recommended */
 Py_Initialize();
 PyRun_SimpleString("from time import time,ctime\n"
 "print('Today is', ctime(time()))\n");
 Py_Finalize();
 PyMem_RawFree(program);
 return 0;
}

and then

gcc -I/usr/local/opt/python-3.9.1/include/python3.9 -I/usr/local/opt/python-3.9.1/include/python3.9 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -c test.c -o test.o

and

gcc -L/usr/local/opt/python-3.9.1/lib/python3.9/config-3.9-arm-linux-gnueabihf -L/usr/local/opt/python-3.9.1/lib -lcrypt -lpthread -ldl -lutil -lm -o test.o

and this time got

/usr/lib/gcc/arm-linux-gnueabihf/4.9/../../../arm-linux-gnueabihf/crt1.o: In function '_start': /build/glibc-P1SmLh/glibc-2.19/csu/../ports/sysdeps/arm/start.S:119: undefined reference to 'main' collect2: error: ld returned 1 exit status

What could the problem be?

Edit: After changing the linker command to:

gcc test.o -L/usr/local/opt/python-3.9.1/lib/python3.9/config-3.9-arm-linux-gnueabihf -L/usr/local/opt/python-3.9.1/lib -lcrypt -lpthread -ldl -lutil -lm -o test

which seems to have worked but threw me a bunch of other errors:

test.o: In function main': /home/pi/Downloads/test.c:6: undefined reference to Py_DecodeLocale' /home/pi/Downloads/test.c:11: undefined reference to Py_SetProgramName' /home/pi/Downloads/test.c:12: undefined reference to Py_Initialize' /home/pi/Downloads/test.c:13: undefined reference to PyRun_SimpleStringFlags' /home/pi/Downloads/test.c:15: undefined reference to Py_Finalize' /home/pi/Downloads/test.c:16: undefined reference to `PyMem_RawFree' collect2: error: ld returned 1 exit status

This seems more serious. Any ideas?

Answer 1

Your linker command was indeed wrong, but you seem to have corrected it in your edit.

Don't hardcode the Python compiler and linker flags, use the python3-config tool instead. You should also use the --embed option, otherwise the -lpython3.9 flag won't be included, and it won't actually link to the Python library.

gcc $(python3-config --cflags) -fPIE -c test.c -o test.o # Compile
gcc test.o $(python3-config --embed --ldflags) -o test # Link
./test # Run

Note that the behavior of the python3-config script was changed in Python 3.8: https://docs.python.org/3/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

• tryfon: Thanks a lot! It finally worked after adjusting my linker command to: gcc test.o -L/usr/local/opt/python-3.9.1/lib/python3.9/config-3.9-arm-linux-gnueabihf -L/usr/local/opt/python-3.9.1/lib -lcrypt -lpthread -ldl -lutil -lm -lpython3.9 -o test