homepage

To build python for embedded Linux systems,
(http://www.yoctoproject.org/docs/2.3.1/yocto-project-qs/yocto-project-qs.html)
The 32-bit python's multiprocessing.BoundedSemaphore could not work.
1. Prerequisite
- Build 32bit python on 64bit host, add '-m32' to gcc
- Cross compling on linux platform, not have `-pthread'
 In configure.ac, ac_cv_pthread=no while cross compiling
 ...
 2001 AC_MSG_CHECKING(whether $CC accepts -pthread)
 2002 AC_CACHE_VAL(ac_cv_pthread,
 2003 [ac_save_cc="$CC"
 2004 CC="$CC -pthread"
 2005 AC_RUN_IFELSE([AC_LANG_SOURCE([[
 [snip]
 2018 ]])],[ac_cv_pthread=yes],[ac_cv_pthread=no],[ac_cv_pthread=no])
 ...
 — Macro: AC_RUN_IFELSE (input, [action-if-true], [action-if-false], [action-if-cross-compiling])
2. Reproduce with simplifying the prerequisite
$ git clone https://github.com/python/cpython.git; cd ./cpython
$ ac_cv_pthread=no CC="gcc -m32" ./configure
$ make -j 10
$ ./python -c "import multiprocessing; pool_sema = multiprocessing.BoundedSemaphore(value=1); pool_sema.acquire(); pool_sema.release()"
Traceback (most recent call last):
 File "<string>", line 1, in <module>
ValueError: semaphore or lock released too many times
And the issue also caused invoking put of 'multiprocessing.Queue' hung.
$ ./python -c "import multiprocessing; queue_instance = multiprocessing.Queue(); queue_instance.put(('install'))"

3. Analysis
1) The multiprocessing invokes named semaphore in C library
 (sem_open/sem_post/sem_getvalue/sem_unlink in
 ./Modules/_multiprocessing/semaphore.c)
2) The glibc defines two different sem_getvalue since the following commit.
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/sem_getvalue.c;h=1432cc795ece84d5bf31c7e5cafe01cc1a09d98d;hb=042e1521c794a945edc43b5bfa7e69ad70420524
The `__new_sem_getvalue' is the sem_getvalue@@GLIBC_2.1
which work with named semaphore `sem_open'
The `__old_sem_getvalue' is the old version sem_getvalue@GLIBC_2.0
which work with unamed semaphore `sem_init'
In 32-bit C library, it provides both of them.
$ nm -g /lib/i386-linux-gnu/libpthread-2.23.so
0000df30 T sem_getvalue@GLIBC_2.0
0000df10 T sem_getvalue@@GLIBC_2.1
3) In multiprocessing, it invokes sem_open, so sem_getvalue@@GLIBC_2.1
is supposed.
If `-pthread' or `-lpthread' not passed to gcc, the compiled
_multiprocessing dynamic library could not explicitly linked to
sem_getvalue@@GLIBC_2.1
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
 U sem_getvalue
 U sem_open
If `-pthread' or `-lpthread' passed to gcc:
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
 U sem_getvalue@@GLIBC_2.1
 U sem_open@@GLIBC_2.1.1
4) On 32-bit OS, the multiprocessing was incorrectly linked to
sem_getvalue@GLIBC_2.0 which caused the issue.

4. Solution
For cross compiling, there is no `-pthread', so we should explicitly add
`-lpthread' to build multiprocessing.
Peterson tried to do it in the following commit:
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson <benjamin@python.org>
Date: Wed Jun 11 16:44:04 2008 +0000
 Merged revisions 64104,64117 via svnmerge from
 svn+ssh://pythondev@svn.python.org/python/trunk
...
git show e711cafab13efc9c1fe6c5cd75826401445eb585 -- setup.py
--- a/setup.py
+++ b/setup.py
@@ -1110,6 +1110,56 @@ class PyBuildExt(build_ext):
 
 # _fileio -- supposedly cross platform
 exts.append(Extension('_fileio', ['_fileio.c']))
+ # Richard Oudkerk's multiprocessing module
+ if platform == 'win32': # Windows
+ macros = dict()
+ libraries = ['ws2_32']
+
+ elif platform == 'darwin': # Mac OSX
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=0,
+ HAVE_FD_TRANSFER=1,
+ HAVE_BROKEN_SEM_GETVALUE=1
+ )
+ libraries = []
+
+ elif platform == 'cygwin': # Cygwin
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=1,
+ HAVE_FD_TRANSFER=0,
+ HAVE_BROKEN_SEM_UNLINK=1
+ )
+ libraries = []
+ else: # Linux and other unices
+ macros = dict(
+ HAVE_SEM_OPEN=1,
+ HAVE_SEM_TIMEDWAIT=1,
+ HAVE_FD_TRANSFER=1
+ )
+ libraries = ['rt']
+
+ if platform == 'win32':
+ multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/semaphore.c',
+ '_multiprocessing/pipe_connection.c',
+ '_multiprocessing/socket_connection.c',
+ '_multiprocessing/win32_functions.c'
+ ]
+
+ else:
+ multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+ '_multiprocessing/socket_connection.c'
+ ]
+
+ if macros.get('HAVE_SEM_OPEN', False):
+ multiprocessing_srcs.append('_multiprocessing/semaphore.c')
+
+ exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+ define_macros=list(macros.items()),
+ include_dirs=["Modules/_multiprocessing"]))
+ # End multiprocessing
It defined variable `libraries' and assigned it according to
host_platform, but forgot to pass it to Extension.
So we should correct it, and add `-lpthread' for linux.

I've just hit this issue using Python-2.7.9, gcc-8.1.0, glibc-2.23.
The patch I made to fix the issue based on comments in this issue:
--- Python-2.7.9/setup.py 2019年01月25日 09:30:39.049501423 +0000
+++ Python-2.7.9/setup.py 2019年01月25日 09:30:55.369609646 +0000
@@ -1670,7 +1670,7 @@
 else: # Linux and other unices
 macros = dict()
- libraries = ['rt']
+ libraries = ['rt', 'pthread']
 if host_platform == 'win32':
 multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
@@ -1690,6 +1690,7 @@
 if sysconfig.get_config_var('WITH_THREAD'):
 exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+ libraries = libraries,
 define_macros=macros.items(),
 include_dirs=["Modules/_multiprocessing"]))
 else:

The behaviour I saw (32-bit only) was a python process getting stuck.
I got this from strace:
...
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
...
And this from gdb:
(gdb) bt
#0 0xb76fbc5d in __kernel_vsyscall ()
#1 0xb74af2a4 in sem_wait () from /lib/libpthread.so.0
#2 0xb69a5429 in ?? () from /usr/lib/python2.7/lib-dynload/_multiprocessing.so
#3 0xb75a262e in call_function (oparg=<optimized out>, pp_stack=0xbfb7f038) at Python/ceval.c:4033
#4 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#5 0xb75a2f7d in PyEval_EvalCodeEx (co=<optimized out>, globals=
 {'Queue': <type at remote 0xb69c43dc>, 'atexit': <module at remote 0xb6d369bc>, 'Semaphore': <type at remote 0xb69b978c>, 'Empty': <type at remote 0xb69b902c>, 'Full': <type at remote 0xb69b9204>, 'SimpleQueue': <type at remote 0xb69c478c>, 'assert_spawning': <function at remote 0xb69ba684>, '__all__': ['Queue', 'SimpleQueue', 'JoinableQueue'], '_multiprocessing': <module at remote 0xb6c16944>, '_sentinel': <object at remote 0xb726f4d0>, '__package__': 'multiprocessing', 'collections': <module at remote 0xb71bce3c>, '__doc__': None, 'Condition': <type at remote 0xb69c402c>, 'JoinableQueue': <type at remote 0xb69c45b4>, '__builtins__': {'bytearray': <type at remote 0xb76c27e0>, 'IndexError': <type at remote 0xb76c6900>, 'all': <built-in function all>, 'help': <_Helper at remote 0xb71f824c>, 'vars': <built-in function vars>, 'SyntaxError': <type at remote 0xb76c6c80>, 'unicode': <type at remote 0xb76d6c00>, 'UnicodeDecodeError': <type at remote 0xb76c6420>, 'memoryview': <type at remote 0xb76cdda0>, 'isinstance...(truncated), locals=0x0, args=0xb535f5cc, argcount=2, kws=0xb535f5d4, kwcount=0, defs=0xb69b6058, defcount=2, closure=0x0) at Python/ceval.c:3265
#6 0xb75a0f3d in fast_function (nk=0, na=<optimized out>, n=2, pp_stack=0xbfb7f178, func=<function at remote 0xb69c51b4>) at Python/ceval.c:4129
#7 call_function (oparg=<optimized out>, pp_stack=0xbfb7f178) at Python/ceval.c:4054
#8 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
...
(gdb) py-bt
#4 Frame 0xb536602c, for file /usr/lib/python2.7/multiprocessing/queues.py, line 101, in put (self=<Queue(_writer=<_multiprocessing.Connection at remote 0x96878c8>, _recv=<built-in method recv of _multiprocessing.Connection object at remote 0x967ddc8>, _thread=None, _sem=<BoundedSemaphore(release=<built-in method release of _multiprocessing.SemLock object at remote 0xb53427c0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0xb53427c0>, _semlock=<_multiprocessing.SemLock at remote 0xb53427c0>) at remote 0xb53425cc>, _buffer=<collections.deque at remote 0xb533f844>, _closed=False, _send=<built-in method send of _multiprocessing.Connection object at remote 0x96878c8>, _jointhread=None, _reader=None, _opid=3752, _rlock=<Lock(release=<built-in method release of _multiprocessing.SemLock object at remote 0xb5342420>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0xb5342420>, _semlock=<_multiprocessing.SemLock at remote 0xb5342420>) at remote 0xb534240c>, _...(truncated)
...