homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing: use SysV semaphores on FreeBSD
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: asksol, db3l, jnoller, neologix, pitrou, vstinner
Priority: normal Keywords: buildbot

Created on 2010年11月08日 01:57 by vstinner, last changed 2022年04月11日 14:57 by admin. This issue is now closed.

Messages (15)
msg120705 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年11月08日 01:57
Support POSIX semaphore on FreeBSD is recent, optional (eg. disabled by default in FreeBSD 7) and limited (30 semaphores). SYSV should be used instead because they are less limited or more adjustable (at runtime: POSIX semaphore requires to recompile the kernel!).
This issue should fix test_concurrent_futures on FreeBSD 7.2 and 8.0: many tests use more than 30 semaphores. The maximum is test_all_completed_some_already_completed: 52 semaphores.
msg120709 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年11月08日 02:35
It looks like SysV semaphores are also preferred on Darwin. So I suppose that Mac OS X would also benefit of this issue. Maybe also other OSes (Solaris?).
See also issue #7272.
msg124795 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年12月28日 13:51
Informations about SysV semaphores:
 - functions: semget(), semop(), semctl(), ftok()
 - http://perldoc.perl.org/IPC/SysV.html
 - http://beej.us/guide/bgipc/output/html/multipage/semaphores.html 
msg124796 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年12月28日 14:01
Examples of programs using SysV semaphores:
http://firebird.cvs.sourceforge.net/viewvc/firebird/firebird2/src/jrd/isc_sync.cpp?revision=HEAD&view=markup
(Firebird, search "#ifdef USE_SYS5SEMAPHORE" sections)
https://github.com/mono/mono/blob/master/mono/io-layer/shared.c#L501
(Mono)
msg124831 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年12月29日 00:14
See also #5725.
msg124838 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010年12月29日 01:24
See also http://semanchuk.com/philip/sysv_ipc/: "System V IPC for Python - Semaphores, Shared Memory and Message Queues"
msg124862 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010年12月29日 17:47
Adding, or moving, to SYSV semaphores is very low on the list of things to do. If someone were to provide a patch, I'm sure we could consider it.
msg125042 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011年01月02日 12:12
More info about FreeBSD.
"sysctl p1003_1b.sem_nsems_max" gives the maximum number of POSIX semaphores (per process? system wide?).
Since FreeBSD 8.1, "sudo sysctl -w p1003_1b.sem_nsems_max=256" can be used to change this limit at runtime.
Before FreeBSD 8.1, SEM_MAX constant should be changed in the kernel source code, and the kernel have to be recompiled. (p1003_1b.sem_nsems_max is not configurable in /etc/sysctl.conf, it is an hardcoded limit).
Before FreeBSD 8.0, the POSIX semaphores are disabled by default: the kernel have to be compiled using P1003_1B_SEMAPHORES option. Extract of sys/conf/NOTES:
#####################################################################
# POSIX P1003.1B
# Real time extensions added in the 1993 POSIX
# _KPOSIX_PRIORITY_SCHEDULING: Build in _POSIX_PRIORITY_SCHEDULING
options 	_KPOSIX_PRIORITY_SCHEDULING
# p1003_1b_semaphores are very experimental,
# user should be ready to assist in debugging if problems arise.
options 	P1003_1B_SEMAPHORES
# POSIX message queue
options 	P1003_1B_MQUEUE

#####################################################################
msg125043 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011年01月02日 12:25
NetBSD.
Extract of the sem_close() manpage
http://www.daemon-systems.org/man/sem_close.3.html
---
STANDARDS
 The sem_open(), sem_close(), and sem_unlink() functions conform to
 ISO/IEC 9945-1:1996 (``POSIX.1'').
HISTORY
 Support for named semaphores first appeared in NetBSD 2.0.
---
Martin wrote on the mailing list:
---
According to
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/uipc_sem.c?rev=1.22&content-type=text/x-cvsweb-markup&only_with_tag=MAIN
SEM_MAX is 128 since 2007, and dynamically adjustable (no reboot).
---
It looks like the sysctl (read/write) option is kern.posix.semmax.
msg125045 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011年01月02日 12:28
Darwin (Mac OS X).
According to the following email (July 2010), Darwin supports POSIX semaphores and the default limit is 10,000 semaphores.
http://osdir.com/ml/darwin-dev/2010-07/msg00012.html
The limit is configurable via sysctl as kern.posix.sem.max.
msg125046 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011年01月02日 12:30
OpenBSD.
According to Martin, OpenBSD doesn't implement POSIX semaphores.
---
I don't have an installation of OpenBSD, but...
In FreeBSD, POSIX semaphores are implemented in sys/kern/uipc_sem.c.
In
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/kern/
that file doesn't exist. Also, in FreeBSD's limits.h,
_POSIX_SEM_NSEMS_MAX is defined (surprisingly to 256);
in
http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/include/limits.h?rev=1.15;content-type=text/plain
this constant doesn't appear. So ISTM that OpenBSD doesn't implement
POSIX semaphores. IIUC, this means that the multiprocessing module
won't be fully functional, and its tests (and the concurrent.futures
tests) will be skipped.
---
msg125160 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011年01月03日 11:42
Martin fixed test_concurrent_futures (#10798), this issue can be implemented later.
msg144908 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011年10月04日 17:30
-1
IMHO, implementing SysV semaphores would be a step backwards, plus the API is a real pain.
I think there's no reason to complicate the code to accomodate such corner cases, especially since the systems that don't support POSIX semaphores will eventually die out...
msg144909 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011年10月04日 17:31
Agreed with Charles-François.
msg144931 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2011年10月04日 23:26
Charles and Antoine's votes match my own, therefore closing the bug wont fix
History
Date User Action Args
2022年04月11日 14:57:08adminsetgithub: 54557
2011年10月04日 23:26:38jnollersetstatus: open -> closed
resolution: wont fix
messages: + msg144931
2011年10月04日 17:31:49pitrousetnosy: + pitrou
messages: + msg144909
2011年10月04日 17:30:56neologixsetnosy: + neologix
messages: + msg144908
2011年01月03日 11:42:13vstinnersetnosy: db3l, vstinner, jnoller, asksol
type: enhancement
messages: + msg125160
versions: + Python 3.3, - Python 3.2
2011年01月02日 12:30:34vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg125046
2011年01月02日 12:28:37vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg125045
2011年01月02日 12:25:38vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg125043
2011年01月02日 12:12:36vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg125042
2010年12月29日 17:47:18jnollersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg124862
2010年12月29日 01:24:44vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg124838
2010年12月29日 00:14:46vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg124831
2010年12月28日 14:01:17vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg124796
2010年12月28日 13:51:42vstinnersetnosy: db3l, vstinner, jnoller, asksol
messages: + msg124795
2010年11月08日 15:04:22pitrousetnosy: + asksol
2010年11月08日 02:35:40vstinnersetnosy: + jnoller

messages: + msg120709
title: multiprocessing: use SYSV semaphores on FreeBSD -> multiprocessing: use SysV semaphores on FreeBSD
2010年11月08日 01:57:23vstinnercreate

AltStyle によって変換されたページ (->オリジナル) /