homepage

There is no Barrier in multiprocessing corresponding to threading.Barrier.

Here is an initial implementation. Differences from threading.Barrier:
- I have not implemented reset().
- wait() returns 0 or -1. One thread returns 0, the remainder return -1.
 This is different to threading.Barrier where each of the N threads 
 returns a unique index in range(N).
- I added an "action_args" parameter to the constructor. This is a tuple 
 which provides argument to the "action" callback. (This is because closures
 are not picklable, making no-argument callbacks rather limiting for 
 multiprocessing.)

barrier_tests.py contains minor modifications of the unit tests for
threading.Barrier. (The two tests using reset() are commented out.)
The implementation passes for me on Linux and Windows.

> Here is an initial implementation.
Wouldn't it be simpler with a mp.Condition?
Otherwise, this should be added to Lib/multiprocesing.synchronize.py, and the tests to test_multiprocessing.

> Wouldn't it be simpler with a mp.Condition?
Well, it is a fair bit shorter than the implementation in threading.py. But that is not a fair comparison because it does implement reset().
I was trying to avoid using shared memory/ctypes since multiprocessing.synchronize does not currently use them. However, I think it would be better (and much simpler) to just subclass threading.Barrier, making self._state and self._counter properties which delegate to RawValue objects. That gets rid of the differences in behaviour. I have this working, although I had to monkey patch multiprocessing.Condition to add a wait_for() method. See Issue 14087.
> Otherwise, this should be added to Lib/multiprocesing.synchronize.py,
> and the tests to test_multiprocessing.
I will provide a patch later.

Patch which subclasses threading.Barrier.

Forgot to mention, mp_barrier.patch needs to be applied on top of cond_wait_for.patch for Issue14087.

The patch looks good.
However, I've had two failures while testing it:
- a BrokenBarrierError on test_default_timeout: I see you've already increased the timeout from the original threading code, but you can probably double it (we have some slow buildbots, and creating processes is expensive on Windows)
- TestNumberOfObjects is failing too:
"""
test_number_of_objects (test.test_multiprocessing.WithManagerTestZZZNumberOfObjects) ... b6cb776c: refcount=1
 <threading.Barrier object at 0xb6cb776c>
 b6d72dfc: refcount=2
 <multiprocessing.pool.Pool object at 0xb6d72dfc>
 b6cb776c: refcount=1
 <threading.Barrier object at 0xb6cb776c>
 b6d72dfc: refcount=2
 <multiprocessing.pool.Pool object at 0xb6d72dfc>
FAIL
"""

RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes.

> RawValue uses ctypes, right? That's problematic for platforms which don't 
> support ctypes.
Are there many posix systems (we care about) where ctypes doesn't work?
It would be fairly easy to use memoryview instead of ctypes. (In fact Value/RawValue could be implemented that way when using a typecode instead of a ctypes type.)

> > RawValue uses ctypes, right? That's problematic for platforms which don't 
> > support ctypes.
> 
> Are there many posix systems (we care about) where ctypes doesn't
> work?
It depends what you call "caring about" :-)
But proprietary Unix C compilers generally don't mix very well with
ctypes.
> It would be fairly easy to use memoryview instead of ctypes. (In fact
> Value/RawValue could be implemented that way when using a typecode
> instead of a ctypes type.)
That would be a good idea then. Also, not using ctypes could make code
theoretically more robust.

The attached patch uses memoryview instead of ctypes.
If the patch for Issue #14953 (reimplementing RawArray/RawValue in terms of memoryview) is applied, then it could be simplified a bit.

New changeset 2d2f206d040e by Richard Oudkerk in branch 'default':
Issue #14059: Implement multiprocessing.Barrier
http://hg.python.org/cpython/rev/2d2f206d040e