homepage

multiprocessing.queues.SimpleQueue is undocumented and doesn't appear in multiprocessing.__all__.

Here's a patch to add SimpleQueue to doc and __all__ .
I didn't document the 'sentinels' argument of SimpleQueue.get() because I got lost into Pipe _pool and can't understand how those sentinels are actually used/useful. Any hint is appreciated :)

Sandro - yep, the sentinels arg is also undocumented in multiprocessing.PipeConnection.recv() and further down the road...

However, sentinels *are* mentioned in the multiprocessing doc, below multiprocessing.Process:
"
sentinel
 A numeric handle of a system object which will become "ready" when the process ends.
 On Windows, this is an OS handle usable with the WaitForSingleObject and WaitForMultipleObjects family of API calls. On Unix, this is a file descriptor usable with primitives from the select module.
 You can use this value if you want to wait on several events at once. Otherwise calling join() is simpler.
 New in version 3.3.
"
From a cursory glance on the implementation in Lib/multiprocessing/connection.py, this is the same sentinel.

Well, the sentinels argument, right now, is meant to be used internally. I don't think it's a good thing to document it, since I don't think it's a very clean API (I know, I introduced it :-)) - it's just so that concurrent.futures can detect dead processes.

> Well, the sentinels argument, right now, is meant to be used 
> internally. I don't think it's a good thing to document it, 
> since I don't think it's a very clean API (I know, I introduced 
> it :-))
Wouldn't a better alternative be to have a wait function which can deal with readable pipe connections and integer handles?
On Unix this would just delegate to select().
On Windows it could work as follows:
* initiate an overlapped read on each connection
* call WaitForMultipleObjects()
* cancel each overlapped read
* continue any read which succeeded but only gave a partial message
* store read messages on associated connection objects
I did start on such a patch. It worked, but I did not get round to writing tests for it...

> Wouldn't a better alternative be to have a wait function which can
> deal with readable pipe connections and integer handles?
> 
> On Unix this would just delegate to select().
> 
> On Windows it could work as follows:
> * initiate an overlapped read on each connection
> * call WaitForMultipleObjects()
> * cancel each overlapped read
> * continue any read which succeeded but only gave a partial message
> * store read messages on associated connection objects
This sounds like a good direction to me.
(of course it must also preserve existing functionality;
test_concurrent_futures will check for that)

Then it appears to me that Sandro's patch is good and can be committed.

> Then it appears to me that Sandro's patch is good and can be committed.
The doc patch is good. However, if you start exposing SimpleQueue at the
top package level, you have to do it in 3.3 only (since that's a new
API), and also mention it somehow in the doc.

Sandro, can you commit, taking Antoine's note into account?

Thanks Eli for the heads-up, I had missed Antoine's comment!
Antoine, I'm probably missing something, but SimpleQueue is present in 2.7 and 3.2 too, so why not mention it in the doc for those versions too?

> Antoine, I'm probably missing something, but SimpleQueue is present in 
> 2.7 and 3.2 too, so why not mention it in the doc for those versions too?
What I mean is that "multiprocessing.SimpleQueue" is a new API, so should be 3.3-only (while "multiprocessing.queues.SimpleQueue" already exists).

It's the way all the subclasses are imported into the main module that got me in fault, I think. OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 and multiprocessing.SimpleQueue in 3.3 also adding the hunk to __init__ (thus changing the API).
I'd say to document int still near to Queue() and JoinableQueue() - do you agree?

> It's the way all the subclasses are imported into the main module that
> got me in fault, I think. OK, so if I got it correctly, I should
> document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 and
> multiprocessing.SimpleQueue in 3.3 also adding the hunk to __init__
> (thus changing the API).
Yes.
> I'd say to document int still near to Queue() and JoinableQueue() - do
> you agree?
Yes, too.

> OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 [...]
s/3.1/3.2/

>> OK, so if I got it correctly, I should document multiprocessing.queue.SimpleQueue in 2.7 and 3.1 [...]
>
> s/3.1/3.2/
yeah, just a typo :)

New changeset 3b127a415643 by Sandro Tosi in branch '2.7':
Issue #11836: document multiprocessing.queues.SimpleQueue
http://hg.python.org/cpython/rev/3b127a415643
New changeset fe5eb6d35025 by Sandro Tosi in branch '3.2':
Issue #11836: document multiprocessing.queues.SimpleQueue
http://hg.python.org/cpython/rev/fe5eb6d35025
New changeset bf536b46d7f2 by Sandro Tosi in branch 'default':
Issue #11836: document and expose multiprocessing.SimpleQueue
http://hg.python.org/cpython/rev/bf536b46d7f2 

Thanks for all you inputs!