homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Queue is empty right after put from the same process/thread
Type: Stage: resolved
Components: Documentation Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, python-dev, sbt, shwouchk
Priority: normal Keywords:

Created on 2013年06月21日 11:30 by shwouchk, last changed 2022年04月11日 14:57 by admin. This issue is now closed.

Messages (10)
msg191566 - (view) Author: (shwouchk) Date: 2013年06月21日 11:30
Consider this:
$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing as mp
>>> q = mp.Queue()
>>> while True:
 q.put(1)
 q.get_nowait()
Traceback (most recent call last):
 File "<stdin>", line 3, in <module>
 File "/usr/lib/python2.7/multiprocessing/queues.py", line 152, in get_nowait
 return self.get(False)
 File "/usr/lib/python2.7/multiprocessing/queues.py", line 134, in get
 raise Empty
Queue.Empty
I believe that similar behavior could be seen in cPython 2.7.3 with the Queue.Queue implementation, but I can't reproduce it now and don't have the old version to test. And it is irrelevant anyway since it work "correctly" now.
I think this behavior is counter intuitive and hampers the development of code that performs stuff with queues in a generic way and works in both single and multi-process environments.
msg191609 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013年06月21日 20:26
This is a very similar issue to #17985.
While it may seem counter-intuitive, I don't see how it makes any difference. Another thread/process might remove the item before you can get it.
I find it very difficult to imagine a real program where you can safely use get_nowait() without being prepared to handle an Empty exception.
msg191613 - (view) Author: (shwouchk) Date: 2013年06月21日 22:12
The major difference with the issue you referenced is that the behavior in #17985 is clearly stated and warned against in the docs (Queue.Empty being inconsistent with Queue.size), whereas this is not.
As for the actual behavior problem: Imagine you build an abstraction atop Queue (For example and specifically, a queue iterator). This iterator might return None or a similar sentinel if there is nothing on the queue.
Since you find the construct useful and would like to keep uniformity in the program, there is a place you use this abstraction in a fairly tight loop to pass messages from one part of the program to another part, in the same process and thread. Now the logic of the program does not work correctly as it was based on the assumption that in a single process Queue would work "as expected".
The problem is not the exception but with having an abstraction built atop Queue that works right with respect to program logic both when you use it in one thread and in multiple ones.
msg191620 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013年06月21日 22:47
Why would you use a multi-process queue to "pass messages from one part of the program to another part, in the same process and thread"? Why not just use a deque?
Is this something you actually did, or are you just trying to come up with a plausible example?
And, of course, if you are sure there must be an item available, you could just use get() instead of get_nowait().
msg191621 - (view) Author: (shwouchk) Date: 2013年06月21日 23:04
Richard,
I think you missed my point. First, yes I did do that.
Second ("the point"):
I did this to use the same abstraction that was used extensively for other purposes, instead of recreating the same abstraction with a deque as its basis. Component reusability is one of the main points of OOP, after all...
And no, an item is not necessarily available - sometimes there is a message and sometimes there isn't. But if one was put into the queue, I claim that I should be able to rely on it being available right away in the application logic.
msg191622 - (view) Author: (shwouchk) Date: 2013年06月21日 23:05
Also, of course I did this or I would not have stumbled into this issue...
msg191762 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013年06月24日 13:53
New changeset 8dcc4e017d42 by Richard Oudkerk in branch '2.7':
Issue #18277: Document quirks of multiprocessing queue.
http://hg.python.org/cpython/rev/8dcc4e017d42
New changeset 0f921e73433a by Richard Oudkerk in branch '3.3':
Issue #18277: Document quirks of multiprocessing queue.
http://hg.python.org/cpython/rev/0f921e73433a
New changeset 06b1447becdc by Richard Oudkerk in branch 'default':
Issue #18277: Merge.
http://hg.python.org/cpython/rev/06b1447becdc 
msg191764 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013年06月24日 14:00
> I did this to use the same abstraction that was used extensively for 
> other purposes, instead of recreating the same abstraction with a deque 
> as its basis. 
So you wanted a FIFO queue and preferred the API of Queue to that of deque? Well it will be *much* less efficient. queue.Queue is also less efficient, but not by such a wide margin.
I have added a documentation note.
msg191783 - (view) Author: (shwouchk) Date: 2013年06月24日 16:22
I agree it might be less efficient, but sometimes it is the price to pay for greater generality/simplicity. After all, If I *really* wanted efficiency perhaps I would have written everything in C++.
Anyway, thanks!
n.p.
1. "but should not cause any pratical difficulties" <-- you have a typo in 'pratical' there.
2. What exactly do you mean by "managed" queues in the new addition?
Also, did part #2 of the note come up in other reports? It seemed somewhat trivial (can't hurt though)...
Finally, I don't know whether I'm supposed to close the issue, but its a good solution as far as I'm concerned.
Thanks again!
msg191790 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013年06月24日 17:19
> 1. "but should not cause any pratical difficulties" <-- you have a typo in 'pratical' there.
> 2. What exactly do you mean by "managed" queues in the new addition?
Woops. Fixed now see 860fc6a2bd21, 347647a1f798. A managed queue is 
one created like
 manager = multiprocessing.Manager()
 queue = manager.Queue()
queue is a proxy for a conventional queue object in a "manager" process.
> Also, did part #2 of the note come up in other reports?
> It seemed somewhat trivial (can't hurt though)...
Yes it did (but I can't find the report now).
History
Date User Action Args
2022年04月11日 14:57:47adminsetgithub: 62477
2013年06月24日 17:20:36sbtsetstatus: open -> closed
assignee: docs@python
components: + Documentation, - Interpreter Core, IO

nosy: + docs@python
type: behavior ->
resolution: fixed
stage: resolved
2013年06月24日 17:19:03sbtsetmessages: + msg191790
2013年06月24日 16:22:42shwouchksetmessages: + msg191783
2013年06月24日 14:00:01sbtsetmessages: + msg191764
2013年06月24日 13:53:26python-devsetnosy: + python-dev
messages: + msg191762
2013年06月21日 23:05:32shwouchksetmessages: + msg191622
2013年06月21日 23:04:26shwouchksetmessages: + msg191621
2013年06月21日 22:47:25sbtsetmessages: + msg191620
2013年06月21日 22:12:42shwouchksetmessages: + msg191613
2013年06月21日 20:26:40sbtsetmessages: + msg191609
2013年06月21日 18:29:53ned.deilysetnosy: + sbt
2013年06月21日 11:30:16shwouchkcreate

AltStyle によって変換されたページ (->オリジナル) /