homepage

The behaviour of multiprocessing.Queue surprised me today in that Queue.get() may raise an exception even if an item is immediately available. I tried to flush entries without blocking by using the timeout=0 keyword argument:
$ /opt/python3/bin/python3
Python 3.4.0b1 (default:247f12fecf2b, Jan 6 2014, 14:50:23) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/opt/python3/lib/python3.4/multiprocessing/queues.py", line 107, in get
 raise Empty
queue.Empty
Actually even passing a small non-zero timeout will not give me my queue entry:
>>> q.get(timeout=1e-6)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/torsten/opensrc/cpython/Lib/multiprocessing/queues.py", line 107, in get
 raise Empty
queue.Empty
Expected behaviour for me would be to return the item that is in the queue. I know that there is a kwarg *block* which gives me the desired behaviour:
>>> q.get(block=False)
'hi'
In my case the get call is embedded in my own module which does not currently expose the block parameter. My local solution is of course to update the wrapper:
if timeout == 0:
 timeout = None
 block = False
However I see a few smells here in the python standard library. First, everything else seems to accept timeout=0 as nonblocking:
>>> import threading
>>> lock = threading.Lock()
>>> lock.acquire(timeout=0)
True
>>> from queue import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
'hi'
Of special note is that queue.Queue behaves as I would have expected. IMHO it should be consistent with multiprocessing.Queue.
Also note that queue.Queue.get() and queue.Queue.put() name their blocking flag "block", while everybody else uses "blocking".
As a side note, I think the current approach is flawed in computing the deadline. Basically it does the following:
 deadline = time.time() + timeout
 if not self._rlock.acquire(block, timeout):
 raise Empty
 timeout = deadline - time.time()
 if timeout < 0 or not self._poll(timeout):
 raise Empty
On my system, just taking the time twice and computing the delta takes 2 microseconds:
>>> import time
>>> t0 = time.time(); time.time() - t0
2.384185791015625e-06
Therefore calling Queue.get(block, timeout) with 0 < timeout < 2e-6 will never return anything from the queue even though Queue.get(block=False) would do that. This contradicts the idea that Queue.get(block=False) will return faster than with block=True with any timeout > 0.
Apart from that, as Python does not currently support waiting on multiple sources, we currently often check a queue with a small timeout concurrently with doing other stuff. In case the system get really loaded, I would expect this to cause problems because the updated timeout may fall below zero.
Suggested patch attached.