[Python-ideas] reducing multiprocessing.Queue contention

Wed Jan 23 12:16:14 CET 2013

Hello,
Currently, multiprocessing.Queue put() and get() methods hold locks
for the entire duration of the writing/reading to the backing
Connection (which can be a pipe, unix domain socket, or whatever it's
called on Windows).
For example, here's what the feeder thread does:
"""
 else:
 wacquire()
 try:
 send(obj)
 # Delete references to object. See issue16284
 del obj
 finally:
 wrelease()
"""
Connection.send() and Connection.recv() have to serialize the data
using pickle before writing them to the underlying file descriptor.
While the locking is necessary to guarantee atomic read/write (well,
it's not necessary if you're writing to a pipe less than PIPE_BUF, and
writes seem atomic on Windows), the locks don't have to be held while
the data is serialized.
Although I didn't make any measurement, my gut feeling is that this
serializing can take a non negligible part of the overall
sending/receiving time, for large data items. If that's the case, then
simply holding the lock for the duration of the read()/write() syscall
(and not during serialization) could reduce contention in case of
large data sending/receiving.
One way to do that would be to refactor the code a bit to provide
maybe a (private) AtomicConnection, which would encapsulate the
necessary locking: another advantage is that this would hide the
platform-dependent code inside Connection (right now, Queue only uses
a lock for ending on Unix platforms, since write is apparently atomic
on Windows).
Thoughts?