homepage

A multiprocessing Queue object managers multiple resources:
* a multiprocessing Pipe
* a thread
* (a lock and a semaphore)
If a Queue is not cleaned up properly, your application may leak many resources.
Try attached queue_leak.py to see an example "leaking a thread".
I suggest to emit a ResourceWarning warning in Queue destrutor. I don't know what should be the test to decide if a warning must be emitted?
* if the queue wasn't closed yet?
* if the thread is alive?
* if the queue wasn't closed yet and/or the thread is alive? (my favorite choice)
Other examples of objects emitting ResourceWarning:
* io files: io.FileIO, io.TextIOWrapper, etc.
* socket.socket
* subprocess.Popen: I recently added a ResourceWarning on that one
* asyncio transports and event loops

Example:
haypo@selma$ ./python queue_leak.py 
number of thread diff: +1
dangling threads!
before: [<_MainThread(MainThread, started 139814961067072)>]
after: [<_MainThread(MainThread, started 139814961067072)>]
Note: queue_leak.py resource check is based on test.support.reap_threads.

Oh, I forgot that I hitted this issue while analyzing issue #30131: test_logging leaks a "dangling" thread. It took me a while to find multiprocessing queues in the big test_logging file!

See also issue #30244: Emit a ResourceWarning in concurrent.futures executor destructors.

The thread seems to be stopped when the Queue object is finalized:
 # Send sentinel to the thread queue object when garbage collected
 self._close = Finalize(
 self, Queue._finalize_close,
 [self._buffer, self._notempty],
 exitpriority=10
 )
I don't think the other resources (pipe, lock, semaphore) need explicit cleaning.

Another example:
http://bugs.python.org/issue30886#msg298014
"The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess."

> The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess.
I don't understand what this means. Can you clarify a bit?

Specifically "the thread wasn't started by a subprocess"...

> Specifically "the thread wasn't started by a subprocess"...
I'm talking about this check in Queue._start_thread() of multiprocessing.queues:
 created_by_this_process = (self._opid == os.getpid())
 if not self._joincancelled and not created_by_this_process:
 self._jointhread = Finalize(
 self._thread, Queue._finalize_join,
 [weakref.ref(self._thread)],
 exitpriority=-5
 )

Let's discuss created_by_this_process in bpo-30886.
This issue is more about adding or not a ResourceWarning.

I don't think a ResourceWarning should be emitted. There is no risk of data loss or resource leak if you don't close a multiprocessing Queue explicitly. Actually, in the real world, I don't think I've ever seen code that closes queues explicitly.

I'm willing to close this issue, as I don't think a ResourceWarning is appropriate here.