homepage

multiprocessing.queues.SimpleQueue should have a close() method. This is needed to explicitly release the two file descriptors of the Pipe used internally. Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy).
I think the following would do:
diff -r 0b72fd1a7641 lib-python/2.7/multiprocessing/queues.py
--- a/lib-python/2.7/multiprocessing/queues.py Sun Jul 16 13:41:28 2017 +0200
+++ b/lib-python/2.7/multiprocessing/queues.py Wed Jul 19 10:45:03 2017 +0200
@@ -358,6 +358,11 @@
 self._wlock = Lock()
 self._make_methods()
+ def close(self):
+ # PyPy extension: CPython doesn't have this method!
+ self._reader.close()
+ self._writer.close()
+
 def empty(self):
 return not self._reader.poll()

Get this solved then we could also solve #23267, which reports pipes leak in pool due to using SimpleQueue.

On Python 3.6+, regrtest is able to detect file descriptor leaks when using --huntrleaks. Is someone interested to backport this feature to 3.5 and/or 2.7 which would mean fix all FD leaks in our test suite?
If someone wants to work on that, I would suggest to first fix all bugs, and when the test suite is fine: modify regrtest.

See also my bpo-30171: "Emit ResourceWarning in multiprocessing Queue destructor".

This issue is marked as also affecting Python 3.7. I don't see a SimpleQueue.close() method in master, but I don't remind any resource warning when running tests on master neither.
Does it mean that our test runner miss such file descriptor leak? Or worse, that SimpleQueue is not tested? I see a TestSimpleQueue test case in Lib/test/_test_multiprocessing.py.
haypo@selma$ ./python -m test test_multiprocessing_spawn -m TestSimpleQueue -R 3:3
(...)
Tests result: SUCCESS
> This is needed to explicitly release the two file descriptors of the Pipe used internally. Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy).
Oh ok, the garbage collector is able to close the file descriptors. The bug is when a SimpleQueue is not collected.
So again, I consider that a ResourceWarning is needed here. The purpose of a ResourceWarning is to warn when a leak may be kept alive longer than expected if it's not closed explicitly.

Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly.

> Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly.
I started with SimpleQueue for the master branch:
https://github.com/python/cpython/pull/2760 

bpo-23267 is marked as a duplicate of this issue.

New changeset 9adccc1384568f4d46e37f698cb3e3a4f6ca0252 by Victor Stinner in branch 'master':
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
https://github.com/python/cpython/commit/9adccc1384568f4d46e37f698cb3e3a4f6ca0252

New changeset 1a275013d1ecc2e3778d64fda86174b2f13d6969 by Victor Stinner in branch 'master':
bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)
https://github.com/python/cpython/commit/1a275013d1ecc2e3778d64fda86174b2f13d6969

This seems complete, can it be closed?