homepage

Starting with commit 468e5fec (bpo-36402: Fix threading._shutdown() race condition (GH-13948)) the following trivial test case leaks one reference and one memory block.
class MiscTestCase(unittest.TestCase):
 def test_without_join(self):
 # Test that a thread without join does not leak references.
 # Use a debug build and run "python -m test -R: test_threading"
 threading.Thread().start()
Attached is a patch, that adds this test case to Lib/test/test_threading.py. After you apply this patch "python -m test -R: test_threading" leaks one (additional) reference. This leak is also present in Python 3.7.4 and 3.8.
I'm not sure, if it correct not to join a thread, but it did work flawlessly and didn't leak in previous releases.
I didn't analyse the root cause yet.

The root cause for the reference leak is the global set threading._shutdown_locks. It contains Thread._tstate_lock locks of non-daemon threads. If a non-daemon thread terminates and no other thread joins the terminated thread, the _tstate_lock remains in threading._shutdown_locks forever.
I could imagine that a long running server could accumulate many locks in threading._shutdown_locks over time. Therefore the leak should be fixed.
There are probably several ways to deal with this issue. A straight forward approach is to discard the lock from within `tstate->on_delete` hook, that is function "void release_sentinel(void *)" in _threadmodule.c. Pull request (GH-15175) implements this idea. Eventually I should add another C-Python specific test-case to the PR.

New changeset d3dcc92778807ae8f7ebe85178f36a29711cd478 by Victor Stinner in branch 'master':
bpo-37788: Fix a reference leak if a thread is not joined (GH-15228)
https://github.com/python/cpython/commit/d3dcc92778807ae8f7ebe85178f36a29711cd478

> https://github.com/python/cpython/commit/d3dcc92778807ae8f7ebe85178f36a29711cd478
This change introduced a regression :-(
https://github.com/python/cpython/pull/15228#issuecomment-522792133
I created PR 15338 to revert it

New changeset d11c2c607768fa549b1aed7899edc061b2ebf19f by Victor Stinner in branch 'master':
Revert "bpo-37788: Fix a reference leak if a thread is not joined (GH-15228)" (GH-15338)
https://github.com/python/cpython/commit/d11c2c607768fa549b1aed7899edc061b2ebf19f

Copy of wmanley's comment:
https://github.com/python/cpython/pull/13948#issuecomment-541076882
"""
This caused a regression for people overriding Thread.join to implement custom thread interruption. See https://stackoverflow.com/questions/50486083/ending-non-daemon-threads-when-shutting-down-an-interactive-python-session
"""

Hummm....is this really a regression? The docs say:
> No other methods (except for the constructor) should be overridden in a subclass. In other words, only override the __init__() and run() methods of this class.
So if someone is overriding join() they are out of contract

I ran into this bug as well, and opened an issue for it (before I saw this issue): https://bugs.python.org/issue39074
Was there a conclusion on the best way to fix this? It seems like the previous __del__ implementation would correct the resource leakage by removing the _tstate_lock from _shutdown_locks.

I marked bpo-39074 as a duplicate of this issue.

I marked bpo-39201 as a duplicate of this issue.

I have reproduced a similar memory leak in the multiprocessing
This is the trace:
913 memory blocks: 33232 Bytes: File "/usr/lib/python3.7/threading.py", line 890; self._bootstrap_inner(); File "/usr/lib/python3.7/threading.py", line 914; self._set_tstate_lock(); File "/usr/lib/python3.7/threading.py", line 904; self._tstate_lock = _set_sentinel();

I marked bpo-42263 as a duplicate of this issue.
This issue is implicated in preventing the desired fix for bpo-37193, where a thread wishes to remove the handle to itself after performing its duty. By removing its own handle, it can never be joined, and thus obviates the most straightforward way to directly remove the handle for a thread within that thread.

bpo-37193 has been fixed.

@Victor: How does the fix for #37193 solve this issue? I think I'm missing something.

Re-opening; I believe this issue is still valid.
Here's output on a debug build on current master (commit 7591d9455eb37525c832da3d65e1a7b3e6dbf613) on my machine:
lovelace:cpython mdickinson$ ./python.exe
Python 3.10.0a6+ (remotes/upstream/master:7591d9455e, Mar 13 2021, 12:04:31) [Clang 11.0.0 (clang-1100033.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, threading, time
>>> for _ in range(10):
... threading.Thread().start()
... time.sleep(0.1)
... print(sys.gettotalrefcount())
... 
109901
109905
109907
109909
109911
109913
109915
109917
109919
109921

Oh, I'm confused. This issue is about the threading module, whereas bpo-37193 is unrelated: it's about the socketserver module.

ping
I also encountered this problem. Is there a fix now?

No. The issue remains open.

Anselm's pull request PR 15175 looked hopeful to me. I've been using those changes in our builds at work for a while.

I created PR 25226. It's a another way to solve this problem.
Compared to PR 15175, `_set_sentinel()` don't need to receive any params :)

New changeset 71dca6ea73aaf215fafa094512e8c748248c16b0 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-37788: Fix reference leak when Thread is never joined (GH-26103) (GH-26138)
https://github.com/python/cpython/commit/71dca6ea73aaf215fafa094512e8c748248c16b0

New changeset b30b25b26663fb6070b8ed86fe3a20dcb557d05d by Antoine Pitrou in branch '3.9':
[3.9] bpo-37788: Fix reference leak when Thread is never joined (GH-26103) (GH-26142)
https://github.com/python/cpython/commit/b30b25b26663fb6070b8ed86fe3a20dcb557d05d

Fix in the main branch:
commit c10c2ec7a0e06975e8010c56c9c3270f8ea322ec
Author: Antoine Pitrou <antoine@python.org>
Date: Fri May 14 21:37:20 2021 +0200
 bpo-37788: Fix reference leak when Thread is never joined (GH-26103)
 
 
 
 When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.