homepage

When running 3.7, we noticed a memory leak in threading._shutdown_locks when non-deamon threads are started but "join()" or "is_alive()" is never called. Here's a test to illustrate the growth:
=========
import threading
import time
import tracemalloc
def test_leaking_locks():
 tracemalloc.start(10)
 snap1 = tracemalloc.take_snapshot()
 def print_things():
 print('.', end='')
 for x in range(500):
 t = threading.Thread(target=print_things)
 t.start()
 time.sleep(5)
 print('')
 gc.collect()
 snap2 = tracemalloc.take_snapshot()
 filters = []
 for stat in snap2.filter_traces(filters).compare_to(snap1.filter_traces(filters), 'traceback')[:10]:
 print("New Bytes: {}\tTotal Bytes {}\tNew blocks: {}\tTotal blocks: {}: ".format(stat.size_diff, stat.size, stat.count_diff ,stat.count))
 for line in stat.traceback.format():
 print(line)
=========
=========
Output in v3.6.8:
New Bytes: 840	Total Bytes 840	New blocks: 1	Total blocks: 1: 
 File "/usr/local/lib/python3.6/threading.py", line 884
 self._bootstrap_inner()
New Bytes: 608	Total Bytes 608	New blocks: 4	Total blocks: 4: 
 File "/usr/local/lib/python3.6/tracemalloc.py", line 387
 self.traces = _Traces(traces)
 File "/usr/local/lib/python3.6/tracemalloc.py", line 524
 return Snapshot(traces, traceback_limit)
 File "/gems/tests/integration/endpoint_connection_test.py", line 856
 snap1 = tracemalloc.take_snapshot()
 File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 198
 testfunction(**testargs)
 File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187
 res = hook_impl.function(*args)
 File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87
 firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
 File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93
 return self._inner_hookexec(hook, methods, kwargs)
 File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286
 return self._hookexec(self, self.get_hookimpls(), kwargs)
 File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 1459
 self.ihook.pytest_pyfunc_call(pyfuncitem=self)
 File "/usr/local/lib/python3.6/site-packages/_pytest/runner.py", line 111
 item.runtest()
==========
Output in v3.7.4:
New Bytes: 36000	Total Bytes 36000	New blocks: 1000	Total blocks: 1000: 
 File "/usr/local/lib/python3.7/threading.py", line 890
 self._bootstrap_inner()
 File "/usr/local/lib/python3.7/threading.py", line 914
 self._set_tstate_lock()
 File "/usr/local/lib/python3.7/threading.py", line 904
 self._tstate_lock = _set_sentinel()
New Bytes: 32768	Total Bytes 32768	New blocks: 1	Total blocks: 1: 
 File "/usr/local/lib/python3.7/threading.py", line 890
 self._bootstrap_inner()
 File "/usr/local/lib/python3.7/threading.py", line 914
 self._set_tstate_lock()
 File "/usr/local/lib/python3.7/threading.py", line 909
 _shutdown_locks.add(self._tstate_lock)
=================
It looks like this commit didn't take into account the tstate_lock cleanup that happens in the C code, and it's not removing the _tstate_lock of completed threads from the _shutdown_locks once the thread finishes, unless the code manually calls "join()" or "is_alive()" on the thread:
https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd
Let me know if I can provide more clarity on this!