numpy.ndarray objects not garbage collected

Question

While trying to fine-tune some memory leaks in the Python bindings for some C/C++ functions I cam across some strange behavior pertaining to the garbage collection of Numpy arrays.

I have created a couple of simplified cases in order to better explain the behavior. The code was run using the memory_profiler, the output from which follows immediately after. It appears that Python's garbage collection is not working as expected when it comes to NumPy arrays:

# File deallocate_ndarray.py
@profile
def ndarray_deletion():
 import numpy as np
 from gc import collect
 buf = 'abcdefghijklmnopqrstuvwxyz' * 10000
 arr = np.frombuffer(buf)
 del arr
 del buf
 collect()
 y = [i**2 for i in xrange(10000)]
 del y
 collect()
if __name__=='__main__':
 ndarray_deletion()

With the following command I invoked the memory_profiler:

python -m memory_profiler deallocate_ndarray.py

This is what I got:

Filename: deallocate_ndarray.py
Line # Mem usage Increment Line Contents
================================================
 5 10.379 MiB 0.000 MiB @profile
 6 def ndarray_deletion():
 7 17.746 MiB 7.367 MiB import numpy as np
 8 17.746 MiB 0.000 MiB from gc import collect
 9 17.996 MiB 0.250 MiB buf = 'abcdefghijklmnopqrstuvwxyz' * 10000
10 18.004 MiB 0.008 MiB arr = np.frombuffer(buf)
11 18.004 MiB 0.000 MiB del arr
12 18.004 MiB 0.000 MiB del buf
13 18.004 MiB 0.000 MiB collect()
14 18.359 MiB 0.355 MiB y = [i**2 for i in xrange(10000)]
15 18.359 MiB 0.000 MiB del y
16 18.359 MiB 0.000 MiB collect()

I don't understand why even the forced calls to collect don't reduce the memory usage of the program by freeing up some memory. Moreover, even if Numpy arrays don't behave normally due to the underlying C constructs, why doesn't the list (which is pure Python) get garbage collected?

I know that del does not directly call the underlying __del__ method, but you will note that all del statements in the code actually end up reducing the reference count of the corresponding objects to zero (thereby making them eligible for garbage collection AFAIK). Typically, I would expect to see a negative entry in the increment column when an object undergoes garbage collection. Can anyone shed some light on what is going on here?

NOTE: This test was run on OS X 10.10.4, Python 2.7.10 (conda), Numpy 1.9.2 (conda), Memory Profiler 0.33 (conda-binstar), psutil 2.2.1 (conda).

Answer 1

In order to see the memory garbage collected, I had to increase the size of buf several orders of magnitude. Maybe the size is too small for memory_profiler to detect the change (it queries the OS, so measurements are not very precise) or maybe its too small for the Python garbage collector to care, I don't know.

For example, replacing 10000 by 100000000 in the factor buf yields

Line # Mem usage Increment Line Contents
================================================
21 10.289 MiB 0.000 MiB @profile
22 def ndarray_deletion():
23 17.309 MiB 7.020 MiB import numpy as np
24 17.309 MiB 0.000 MiB from gc import collect
25 2496.863 MiB 2479.555 MiB buf = 'abcdefghijklmnopqrstuvwxyz' * 100000000
26 2496.867 MiB 0.004 MiB arr = np.frombuffer(buf)
27 2496.867 MiB 0.000 MiB del arr
28 17.312 MiB -2479.555 MiB del buf
29 17.312 MiB 0.000 MiB collect()
30 17.719 MiB 0.406 MiB y = [i**2 for i in xrange(10000)]
31 17.719 MiB 0.000 MiB del y
32 17.719 MiB 0.000 MiB collect()