Python memory management - dictionary

Question 1

I have a dictionary saved in a file. I load the dictionary into memory from a python interactive shell, and my system monitor says that the python process consumes 4GB. The following commands give the following outputs:

size1 = sys.getsizeof(mydict)/(1024**2)
print size1

96

size2 = 0
for i in mydict.keys():
 size2 += sys.getsizeof(i)
print size2/(1024**2)

37

size3 = 0
for i in mydict.keys():
 size3 += sys.getsizeof(mydict[i])
print size3/(1024**2)

981

size4 = 0
for i in mydict.keys():
 for j in mydict[i]:
 size4 += j
print size4/(1024**2)

2302

print str(size1 + size2 + size3 + size4)

3416

Now if i delete the dictionary

del(mydict)
gc.collect()

less than 400MB are freed from memory. Even if i remove first all the items one by one from the lists inside the dictionary, the memory freed is no more than 450-500 MB.. So i end up with no variables in my shell, but still 3,5GB are consumed.. Can anyone explain what is happening?

Question 2

What happens if you set mydict = None before deleting?

Question 3

Same thing.. 400MB freed

Question 4

What are the contents of the dictionary? It is possible that you are double counting in your size calculation due to the way in which python handles small numbers or identical objects.

Question 5

There are two things to keep in mind:

Even if you delete the entire object, python might reserve that memory for further use(instead of allocating memory once again later). The footprint that python leaves in the os won't significantly change.

On Linux and UNIX-based systems processes do not necessarily give back the allocated memory until the application dies.

In order to successfully release the memory you might want to take a look at the garbage collector. With it you can force python to free the allocated memory.

Question 6

Thanks for your reply! it does clear thing up a bit. But i already use the garbage collector (gc.collect()) to clean up after i delete the dictionary content and dictionary, and it doesn't help. I will take a better look!

Question 7

you, sir, where correct! i loaded the dictionary, deleted it, and loaded it again and the total memory used was the same as the first time! thanks

Alexander Ejbekov 6,0501 gold badge28 silver badges27 bronze badges · Accepted Answer · 2014-04-20 16:15:49Z

There are two things to keep in mind:

Even if you delete the entire object, python might reserve that memory for further use(instead of allocating memory once again later). The footprint that python leaves in the os won't significantly change.

On Linux and UNIX-based systems processes do not necessarily give back the allocated memory until the application dies.

In order to successfully release the memory you might want to take a look at the garbage collector. With it you can force python to free the allocated memory.

Thanks for your reply! it does clear thing up a bit. But i already use the garbage collector (gc.collect()) to clean up after i delete the dictionary content and dictionary, and it doesn't help. I will take a better look!
you, sir, where correct! i loaded the dictionary, deleted it, and loaded it again and the total memory used was the same as the first time! thanks

CollectivesTM on Stack Overflow

Python memory management - dictionary

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Hot Network Questions

CollectivesTM on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Related