homepage

Hi,
I'm looking in to the memory usage of our testsuite, which does a fair amount of
 def setUp():
 patcher = patch.multiple(...)
 self.mock_whatever = patcher.start()
 self.addCleanup(patcher.stop)
or other ways of creating a mock and assigning it to an instance variable on a TestCase.
This means that by the end of a run, we have quite a lot of references to MagicMocks.
This then becomes the majority of the memory usage of the python process: (from meliae)
Total 1157176 objects, 1189 types, Total size = 327.1MiB (342972340 bytes)
 Index Count % Size % Cum Max Kind
 0 575750 49 198058000 57 57 344 MagicProxy
 1 49955 4 52034888 15 72 196888 dict
 2 124127 10 11881628 3 76 386477 str
 3 12997 1 11749288 3 79 904 type
 4 8225 0 9146200 2 82 1112 MagicMock
 5 66310 5 5282392 1 84 80056 tuple
 6 38161 3 4579320 1 85 120 function
 7 1503 0 3972281 1 86 49488 module
 8 28506 2 3648768 1 87 128 code
 9 25768 2 2869680 0 88 69168 list
 10 12649 1 2803196 0 89 66356 unicode
 11 2251 0 2503112 0 89 1112 ClientHandler
 12 2228 0 2477536 0 90 1112 _patch
 13 28223 2 2257840 0 91 80 instancemethod
 14 2014 0 2239568 0 91 1112 BoundMethodWeakref
 15 2003 0 2227336 0 92 1112 DummyCache
 16 24681 2 2221112 0 93 792 _CallList
 17 18555 1 1632840 0 93 88 weakref
 18 1457 0 1550248 0 94 1064 Morsel
 19 46258 3 1110192 0 94 24 int
The fact that each MagicMock creates 72 MagicProxies means that it is a significant chunk of memory, despite each being small.
John Arbash Meinel suggested setting __slots__ = ['name', 'parent'] on MagicProxy to reduce the memory usage of this simple object.
This helps with the memory usage:
Total 1140377 objects, 1189 types, Total size = 169.5MiB (177755867 bytes)
 Index Count % Size % Cum Max Kind
 0 47744 4 51347840 28 28 196888 dict
 1 574210 50 36749440 20 49 64 MagicProxy
 2 122020 10 11769659 6 56 386477 str
 3 12975 1 11729400 6 62 904 type
 4 8203 0 9121736 5 67 1112 MagicMock
 5 64125 5 5141368 2 70 80056 tuple
 6 36024 3 4322880 2 73 120 function
 7 1503 0 3972281 2 75 49488 module
 8 28506 2 3648768 2 77 128 code
 9 12643 1 2801540 1 79 66356 unicode
 10 23634 2 2716064 1 80 69168 list
 11 2251 0 2503112 1 82 1112 ClientHandler
 12 28223 2 2257840 1 83 80 instancemethod
 13 2014 0 2239568 1 84 1112 BoundMethodWeakref
 14 2003 0 2227336 1 85 1112 DummyCache
 15 24615 2 2214536 1 87 792 _CallList
 16 18482 1 1626416 0 87 88 weakref
 17 1457 0 1550248 0 88 1064 Morsel
 18 46259 4 1110216 0 89 24 int
 19 2496 0 858624 0 89 344 ModelState
I'm going to go through and drop references so that these can get garbage collected, but making Mock less memory-intensive would probably be appreciated by its users.
Reducing the memory usage of the tiny MagicProxies would be good, but also if there is a way to reduce the number of them by not pre-assiging 72 of them for every MagicMock, when each is very unlikely to be used, then that would be good as well.
Thanks,
James