homepage

Hi,
I'm looking in to the memory usage of our testsuite, which does a fair amount of
 def setUp():
 patcher = patch.multiple(...)
 self.mock_whatever = patcher.start()
 self.addCleanup(patcher.stop)
or other ways of creating a mock and assigning it to an instance variable on a TestCase.
This means that by the end of a run, we have quite a lot of references to MagicMocks.
This then becomes the majority of the memory usage of the python process: (from meliae)
Total 1157176 objects, 1189 types, Total size = 327.1MiB (342972340 bytes)
 Index Count % Size % Cum Max Kind
 0 575750 49 198058000 57 57 344 MagicProxy
 1 49955 4 52034888 15 72 196888 dict
 2 124127 10 11881628 3 76 386477 str
 3 12997 1 11749288 3 79 904 type
 4 8225 0 9146200 2 82 1112 MagicMock
 5 66310 5 5282392 1 84 80056 tuple
 6 38161 3 4579320 1 85 120 function
 7 1503 0 3972281 1 86 49488 module
 8 28506 2 3648768 1 87 128 code
 9 25768 2 2869680 0 88 69168 list
 10 12649 1 2803196 0 89 66356 unicode
 11 2251 0 2503112 0 89 1112 ClientHandler
 12 2228 0 2477536 0 90 1112 _patch
 13 28223 2 2257840 0 91 80 instancemethod
 14 2014 0 2239568 0 91 1112 BoundMethodWeakref
 15 2003 0 2227336 0 92 1112 DummyCache
 16 24681 2 2221112 0 93 792 _CallList
 17 18555 1 1632840 0 93 88 weakref
 18 1457 0 1550248 0 94 1064 Morsel
 19 46258 3 1110192 0 94 24 int
The fact that each MagicMock creates 72 MagicProxies means that it is a significant chunk of memory, despite each being small.
John Arbash Meinel suggested setting __slots__ = ['name', 'parent'] on MagicProxy to reduce the memory usage of this simple object.
This helps with the memory usage:
Total 1140377 objects, 1189 types, Total size = 169.5MiB (177755867 bytes)
 Index Count % Size % Cum Max Kind
 0 47744 4 51347840 28 28 196888 dict
 1 574210 50 36749440 20 49 64 MagicProxy
 2 122020 10 11769659 6 56 386477 str
 3 12975 1 11729400 6 62 904 type
 4 8203 0 9121736 5 67 1112 MagicMock
 5 64125 5 5141368 2 70 80056 tuple
 6 36024 3 4322880 2 73 120 function
 7 1503 0 3972281 2 75 49488 module
 8 28506 2 3648768 2 77 128 code
 9 12643 1 2801540 1 79 66356 unicode
 10 23634 2 2716064 1 80 69168 list
 11 2251 0 2503112 1 82 1112 ClientHandler
 12 28223 2 2257840 1 83 80 instancemethod
 13 2014 0 2239568 1 84 1112 BoundMethodWeakref
 14 2003 0 2227336 1 85 1112 DummyCache
 15 24615 2 2214536 1 87 792 _CallList
 16 18482 1 1626416 0 87 88 weakref
 17 1457 0 1550248 0 88 1064 Morsel
 18 46259 4 1110216 0 89 24 int
 19 2496 0 858624 0 89 344 ModelState
I'm going to go through and drop references so that these can get garbage collected, but making Mock less memory-intensive would probably be appreciated by its users.
Reducing the memory usage of the tiny MagicProxies would be good, but also if there is a way to reduce the number of them by not pre-assiging 72 of them for every MagicMock, when each is very unlikely to be used, then that would be good as well.
Thanks,
James

Switching to Mock instead of MagicMock may help, as that doesn't have the magic proxies at all. (patch has an argument to specify which class of Mock should be used to create the mock object, MagicMock is just the default.)
Other wise using __slots__ would be fine. An alternative approach would be for MagicMock to use __getattribute__ instead of the magic proxies.

> This means that by the end of a run, we have quite a lot of references to MagicMocks.
It sounds like you are bitten by unittest keeping references to all past TestCase instances. This has been fixed recently (see #11798).

James, could you check again with a recent Python and see if the #11798 fix makes things any better?

James, please confirm that this issue still exists in 3.6 and 3.7.