Tracking object allocation in python

Question

Is there any method I can override that will allow me to use print statements / pdb / etc. to keep track of every time an instance of my class is allocated? While unpickling some objects I am seeming to get some that never have either __setstate__ or __init__ called on them. I tried overriding __new__ and printing out the id of every object I make in __new__, but I am still encountering objects with ids that were never printed.

Edit: here is my code I use for altering (instrumenting) __new__ of my class and all of its super-classes except for object itself:

class Allocator:
 def __init__(self, my_class):
 self.my_class = my_class
 self.old_new = my_class.__new__
 def new(self, * args, ** kargs):
 rval = self.old_new(*args, ** kargs)
 #rval = super(self.my_class,cls).__new__(cls)
 print 'Made '+str(self.my_class)+' with id '+str(id(rval))
 return rval
def replace_allocator(cls):
 if cls == object:
 return
 setattr(cls,'__new__',Allocator(cls).new)
 print cls.__base__
 try:
 for parent in cls.__base__:
 replace_allocator(parent)
 except:
 replace_allocator(cls.__base__)

I call replace_allocator on my classes' parent class as soon as it is imported in the main script. My class has a custom __new__ to begin with, which also prints out the id.

• Rosh Oxymoron: Are you absolutely certain they are different instances? Have you checked if id(instance) matches?
• Apalala: Are you trying to use Allocator(cls) as a type cast? If so, it is wrong, because in Python it is a constructor call for an Allocator object, and the .new part will get a new bound-method instance.
• Ian Goodfellow: Rosh: Yes. My allocator function shows the way that I print out id(instance). Later, when I find an object that is missing some fields that should have been added by setstate or init, I print out its id too. If I grep for that id in the earlier printouts there are no hits.
• Ian Goodfellow: Apalala: It is indeed my intention to use Allocator(cls) as a constructor call for an Allocator object. The point of an Allocator constructed with cls is to call the original __new__ method of cls and report the id of the resulting object instance.
• Sven Marnach: The iteration should be for parent in cls.__bases__. You don't need the fall-back to cls.__base__. And don't use catch-all except: clauses. They always bite you.

Answer 1

(This is more of a comment than an answer.)

Quoting Guido's Unifying types and classes in Python 2.2:

There are situations where a new instance is created without calling __init__ (for example when the instance is loaded from a pickle). There is no way to create a new instance without calling __new__ (although in some cases you can get away with calling a base class's __new__).

If you are using new-style classes (descendants of object), __new__() should always be called. I don't think the obscure cases "you can get away with calling a base class's __new__" in will happen accidently, though I don't know what these cases actually are.

And just to add an example:

In [1]: class A(object):
 ...: def __new__(cls): 
 ...: print "A"
 ...: return object.__new__(cls)
 ...: 
In [2]: A()
A
Out[2]: <__main__.A object at 0xa3a95cc>
In [4]: object.__new__(A)
Out[4]: <__main__.A object at 0xa3a974c>

• Ian Goodfellow: This sounds like a feasible approach but unfortunately if you can get away with calling the base class, then that might explain why there are objects getting allocated without my knowledge. I'm only allowed to override __new__ for classes other than object itself.

Answer 2

Are you using new-style classes? For Pickle to call __setstate__, the __getstate__ method should also be defined on the class returning a non-False value.

• Ian Goodfellow: This does not answer the question of how to detect objects as they are created. But yes I am using new-style classes and I do have a __getstate__ implemented. Also, __setstate__ is getting called on most but not all instances of the class.

Answer 3

Not sure if this can help you, but Python's garbage collector has introspective capabilities that might be worth taking a look at.

• Sven Marnach: The question is about keeping track of the creation of objects.
• s.m.: You're right, I most probably misunderstood the question. But now I wonder, won't the garbage collector need itself to monitor object allocation? If so, one might devise a method to hook into the gc and figure out when an object of interested gets allocated. I'm not questioning what you said, I'm just curious.