[Python-ideas] Weak-referencing/weak-proxying of (bound) methods

Sat Jun 16 10:42:57 CEST 2012

Jan Kaliszewski dixit (2012年06月16日, 01:41):
> Tal Einat dixit (2012年06月15日, 15:41):
>> > On Mon, Jun 11, 2012 at 2:16 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> [snip]
> > > >>> import weakref
> > > >>> class A:
> > > ... def method(self): print(self)
> > > ...
> > > >>> A.method
> > > <function method at 0xb732926c>
> > > >>> a = A()
> > > >>> a.method
> > > <bound method A.method of <__main__.A object at 0xb7326bec>>
> > > >>> r = weakref.ref(a.method) # creating a weak reference
> > > >>> r # ...but it appears to be dead
> > > <weakref at 0xb7327d9c; dead>
> > > >>> w = weakref.proxy(a.method) # the same with a weak proxy
> > > >>> w
> > > <weakproxy at 0xb7327d74 to NoneType at 0x829f7d0>
> > > >>> w()
> > > Traceback (most recent call last):
> > > File "<stdin>", line 1, in <module>
> > > ReferenceError: weakly-referenced object no longer exists
> > >
> > > This behaviour is perfectly correct -- but still surprising,
> > > especially for people who know little about method creation
> > > machinery, descriptors etc.
> > >
> > > I think it would be nice to make this 'trap' less painful --
> [snip]
> > > A prototype implementation:
> > >
> > > class InstanceCachedMethod(object):
> > >
> > > def __init__(self, func):
> > > self.func = func
> > > (self.instance_attr_name
> > > ) = '__{0}_method_ref'.format(func.__name__)
> > >
> > > def __get__(self, instance, owner):
> > > if instance is None:
> > > return self.func
> > > try:
> > > return getattr(instance, self.instance_attr_name)
> > > except AttributeError:
> > > method = types.MethodType(self.func, instance)
> > > setattr(instance, self.instance_attr_name, method)
> > > return method
> [snip]
> > I was bitten by this issue a while ago as well. It made working with
> > weakref proxies much more involved than I expected it would be.
> > 
> > Wouldn't it be better to approach the issue from the opposite end, and
> > improve/wrap/replace weakref.proxy with something that can handle bound
> > methods?
>> Indeed, probably could it be done by wrapping weakref.ref()/proxy()
> with something like the following:
>> # here `obj` is the object that is being weak-referenced...
> if isinstance(obj, types.MethodType):
> try:
> cache = obj.__self__.__method_cache__
> except AttributeError:
> cache = obj.__self__.__method_cache__ = WeakKeyDictionary()
> method_cache.setdefault(obj.__func__, set()).add(obj)
>> (Using WeakKeyDictionary with corresponding function objects as weak
> keys -- to provide automagic cleanup when a function is deleted, e.g.
> replaced with another one. In other words: the actual weak ref/proxy
> to a method lives as long as the corresponding function does).

On second thought -- no, it shouldn't be done on the side of
weakref.ref()/proxy().
Why? My last idea described just above has such a bug: each time
you create a new weak reference to the method another method object
is cached (added to __method_cache__[func] set).
You could think that caching only one object (just in
__method_cache__[func]) would be a better idea, but it wouldn't:
such a behaviour would be strange and unstable: after creating
a new weakref to the method, the old weakref would became invalid...
And yes, we can prevent it by ensuring that each time you take
the method from a class instance you get the same object (per class
instance) -- but then we come back to my previous idea of a
descriptor-decorator. And IMHO such a decorator should not be
applied on the class dictionary implicitly by weakref.ref()/proxy()
but explicitly in the class body with the decorator syntax
(applying such a decorater, i.e. replacing a function with a
caching descriptor is a class dict, is too invasive operation to
be done silently).
So I renew (and update) my previous descriptor-decorator that
could be added to functools (or to weakref as a helper?) and
applied explicitly by programmers, when needed:
 class CachedMethod(object):
 def __init__(self, func):
 self.func = func
 def __get__(self, instance, owner):
 if instance is None:
 return self.func
 try:
 cache = instance.__method_cache__
 except AttributeError:
 # not thread-safe :-(
 cache = instance.__method_cache__ = WeakKeyDictionary()
 return cache.setdefault(
 self.func,
 types.MethodType(self.func, instance))
Usage:
 class MyClass(object):
 @CachedMethod
 def my_method(self):
 ...
 instance = MyClass()
 method_weak_proxy = weakref.proxy(instance.my_method)
 method_weak_proxy() # works!
It should be noted that caching a reference to a method in an
instance causes circular referencing (class <-> instance).
However, ofter it is not a problem and can help avoiding
circular references involving other objects which we want to
have circular-ref-free (typical use case: passing a bound
method as a callback).
Cheers.
*j