Is it possible to define a protocol for a method? · python/typing · Discussion #1040

tobinus
Jan 19, 2022

To avoid repeating myself too much, I sometimes like to create helper functions that return a new function which is intended to be used as a method. The "method creating function" would typically take in some arguments. It would then create a new function that receives self as its first argument along with some other arguments, while making use of the arguments that were given to the method creating function. This new function would be returned and used as any other method.

What I'm looking for, is a way to:

Define a "method creating function" that takes some arguments and returns a new function
...with the new function being passed the instance as its first argument (self)
...and being properly typechecked, so that any users of the class can have their method calls checked
...and using overloads when defining the new function's signature

Using Callable

Typing this using Callable works properly (please don't mind the examples here not justifying the use of a method creating function):

from collections.abc import Callable
from typing import Any
def create_int_converter() -> Callable[[Any, str], int]:
 return lambda self, s: int(s)
class ClassWithDynamicallyCreatedMethod:
 convert_to_int = create_int_converter()
o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ✅
Python: ✅

Using callback protocol without extra self argument

However, I'd like to define overloads for the generated method. To do this, I thought I could build on the concept of "Callback protocols" as mentioned in PEP 544:

from typing import Any, Protocol
class ConverterProtocol(Protocol):
 def __call__(self, s: str) -> int: ...
def create_int_converter() -> ConverterProtocol:
 def converter(self: Any, s: str) -> int:
 return int(s)
 return converter
class ClassWithDynamicallyCreatedMethod:
 greeting = "Hi there!"
 convert_to_int = create_int_converter()
o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ❌
Python: ✅

Mypy error:

test_dynamic_method_mypy.py:13: error: Incompatible return value type (got "Callable[[Any, str], int]", expected "ConverterProtocol")
test_dynamic_method_mypy.py:13: note: "ConverterProtocol.__call__" has type "Callable[[Arg(str, 's')], int]"
Found 1 error in 1 file (checked 1 source file)

Using callback protocol with extra self argument

I tried experimenting with adding an explicit self argument, in addition to the one that is needed because __call__ is a method itself:

from typing import Any, Protocol
class ConverterProtocol(Protocol):
 def __call__(self_, self: Any, s: str) -> int: ...
def create_int_converter() -> ConverterProtocol:
 def converter(self: Any, s: str) -> int:
 return int(s)
 return converter
class ClassWithDynamicallyCreatedMethod:
 greeting = "Hi there!"
 convert_to_int = create_int_converter()
o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ❌
Python: ✅ (since we only changed the annotated Protocol)

Mypy error:

test_dynamic_method_mypy.py:21: error: Missing positional argument "s" in call to "__call__" of "ConverterProtocol"
Found 1 error in 1 file (checked 1 source file)

Thoughts

I first thought this would be a bug in Mypy. And maybe it is? But I'm also thinking there is a nuanced difference between how a function and a callable user object are treated with respect to the first self argument? Because if I make it so that create_int_converter() returns a proper instance of a class that implements ConverterProtocol, I do in fact not receive o as any argument, thus Python fails with the error that Mypy reported. Meanwhile, the converter() function receives o as its first argument.

I'm starting to think that the "callback protocol" approach is not usable for this usecase, because an instance of a class with a __call__ method is not functionally equivalent to a function when assigned as a class attribute. And the "callback protocol" approach wrongfully assumes those two are functionally equivalent. Is there any other way of creating a method creating function whose return value has an overloaded signature?

The following two Mypy issues may be related: python/mypy#708 and python/mypy#5485 but I don't think they address the inconsistencies the latter two examples demonstrate? I'll consider opening issues on that later.

Answered by Akuli

Jan 19, 2022

I'm starting to think that the "callback protocol" approach is not usable for this usecase, because an instance of a class with a __call__ method is not functionally equivalent to a function when assigned as a class attribute.

That's correct. When Python looks up a method from an instance, it calls __get__ to get a method object, which it then __call__()s. If there is no __get__, the object is called as is, without passing self.

Consider this, for example:

>>> class Foo:
... def method(self):
... print("hi")
... 
>>> f = Foo()
>>> f.method
<bound method Foo.method of <__main__.Foo object at 0x7f58edb5f310>>
>>> Foo.method
<function Foo.method at 0x7f58edb30790>
>>> Foo.m...

View full answer

Replies: 2 comments 9 replies

Akuli
Jan 19, 2022

I'm starting to think that the "callback protocol" approach is not usable for this usecase, because an instance of a class with a __call__ method is not functionally equivalent to a function when assigned as a class attribute.

Consider this, for example:

>>> class Foo:
... def method(self):
... print("hi")
... 
>>> f = Foo()
>>> f.method
<bound method Foo.method of <__main__.Foo object at 0x7f58edb5f310>>
>>> Foo.method
<function Foo.method at 0x7f58edb30790>
>>> Foo.method.__get__(f)
<bound method Foo.method of <__main__.Foo object at 0x7f58edb5f310>>

For function objects, the __get__ method returns a method object, which ensures that self is passed: f.method, aka Foo.method.__get__(f), behaves a lot like functools.partial(Foo.method, f). But the return value of __get__ can be anything, and __get__() can do whatever it wants instead of even calling your method. This is how @property and @staticmethod work, for example.

To tell mypy that your object isn't just a callable, but a method, you need to explain to mypy that __get__ behaves just like shown above. You can do this by adding a __get__ method to your protocol. I believe it works, but I haven't tried it. Let me know if you have trouble getting this to work, and I'll try to write some example code :)

9 replies

@tobinus

tobinus Jan 20, 2022
Author

Hm, okay, thank you! Just so I understand you correctly: In the solution for the first problem, the protocols are generalized to work for any function and do not contain any information about the function signature, other than how you no longer need to provide the first argument when accessing the method as an instance member? So the function's signature is inferred from how it looks inside create_int_converter, where the @overload decorator could have been used (for a more complex example)?

Regarding pyright: Thanks for the pointer, I may try to use it. Mypy just happened to be the first static type checking tool I was introduced to in Python.

Regarding types.MethodType: I'm not sure if it is possible to make it generic in typeshed, considering subscription of types.MethodType would likely fail at runtime? So the standard library would need to implement types.MethodType.__class_getitem__ if it is not implemented already. Anyways, I don't think I can prioritize improving this in typeshed, unfortunately.

@Akuli

Akuli Jan 20, 2022

the protocols are generalized to work for any function and do not contain any information about the function signature, other than how you no longer need to provide the first argument when accessing the method as an instance member? So the function's signature is inferred from how it looks inside create_int_converter

Yes, the ParamSpec is inferred from the function's signature. Apart from inferring from an existing function, I don't think there is currently a way to tell a ParamSpec to represent specific parameters. I believe ParamSpec works with @overload, but I haven't checked.

So the standard library would need to implement types.MethodType.__class_getitem__ if it is not implemented already.

CPython has previously accepted pull requests like this, but it's not super important that it exists at runtime. It's often possible to use from __future__ import annotations or if TYPE_CHECKING: to work around missing __class_getitem__ methods.

@tobinus

tobinus Jan 21, 2022
Author

Okay, thanks!

@tobinus

tobinus Jan 21, 2022
Author

I had overlooked that the __get__ method is called with its second argument being None when accessed on the class object, and not an instance. The protocols should reflect this, so I've updated the example I posted (in case anyone else finds this useful):

from __future__ import annotations
from typing import Protocol, Type, cast, overload
class WithDefaultInt(Protocol):
 default_int: int
class BoundConverter(Protocol):
 def __call__(self, s: str) -> int:
 ...
class Converter(Protocol):
 def __call__(self_, self: WithDefaultInt, s: str) -> int:
 ...
 @overload
 def __get__(
 self, obj: WithDefaultInt, objtype: Type[WithDefaultInt] | None = None
 ) -> BoundConverter:
 ...
 @overload
 def __get__(
 self, obj: None, objtype: Type[WithDefaultInt] | None = None
 ) -> Converter:
 ...
 def __get__(
 self, obj: WithDefaultInt | None, objtype: Type[WithDefaultInt] | None = None
 ) -> BoundConverter | Converter:
 ...
def create_int_converter(accept_errors: bool = False) -> Converter:
 def converter(self: WithDefaultInt, s: str) -> int:
 try:
 return int(s)
 except ValueError:
 if accept_errors:
 return self.default_int
 else:
 raise
 return cast(Converter, converter)
class ConverterCollection:
 default_int = 0
 convert_to_int = create_int_converter()
 convert_to_int_with_default = create_int_converter(True)
# Happy case:
converters = ConverterCollection()
result: int = converters.convert_to_int("123")
another_result: int = converters.convert_to_int_with_default("text")
print(repr(result), repr(another_result)) # prints 123 0
# This does not typecheck, ensuring us that our method signature is used:
str_result: str = converters.convert_to_int("456")
# Case which should fail because we are leaving out default_int:
class ConverterCollectionWithoutDefault:
 convert_to_int = create_int_converter(True)
converters_wo_default = ConverterCollectionWithoutDefault()
# This fails as it should, ensuring the method is not used with an incompatible class:
third_result: int = converters_wo_default.convert_to_int("testing")

@cj81499

cj81499 May 1, 2025

Also, I forgot to mention that types.MethodType is a joke. At runtime, it's same as types.FunctionType.

fwiw, this no longer seems to be the case.

MethodType: https://github.com/python/typeshed/blob/a46eea77e3c5afdaa9d02a2caf17a7375547164c/stdlib/types.pyi#L456-L475

FunctionType: https://github.com/python/typeshed/blob/a46eea77e3c5afdaa9d02a2caf17a7375547164c/stdlib/types.pyi#L69-L113

import types
def normal_fn[T](x: T) -> T:
 return x
lambda_fn = lambda a: a
class MyClass:
 def my_method(self) -> None:
 pass
my_instance = MyClass()
for ele in [
 lambda_fn,
 normal_fn,
 MyClass.my_method,
 my_instance.my_method,
]:
 print(f"{ele=} {isinstance(ele, types.MethodType)=} {isinstance(ele, types.FunctionType)=}")

outputs:

ele=<function <lambda> at 0x7f2c8ad4fc40> isinstance(ele, types.MethodType)=False isinstance(ele, types.FunctionType)=True
ele=<function normal_fn at 0x7f2c8ace3600> isinstance(ele, types.MethodType)=False isinstance(ele, types.FunctionType)=True
ele=<function MyClass.my_method at 0x7f2c8ad78220> isinstance(ele, types.MethodType)=False isinstance(ele, types.FunctionType)=True
ele=<bound method MyClass.my_method of <__main__.MyClass object at 0x7f2c8ae53620>> isinstance(ele, types.MethodType)=True isinstance(ele, types.FunctionType)=False

Observe that nothing is a MethodType except for my_instance.my_method and observe that everything other than my_instance.my_method is a FunctionType.

Unfortunately, mypy doesn't seem to understand that my_instance.my_method is a MethodType without using isinstance first: https://mypy-play.net/?mypy=latest&python=3.13&gist=f64203caf7d696160267bf4dd5cb9aad

Answer selected by tobinus

zoranbosnjak
Aug 11, 2024

@tobinus @Akuli
I have read many issues about this problem and this discussion is the closest I could find. I am trying to use this approach in the following example:

from typing import *
class Test1:
 @overload
 def f(self, key: Literal['A']) -> int: ...
 @overload
 def f(self, key: Literal['B']) -> str: ...
 def f(self, key: Any) -> Any:
 if key == 'A': return 1
 elif key == 'B': return 'x'
 assert_never(key)
class Test2:
 def __init__(self, arg: Test1):
 self._arg = arg
 def get_arg(self) -> Test1:
 return self._arg
 # not sure how to deduce the type signature from Test1.f
 def f(self, key):
 return self.get_arg().f(key)
test2 = Test2(Test1())
# expecting the same type
reveal_type(test2.get_arg().f)
reveal_type(test2.f)

... but I could not make it work with mypy.

With this approach, I get Invalid self argument "Test2" to attribute function "f"..., which kind of makes sense.

T = TypeVar('T')
def same_as(f: T) -> Callable[[Callable[..., Any]], T]:
 def wrap(_g): # ignore wrapped function, use 'f'
 def wrapped_f(*args, **kwargs):
 return f(*args, **kwargs)
 return wrapped_f
 return wrap
class Test2:
 ...
 @same_as(Test1.f)
 def f(self, key):
 ...

But If I try with a protocol, like this:

from typing import *
P = ParamSpec("P")
R = TypeVar("R", covariant=True)
class Method(Protocol, Generic[P, R]):
 def __get__(self, instance: Any, owner: type | None = None) -> Callable[P, R]:
 ...
 def __call__(self_, self: Any, *args: P.args, **kwargs: P.kwargs) -> R:
 ...
def same_as(f: Method[P, R]) -> Callable[[Callable[..., Any]], Method[P, R]]:
 raise NotImplementedError
class Test1:
 @overload
 def f(self, key: Literal['A']) -> int: ...
 @overload
 def f(self, key: Literal['B']) -> str: ...
 def f(self, key: Any) -> Any:
 if key == 'A': return 1
 elif key == 'B': return 'x'
 assert_never(key)
class Test2:
 def __init__(self, arg: Test1):
 self._arg = arg
 def get_arg(self) -> Test1:
 return self._arg
 @same_as(Test1.f)
 def f(self, key: Any) -> Any:
 ...
test2 = Test2(Test1())
# expecting the same type
reveal_type(test2.get_arg().f)
reveal_type(test2.f)

... I get the wrong type for test2.f, like if the @overload is ignored.

test1.py:40: note: Revealed type is "Overload(def (key: Literal['A']) -> builtins.int, def (key: Literal['B']) -> builtins.str)"
test1.py:41: note: Revealed type is "def (key: Literal['A']) -> Any"

I would appreciate any suggestion how to resolve the problem, that is: test2.f to be the same as test2.get_arg().f

0 replies

Is it possible to define a protocol for a method? #1040

Uh oh!

Uh oh!

tobinus Jan 19, 2022

Using Callable

Using callback protocol without extra self argument

Using callback protocol with extra self argument

Thoughts

Replies: 2 comments · 9 replies

Uh oh!

Uh oh!

Akuli Jan 19, 2022

Uh oh!

tobinus Jan 20, 2022 Author

Uh oh!

Uh oh!

Akuli Jan 20, 2022

Uh oh!

tobinus Jan 21, 2022 Author

Uh oh!

Uh oh!

tobinus Jan 21, 2022 Author

Uh oh!

cj81499 May 1, 2025

Uh oh!

zoranbosnjak Aug 11, 2024

tobinus
Jan 19, 2022

Replies: 2 comments 9 replies

Akuli
Jan 19, 2022

tobinus Jan 20, 2022
Author

tobinus Jan 21, 2022
Author

tobinus Jan 21, 2022
Author

zoranbosnjak
Aug 11, 2024