Mutable Named Tuple - or - Slotted Data Structure

Question

A question I answered on Stack Overflow asked for a mutable named tuple.

The naive answer would probably tell them they just want to define a custom object.

However, named tuples take up a lot less space than custom objects - unless those objects use __slots__.

Further, a namedtuple is semantically just a record of data (maybe with some functionality associated with it.) So what if we really want something like a mutable "namedtuple" then?

I present, for your edification, a MutableNamedTuple (lifted and somewhat modified from my answer):

from collections import Sequence
class MutableNamedTuple(Sequence): 
 """Abstract Base Class for objects as efficient as mutable
 namedtuples. 
 Subclass and define your named fields with __slots__.
 """
 __slots__ = ()
 def __init__(self, *args):
 for slot, arg in zip(self.__slots__, args):
 setattr(self, slot, arg)
 def __repr__(self):
 return type(self).__name__ + repr(tuple(self))
 # more direct __iter__ than Sequence's
 def __iter__(self): 
 for name in self.__slots__:
 yield getattr(self, name)
 # Sequence requires __getitem__ & __len__:
 def __getitem__(self, index):
 return getattr(self, self.__slots__[index])
 def __len__(self):
 return len(self.__slots__)

And here's some examples of usage:

class MNT(MutableNamedTuple):
 """demo mutable named tuple with metasyntactic names"""
 __slots__ = 'foo bar baz quux'.split()
>>> mnt = MNT(*'abcd')
>>> mnt
MNT('a', 'b', 'c', 'd')
>>> mnt.foo
'a'
>>> mnt.bar
'b'
>>> mnt.baz
'c'
>>> mnt.quux
'd'

It is indexable just like a tuple:

>>> foo, bar, baz, quux = mnt
>>> foo
'a'
>>> bar
'b'
>>> baz
'c'
>>> quux
'd'
>>> mnt[0]
'a'
>>> mnt[3]
'd'

It even raises IndexError where apropos:

>>> mnt[4]
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<stdin>", line 18, in __getitem__
IndexError: list index out of range
>>> for c in mnt: print(c)
... 
a
b
c
d

Help isn't too bad if you give a decent docstring:

>>> help(MNT)
Help on class MNT in module __main__:
class MNT(MutableNamedTuple)
 | demo mutable named tuple with metasyntactic names
...

Thanks to the ABC's mixin methods we can reverse it, check its length, and check for membership:

>>> reversed(mnt)
<generator object Sequence.__reversed__ at 0x7ff46c64b678>
>>> list(reversed(mnt))
['d', 'c', 'b', 'a']
>>> sorted(mnt)
['a', 'b', 'c', 'd']
>>> len(mnt)
4
>>> 'a' in mnt
True
>>> 'e' in mnt
False

Because we don't define __setitem__ or __delitem__ we get the same behavior (TypeError, similar message) as tuples when we attempt to mutate it by index:

>>> del mnt[3]
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: 'MNT' object doesn't support item deletion
>>> mnt[3] = 'e'
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: 'MNT' object does not support item assignment

Perhaps this implementation of a mutable named tuple is even Liskov substitutable? We are not allowed to subclass tuple, because we use non-empty __slots__. If we just look at methods tuples have that MNT doesn't have, we only see a few methods we don't implement:

>>> set(dir(tuple)) - set(dir(MNT))
{'__rmul__', '__add__', '__getnewargs__', '__mul__'}

__mul__, __rmul__, and __add__ all don't really make semantic sense for a fixed length data structure.

And __getnewargs__ relates to pickling, but the object pickles and unpickles just fine:

>>> import pickle
>>> pickle.loads(pickle.dumps(mnt))
MNT('a', 'b', 'c', 'd')

I think this is rather clever - so I'm putting this out for your scrutiny and amusement.

(削除) Roast me! (削除ここまで) Review my code! And suggest improvements! (Maybe I could improve on the constructor? Maybe subclass the ABCMeta metaclass? Can I enforce a subclass to create a docstring? Should I assert that the number of arguments is the same as the number of slots? How should I unittest it? Etc...)

Answer 1

Here are some issues I noticed in your implementation.

• Currently you're simply iterating over __slots__ everywhere, but this doesn't handle the case when __slots__ is a string parameter like 'bar'. i.e this shouldn't result in three slots 'b', 'a' and 'r'.
• Your abstract base class MutableNamedTuple is directly instantiable.
• The __init__ method only works with positional arguments and the caller has no idea about its signature and will have to peak into its definition all the time.
• There's no verification of the arguments passed, hence I can pass more or less number of items to __init__ and it wouldn't complain(in case of less items we will get AttributeError later on).

The following modifications try to address the above issues using meta-programming and other features of Python 3:

from collections import Sequence
from inspect import Signature, Parameter
class MutableNamedTuple(Sequence): 
 """Abstract Base Class for objects as efficient as mutable
 namedtuples.
 Subclass and define your named fields with __slots__.
 """
 @classmethod
 def get_signature(cls):
 parameters = [
 Parameter(name=slot, kind=Parameter.POSITIONAL_OR_KEYWORD) for slot in cls.__slots__
 ]
 return Signature(parameters=parameters)
 def __init_subclass__(cls, **kwargs):
 super().__init_subclass__(**kwargs)
 slots = cls.__slots__
 cls.__slots__ = tuple(slots.split()) if isinstance(slots, str) else tuple(slots)
 cls.__signature__ = cls.get_signature()
 cls.__init__.__signature__ = cls.get_signature()
 cls.__doc__ = '{cls.__name__}{cls.__signature__}\n\n{cls.__doc__}'.format(
 cls=cls)
 def __new__(cls, *args, **kwargs):
 if cls is MutableNamedTuple:
 raise TypeError("Can't instantiate abstract class MutableNamedTuple")
 return super().__new__(cls)
 @classmethod
 def _get_bound_args(cls, args, kwargs):
 return Signature.bind(cls.__signature__, *args, **kwargs).arguments.items()
 __slots__ = ()
 def __init__(self, *args, **kwargs):
 bound_args = self._get_bound_args(args, kwargs)
 for slot, value in bound_args:
 setattr(self, slot, value)
 def __repr__(self):
 return type(self).__name__ + repr(tuple(self))
 def __iter__(self): 
 for name in self.__slots__:
 yield getattr(self, name)
 def __getitem__(self, index):
 return getattr(self, self.__slots__[index])
 def __len__(self):
 return len(self.__slots__)

Demo:

>>> MutableNamedTuple()
...
TypeError: Can't instantiate abstract class MutableNamedTuple
>>> help(MNT)
Help on class MNT in module __main__:
class MNT(MutableNamedTuple)
 | MNT(foo, bar, baz, quux)
 |
 | demo mutable named tuple with metasyntactic names
 |
...
>>> inspect.getfullargspec(MNT)
>>> FullArgSpec(args=['foo', 'bar', 'baz', 'quux'], varargs=None, varkw=None, defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})
>>> MNT(1, 2)
...
TypeError: missing a required argument: 'baz'
>>> MNT(1, bar=2, baz=2, quux=10, spam='eggs')
...
TypeError: got an unexpected keyword argument 'spam'
>>> m = MNT(1, 2, baz=2, quux=10)
>>> m.foo, m.bar, m.baz, m.quux
(1, 2, 2, 10)

• Aaron Hall: Good catch on the __slots__ being a string. Yes the ABC is directly instantiable, but fairly useless with no slots. Other points are quite valid as well. +1 (hmmm... if we use a mapping for __slots__ we could also type-check.)

Answer 2

This Python is mostly over my head — and Ashwini just gave a more complete answer — but I do notice this infelicity:

>>> x = MNT(*'abc')
>>> x
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<stdin>", line 11, in __repr__
 File "<stdin>", line 15, in __iter__
AttributeError: quux

It's possible to create an object of type MNT which is un-repr-able! Definitely you should fix this. Either the constructor should throw an exception, or it should initialize the uninitialized slots to None, or at worst it should leave them undefined and you should change __repr__ to return "incomplete tuple" or something. Having a __repr__ that throws is just bad news, IMO.

---

And since you mentioned pickling: it doesn't work for me, even with a not-incomplete MNT instance. I'm guessing you have a more recent Python version, maybe? I'm on 2.7.10.

>>> x = MNT(*'abcd')
>>> pickle.loads(pickle.dumps(x))
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
 Pickler(file, protocol).dump(obj)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
 self.save(obj)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
 rv = reduce(self.proto)
 File "/Users/ajo/env/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
 raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled 
>>> pickle.__version__
'$Revision: 72223 $'

---

Can I enforce a subclass to create a docstring?

Why would you want to? :) That's definitely a style choice that should be made at the style-guide or commit-hook level, not enforced at runtime by some random user code.

But if this is part of your style-guide, then yes, pip install pydocstyle and then pydocstyle . will do the trick!

• Aaron Hall: That pickle error should be updated to note that it's the version of the pickle protocol that matters - pass a protocol argument to dumps: docs.python.org/2/library/pickle.html#pickle.dumps (or upgrade to Python 3) - the repr issue has no semantic mapping to namedtuple - probably best solved by preventing deletion of attributes - and using (i)zip_longest or checking the number of args in the __init__ or __new__.