Generator using item[n-1] + item[n] memory

Sat Feb 15 06:27:19 EST 2014

Chris Angelico wrote:
> On Sat, Feb 15, 2014 at 6:27 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
>> On Fri, Feb 14, 2014 at 8:31 PM, Nick Timkovich <prometheus235 at gmail.com>
>> wrote:
>>> OK, now the trick; adding `data = None` inside the generator works, but
>>> in my actual code I wrap my generator inside of `enumerate()`, which
>>> seems to
>>> obviate the "fix". Can I get it to play nice or am I forced to count
>>> manually. Is that a feature?
>>>> Yeah, looks like enumerate also doesn't release its reference to the
>> previous object until after it gets the next one. You'll just have to
>> make do without.
>> You could write your own enumerate function.
>> def enumerate(it, i=0):
> it = iter(it)
> while True:
> yield i, next(it)
> i += 1
>> That shouldn't keep any extra references around.

An alternative approach ist to yield weak refs and thus have the generator 
control the object lifetime. This doesn't work with the built-in list type 
though:
import weakref
try:
 from itertools import imap # py2
except ImportError:
 imap = map # py3
N = 0
def log_deleted(*args):
 global N
 N -= 1
 print("deleted, new N: {}".format(N))
def log_created():
 global N
 N += 1
 print("created, new N: {}".format(N))
def weakrefs(f):
 def weakrefs(*args, **kw):
 return imap(lambda x: weakref.proxy(x, log_deleted), f(*args, **kw))
 return weakrefs
class List(list):
 def __str__(self):
 s = str(self[:5])
 if len(self) > 10:
 s = s[:-1] + ", ... ]"
 return s
@weakrefs
def biggen():
 sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
 for size in sizes:
 data = List([1] * int(size * 1e4))
 log_created()
 yield data
 data = None
if __name__ == "__main__":
 for i, x in enumerate(biggen()):
 print("{} {}".format(i, x))