homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Anton.Afanasyev
Recipients Anton.Afanasyev, rhettinger
Date 2014年04月21日.12:13:49
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1398082430.51.0.59207810702.issue21321@psf.upfronthosting.co.za>
In-reply-to
Content
This issue results in redundant memory consumption for e.g. in this case:
================================================
from itertools import *
def test_islice():
 items, lookahead = tee(repeat(1, int(1e9)))
 lookahead = islice(lookahead, 10)
 for item in lookahead:
 pass
 for item in items:
 pass
if __name__ == "__main__":
 test_islice()
================================================
This demo is taken from real case where one needs to look ahead input stream before processing it. For my PC this demo stops with 'Segmentation fault' message after exhausting all PC memory, while running it with patched python consumes only 0.1% of memory till the end.
When one uses pure pythonic implementation of itertools.islice() (taken from docs), the issue goes away as well, since this implementation doesn't store redundant reference to source iterator.
================================================
def islice(iterable, *args):
 s = slice(*args)
 it = iter(xrange(s.start or 0, s.stop or sys.maxint, s.step or 1))
 nexti = next(it)
 for i, element in enumerate(iterable):
 if i == nexti:
 yield element
 nexti = next(it)
================================================
Attaching patch for this issue. Have to change '__reduce__()' method since now unpickling of exhausted 'islice()' object cannot return old source iterator.
History
Date User Action Args
2014年04月21日 12:13:50Anton.Afanasyevsetrecipients: + Anton.Afanasyev, rhettinger
2014年04月21日 12:13:50Anton.Afanasyevsetmessageid: <1398082430.51.0.59207810702.issue21321@psf.upfronthosting.co.za>
2014年04月21日 12:13:50Anton.Afanasyevlinkissue21321 messages
2014年04月21日 12:13:49Anton.Afanasyevcreate

AltStyle によって変換されたページ (->オリジナル) /