homepage

➜

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Recipients
Author	rb
Date	2010年05月25日.16:57:25
SpamBayes Score	0.0
Marked as misclassified	No
Message-id
In-reply-to

Content
I've looked into this in some more depth. The problem is that Untokenizer.compat is assuming that iterable can restart from the beginning, when Untokenizer.untokenize has already had the first element out. So it works with a list, but not with a generator. In particular, untokenize is broken for any input that is a generator which only supplies the first two elements. Workaround: never hand untokenize a generator. Expand generators to lists first instead.

Content

I've looked into this in some more depth.
The problem is that Untokenizer.compat is assuming that iterable can restart from the beginning, when Untokenizer.untokenize has already had the first element out. So it works with a list, but not with a generator.
In particular, untokenize is broken for any input that is a generator which only supplies the first two elements.
Workaround: never hand untokenize a generator. Expand generators to lists first instead.

History
Date	User	Action	Args
2010年07月10日 04:08:08	loewis	set	spambayes_score: 0.930489 -> 0.0
2010年05月25日 16:57:26	rb	link	issue8478 messages
2010年05月25日 16:57:25	rb	create