Message 117554 - Python tracker

➜

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

In-reply-to
Author	ncoghlan
Recipients	meador.inge, michael.foord, ncoghlan
Date	2010年09月28日.21:54:16
SpamBayes Score	0.00853855
Marked as misclassified	No
Message-id	<1285710858.05.0.587248130168.issue9969@psf.upfronthosting.co.za>

Content
Possible approach (untested): def get_tokens(source): if hasattr(source, "encode"): # Already decoded, so bypass encoding detection return _tokenize(io.StringIO(source).readline, None) # Otherwise attempt to detect the correct encoding return tokenize(io.BytesIO(source).readline)

Content

Possible approach (untested):
def get_tokens(source):
 if hasattr(source, "encode"):
 # Already decoded, so bypass encoding detection
 return _tokenize(io.StringIO(source).readline, None)
 # Otherwise attempt to detect the correct encoding
 return tokenize(io.BytesIO(source).readline)

History
Date	User	Action	Args
2010年09月28日 21:54:18	ncoghlan	set	recipients: + ncoghlan, michael.foord, meador.inge
2010年09月28日 21:54:18	ncoghlan	set	messageid: <1285710858.05.0.587248130168.issue9969@psf.upfronthosting.co.za>
2010年09月28日 21:54:16	ncoghlan	link	issue9969 messages
2010年09月28日 21:54:16	ncoghlan	create

homepage