Message 62956 - Python tracker

➜

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

In-reply-to
Author	jaredgrubb
Recipients	jaredgrubb
Date	2008年02月25日.01:59:16
SpamBayes Score	0.005110738
Marked as misclassified	No
Message-id	<1203904758.19.0.84784431119.issue2180@psf.upfronthosting.co.za>

Content
tokenize does not handle line joining properly, as the following string fails the CPython tokenizer but passes the tokenize module. Example 1: >>> s = "if 1:\n \\\n #hey\n print 1" >>> exec s Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 3 #hey ^ SyntaxError: invalid syntax >>> tokenize.tokenize(StringIO(s).readline) 1,0-1,2: NAME 'if' 1,3-1,4: NUMBER '1' 1,4-1,5: OP ':' 1,5-1,6: NEWLINE '\n' 2,0-2,2: INDENT ' ' 3,2-3,6: COMMENT '#hey' 3,6-3,7: NEWLINE '\n' 4,2-4,7: NAME 'print' 4,8-4,9: NUMBER '1' 5,0-5,0: DEDENT '' 5,0-5,0: ENDMARKER ''

Content

tokenize does not handle line joining properly, as the following string
fails the CPython tokenizer but passes the tokenize module.
Example 1:
>>> s = "if 1:\n \\\n #hey\n print 1"
>>> exec s
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<string>", line 3
 #hey
 ^
SyntaxError: invalid syntax
>>> tokenize.tokenize(StringIO(s).readline)
1,0-1,2:	NAME	'if'
1,3-1,4:	NUMBER	'1'
1,4-1,5:	OP	':'
1,5-1,6:	NEWLINE	'\n'
2,0-2,2:	INDENT	' '
3,2-3,6:	COMMENT	'#hey'
3,6-3,7:	NEWLINE	'\n'
4,2-4,7:	NAME	'print'
4,8-4,9:	NUMBER	'1'
5,0-5,0:	DEDENT	''
5,0-5,0:	ENDMARKER	''

History
Date	User	Action	Args
2008年02月25日 01:59:18	jaredgrubb	set	spambayes_score: 0.00511074 -> 0.005110738 recipients: + jaredgrubb
2008年02月25日 01:59:18	jaredgrubb	set	spambayes_score: 0.00511074 -> 0.00511074 messageid: <1203904758.19.0.84784431119.issue2180@psf.upfronthosting.co.za>
2008年02月25日 01:59:17	jaredgrubb	link	issue2180 messages
2008年02月25日 01:59:16	jaredgrubb	create

homepage