Message 63895 - Python tracker

➜

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

In-reply-to
Author	ocean-city
Recipients	ocean-city
Date	2008年03月18日.05:22:29
SpamBayes Score	0.0045523075
Marked as misclassified	No
Message-id	<1205817752.04.0.892564898323.issue2382@psf.upfronthosting.co.za>

Content
Hello. I found another problem related to issue2301. SyntaxError cursor "^" is shifted when multibyte characters are in line (before "^"). I think this is because err->text is stored as UTF-8 which requires 3 bytes for multibyte character, but actually cp932 (my console encoding) requires only 2 bytes for it. So "^" is shited to right 5 bytes because there is 5 multibyte chars. C:\Documents and Settings\WhiteRabbit>py3k x.py push any key.... File "x.py", line 3 print "あいうえお" ^ SyntaxError: invalid syntax [22567 refs] Sorry, I didn't know what PyTokenizer_RestoreEncoding really doing. That function adjusted err_ret->offset for this encoding conversion. So, Python2.5 can output cursor in right place. (Of course, if source encoding is not compatible for console encoding, broken string is printed though. Anyway, cursor is right) C:\Documents and Settings\WhiteRabbit>py a.py File "a.py", line 2 x "、「、、、ヲ、ィ、ェ" ^ SyntaxError: invalid syntax [8728 refs] I tried to fix this problem, but I'm not sure how to fix this.

Content

Hello. I found another problem related to issue2301.
SyntaxError cursor "^" is shifted when multibyte
characters are in line (before "^").
I think this is because err->text is stored as UTF-8
which requires 3 bytes for multibyte character,
but actually cp932 (my console encoding) requires only 2 bytes for it.
So "^" is shited to right 5 bytes because there is 5 multibyte chars.
C:\Documents and Settings\WhiteRabbit>py3k x.py
push any key....
 File "x.py", line 3
 print "あいうえお"
 ^
SyntaxError: invalid syntax
[22567 refs]
Sorry, I didn't know what PyTokenizer_RestoreEncoding really doing.
That function adjusted err_ret->offset for this encoding conversion.
So, Python2.5 can output cursor in right place. (Of course, if source
encoding is not compatible for console encoding, broken string is printed
though. Anyway, cursor is right)
C:\Documents and Settings\WhiteRabbit>py a.py
 File "a.py", line 2
 x "、「、、、ヲ、ィ、ェ"
 ^
SyntaxError: invalid syntax
[8728 refs]
I tried to fix this problem, but I'm not sure how to fix this.

History
Date	User	Action	Args
2008年03月18日 05:22:32	ocean-city	set	spambayes_score: 0.00455231 -> 0.0045523075 recipients: + ocean-city
2008年03月18日 05:22:32	ocean-city	set	spambayes_score: 0.00455231 -> 0.00455231 messageid: <1205817752.04.0.892564898323.issue2382@psf.upfronthosting.co.za>
2008年03月18日 05:22:30	ocean-city	link	issue2382 messages
2008年03月18日 05:22:29	ocean-city	create

homepage