homepage

>>> ¡TM£¢∞§¶•ao
 File "<stdin>", line 1
 ¡TM£¢∞§¶•ao
 ^
SyntaxError: invalid character in identifier
It looks like strlen() is used instead of number of characters in the decoded string.

I am attaching a patch that seems to fix the issue. Note that I considered fixing the problem in parsetok.c where offset is originally computed, but this is part of pgen which has to be compiled without unicode support.
The test case suitable to be included in unittests is:
try:
 eval(b'\xc2\xa1'.decode('utf-8'))
except SyntaxError as err:
 assert(err.offset == 1)

See also #2382: I wrote patches two years ago for this issue.

haypo> See also #2382: I wrote patches two years ago for this issue.
Yes, this is the same issue. I don't want to close this as a duplicate because #2382 contains a much more ambitious set of patches. What I am trying to achieve here is similar to the adjust_offset.patch there.
I am attaching a patch that takes an alternative approach and computes the number of characters in the parser. I strongly believe that the buffer in the tokenizer always contains UTF-8 encoded text. If it is not so already, I would consider making it so by replacing a call to _PyUnicode_AsDefaultEncodedString() with a call to PyUnicode_AsUTF8String(). (if that matters)
The patch still needs unittests and possibly has some off-by-one issues, but I would like to get to an agreement that this is the right level at which the problem should be fixed first.

The latest patch at #2382 is simpler than mine, so I am closing this as duplicate.