homepage

Here is an example involving the unicode character MIDDLE DOT · : The line
ab·cd = 7
is valid Python 3 code and is happily accepted by the CPython interpreter. However, tokenize.py does not like it. It says that the middle-dot is an error token. Here is an example you can run to see that:
 import tokenize
 from io import BytesIO
 
 test = 'ab·cd = 7'.encode('utf-8')
 
 x = tokenize.tokenize(BytesIO(test).readline)
 for i in x: print(i)
For reference, the official definition of identifiers is: 
https://docs.python.org/3.6/reference/lexical_analysis.html#identifiers
and details about MIDDLE DOT are at
https://www.unicode.org/Public/10.0.0/ucd/PropList.txt
MIDDLE DOT has the "Other_ID_Continue" property, so I think the interpreter is behaving correctly (i.e. consistent with the documented spec), while tokenize.py is wrong.

I verified on Win10 with 3.5 (which cannot be patched) and 3.7.0b2 that ab·cd is accepted as a name and that tokenize fails as described.

I believe this may be a duplicate of issue 12486.

I think the issues are slightly different. #12486 is about the awkwardness of the API. This is about a false error after jumping through the hoops, which I think Steve B did correctly.
Following the link, the Other_ID_Continue chars are
00B7 ; Other_ID_Continue # Po MIDDLE DOT
0387 ; Other_ID_Continue # Po GREEK ANO TELEIA
1369..1371 ; Other_ID_Continue # No [9] ETHIOPIC DIGIT ONE..ETHIOPIC DIGIT NINE
19DA ; Other_ID_Continue # No NEW TAI LUE THAM DIGIT ONE
# Total code points: 12
The 2 Po chars fail, the 2 No chars work. After looking at the tokenize module, I believe the problem is the re for Name is r'\w+' and the Po chars are not seen as \w word characters.
>>> r = re.compile(r'\w+', re.U) 
>>> re.match(r, 'ab\u0387cd')
<re.Match object; span=(0, 2), match='ab'>
I don't know if the bug is a too narrow definition of \w in the re module("most characters that can be part of a word in any language, as well as numbers and the underscore") or of Name in the tokenize module.
Before patching anything, I would like to know if the 2 Po Other chars are the only 2 not matched by \w. Unless someone has done so already, at least a sample of chars from each category included in the definition of 'identifier' should be tested.

This issue and issue12486 doesn't have any common except that both are related to the tokenize module.
There are two bugs: a too narrow definition of \w in the re module (see issue12731 and issue1693050) and a too narrow definition of Name in the tokenize module.
>>> allchars = list(map(chr, range(0x110000)))
>>> start = [c for c in allchars if c.isidentifier()]
>>> cont = [c for c in allchars if ('a'+c).isidentifier()]
>>> import re, regex, unicodedata
>>> for c in regex.findall(r'\W', ''.join(start)): print('%r U+%04X %s' % (c, ord(c), unicodedata.name(c, '?')))
... 
'℘' U+2118 SCRIPT CAPITAL P
'℮' U+212E ESTIMATED SYMBOL
>>> for c in regex.findall(r'\W', ''.join(cont)): print('%r U+%04X %s' % (c, ord(c), unicodedata.name(c, '?')))
... 
'·' U+00B7 MIDDLE DOT
'·' U+0387 GREEK ANO TELEIA
'፩' U+1369 ETHIOPIC DIGIT ONE
'፪' U+136A ETHIOPIC DIGIT TWO
'፫' U+136B ETHIOPIC DIGIT THREE
'፬' U+136C ETHIOPIC DIGIT FOUR
'፭' U+136D ETHIOPIC DIGIT FIVE
'፮' U+136E ETHIOPIC DIGIT SIX
'፯' U+136F ETHIOPIC DIGIT SEVEN
'፰' U+1370 ETHIOPIC DIGIT EIGHT
'፱' U+1371 ETHIOPIC DIGIT NINE
'᧚' U+19DA NEW TAI LUE THAM DIGIT ONE
'℘' U+2118 SCRIPT CAPITAL P
'℮' U+212E ESTIMATED SYMBOL
>>> for c in re.findall(r'\W', ''.join(start)): print('%r U+%04X %s' % (c, ord(c), unicodedata.name(c, '?')))
... 
'ᢅ' U+1885 MONGOLIAN LETTER ALI GALI BALUDA
'ᢆ' U+1886 MONGOLIAN LETTER ALI GALI THREE BALUDA
'℘' U+2118 SCRIPT CAPITAL P
'℮' U+212E ESTIMATED SYMBOL
>>> for c in re.findall(r'\W', ''.join(cont)): print('%r U+%04X %s' % (c, ord(c), unicodedata.name(c, '?')))
... 
'·' U+00B7 MIDDLE DOT
'̀' U+0300 COMBINING GRAVE ACCENT
'́' U+0301 COMBINING ACUTE ACCENT
'̂' U+0302 COMBINING CIRCUMFLEX ACCENT
'̃' U+0303 COMBINING TILDE
...
[total 2177 characters]
The second bug can be solved by adding 14 more characters in the pattern for Name.
 Name = r'[\w\xb7\u0387\u1369-\u1371\u19da\u2118\u212e]+'
or
 Name = r'[\w\u2118\u212e][\w\xb7\u0387\u1369-\u1371\u19da\u2118\u212e]*'
But first the issue with \w should be resolved (if we don't want to add 2177 characters).
The other solution is implementing property support in re (issue12734).

#24194 is about tokenize failing, including on middle dot. There is another tokenize name issue, already closed. I referenced Serhiy's analysis there and on the two \w issues, and closed one of them.