homepage

Currently Lib/token.py is generated from Include/token.h. This contradicts common practice when the C code is generated from the Python code (see for example opcode.py and sre_constants.py). In additional the table in Parser/tokenizer.c should be manually supported matching Include/token.h.
Generating Include/token.h and Parser/tokenizer.c from Lib/token.py would be simpler and more reliable.

I like the idea.

I can work on it

I already write a patch.

PR 1860 makes following files be generated from token.py:
* Include/token.h
* Parser/token.c. New file containing the array of token names _PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), PyToken_ThreeChars(), moved from Parser/tokenizer.c.
* Doc/library/token-list.inc. New file containing the list of token.py constants, it is included in Doc/library/token.rst.
New Makefile target regen-token regenerates these files.
The dict EXACT_TOKEN_TYPES that maps operator strings to token names now is automatically generated and moved from tokenize.py to token.py. Tokens COMMENT, NL and ENCODING used only in tokenize.py now are added in token.py as in issue25324.

I think this covers all the changes from PR #1608. Looks a lot nicer too, building it every time from the make file. 
You may want to add to the docs that token.py is now the source of the tokens.

The regular expression tokenize.Funny also can be generated. Information is not enough for distinguish between Operator, Bracket and Special, but seems this isn't needed.
Some token names can be generated from Grammar/Grammar. But needed an additional mapping for relations between token strings and names ('+' <-> PLUS, etc).

Alternate PR 10370 generates all files from a single file Grammar/Tokens using a single script Tools/scripts/generate_token.py.
In addition, the script doesn't write files when the content is not changed. Thus it can be used with read-only sources.

Could anybody please make a review? There are two alternate PRs: PR 1860 and PR 10370. The difference between them is that the former one uses Lib/token.py as a source, and the latter one uses Grammar/Tokens as a source and generates Lib/token.py too.

If there are no objections I am going to merge PR 10370 in few days.

> If there are no objections I am going to merge PR 10370 in few days.
LGTM. I guess that PR 9343 should be closed once PR 10370 is merged.

New changeset 8ac658114dec4964479baecfbc439fceb40eaa79 by Serhiy Storchaka in branch 'master':
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
https://github.com/python/cpython/commit/8ac658114dec4964479baecfbc439fceb40eaa79