homepage

you already have that right now because the `tokenize` module is exposed. (except that every change to the tokenization requires it to be implemented once in C and once in python)
it's much more frustrating when the two differ as well
I don't think all the internals of the C tokenization need to be exposed, my main goals would be:
- expose enough information to reimplement Lib/tokenize.py
- replace Lib/tokenize.py with the C tokenizer
and the reasons would be:
- eliminate the (potential) drift and complexity between the two
- get a fast tokenizer
Unlike the AST, the tokenization changes much less frequently (last major addition I can remember is the `@` operator
We can hide almost all of the details of the tokenization behind an opaque struct and getter functions