homepage

Unfortunately I think this will break many users of tokenize.py.
e.g. http://browsershots.googlecode.com/svn/trunk/devtools/pep8/pep8.py 
has code like:
if (token_type == tokenize.OP and text in '([' and ...):
If tokenize now returns LPAR, this code will no longer work correctly.
Tools/i18n/pygettext.py, pylint, WebWare, pyfuscate, all have similar code. So I think we can't change the API this radically. Adding a parameter to enable more precise handling of tokens, and defaulting it to off, is probably the only way to change this.