[Python-Dev] Is it possible to view tokenizer output?

Hi, I'm just getting into the CPython codebase just for fun, and I've 
just started messing around with the tokenizer and the grammar. I was 
wondering, is there a way to just print out the results of the tokenizer 
(as in just the stream of tokens it generates) in a human readable 
format? It would be really helpful for debugging. Hope the question's 
not too basic.
Cheers.
_______________________________________________
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/2ZTZBAN5H2ET2IB7EXTKD27R5T6QVHZB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Is it possible to view tokenizer output?

Reply via email to