On Mon, May 30, 2022 at 1:40 AM Eric V. Smith <[email protected]> wrote: > python -m tokenize < file-to-parse.py > > See the comment at the top of tokenize.py. IIRC, it re-implements the > tokenizer, it does not call the one used for python code.
Ah right, I would be surprised that there would be a public Python API to get the tokenizer output, since there is no public C API for that :-) I just removed <token.h> header file since it was never usable outside Python C internals: there is no public C API to just run the tokenizer and gets its output. Victor _______________________________________________ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/ Code of Conduct: http://python.org/psf/codeofconduct/