Message149815
| Author |
meador.inge |
| Recipients |
akuchling, docs@python, eric.snow, ezio.melotti, gpolo, meador.inge, ncoghlan, terry.reedy |
| Date |
2011年12月19日.04:50:20 |
| SpamBayes Score |
7.2466025e-07 |
| Marked as misclassified |
No |
| Message-id |
<1324270283.73.0.22714523648.issue2134@psf.upfronthosting.co.za> |
| In-reply-to |
| Content |
The proposed documentation text seems too complicated and language expert speaky to me. We should try to link to standard definitions when possible to reduce the text here. For example, I believe the "Operators" and "Delimiters" tokens in the "Lexical Analysis" section of the docs (http://docs.python.org/dev/reference/lexical_analysis.html#operators) are exactly what we are trying to describe when referencing "literal tokens" and "affected tokens".
I like Nick's idea to introduce a new attribute for the exact type, while keeping the tuple structure itself backwards compatible. Attached is a patch for 3.3 that updates the docs, adds exact_type, adds new unit tests, and adds a new CLI option for displaying token names using the exact type.
An example of the new CLI option is:
$ echo '1+2**4' | ./python -m tokenize
1,0-1,1: NUMBER '1'
1,1-1,2: OP '+'
1,2-1,3: NUMBER '2'
1,3-1,5: OP '**'
1,5-1,6: NUMBER '4'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
$ echo '1+2**4' | ./python -m tokenize -e
1,0-1,1: NUMBER '1'
1,1-1,2: PLUS '+'
1,2-1,3: NUMBER '2'
1,3-1,5: DOUBLESTAR '**'
1,5-1,6: NUMBER '4'
1,6-1,7: NEWLINE '\n'
2,0-2,0: ENDMARKER '' |
|