Message140055
| Author |
vstinner |
| Recipients |
Devin Jeanpierre, eric.araujo, petri.lehtinen, terry.reedy, vstinner |
| Date |
2011年07月09日.09:03:45 |
| SpamBayes Score |
3.0752268e-05 |
| Marked as misclassified |
No |
| Message-id |
<1310202226.54.0.882751009246.issue12486@psf.upfronthosting.co.za> |
| In-reply-to |
| Content |
The compiler has a PyCF_SOURCE_IS_UTF8 flag: see compile() builtin. The parser has a flag to ignore the coding cookie: PyPARSE_IGNORE_COOKIE.
Patch tokenize to support Unicode is simple: use PyCF_SOURCE_IS_UTF8 and/or PyPARSE_IGNORE_COOKIE flags and encode the strings to UTF-8.
Rewrite the parser to work directly on Unicode is much more complex and I don't think that we need that. |
|