homepage

Py_CHARMASK should return a non-negative integer. As I understand it:
tokenizer.c:tok_get around line 1368:
 while (Py_ISALNUM(c) || c == '_') {
 c = tok_nextc(tok);
 }
1) tok_nextc(tok) returns EOF (correct).
2) c is an int.
3) c == -1 gets passed to Py_ISALNUM(c):
 #define Py_ISALNUM(c) (_Py_ctype_table[Py_CHARMASK(c)] & PY_CTF_ALNUM)
So either it should be enforced that only chars are passed to
Py_CHARMASK, or a cast for the EOF case is needed (but it should
be to (unsigned char)).
Sridhar, did I sum this up correctly?