homepage

I've attached a patch to test_tokenizer.py and a bunch of text files 
(that should be dropped into Lib/test) that highlight this issue a 
*lot* better than the current state of affairs.
The existing implementation defines roundup() in the doctest, then 
proceeds to define it again in the code body. The last for loop in the 
doctest is failing every so often -- what it's failing on isn't at all 
clear as a) ten random files are selected out of 332 in Lib/test, and 
b) there's no way of figuring out which files are causing it to fail 
unless you hack another method into the test case to try and replicate 
what the doctest is doing, with some additional print statements (which 
is the approach I took, only to get bitten by the fact that roundup() 
was being resolved to the bogus definition that's in the code body, not 
the functional one in the doctest, which resulted in even more 
misleading behaviour).
FWIW, the file that causes the exception is test_doctest2.py as it 
contains encoded characters.
So, the approach this patch takes is to drop the 'pick ten random test 
files and untokenize/tokenize' approach and add a class that 
specifically tests for the tokenizer's compliance with PEP 0263.
I'll move on to a patch to tokenizer.py now, but this patch is ok to 
commit now -- it'll clean up the misleading errors being reported by 
the plethora of red 3.0 buildbots at the moment at the very least.