homepage

With Python 2.7, both b'hello' and br'hello' are wrong.
With Python 3.3, b'hello' is wrong.
$ python2.7 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello'"
1,0-1,7:	STRING	"'hello'"
1,7-1,8:	OP	','
1,9-1,17:	STRING	"u'hello'"
1,17-1,18:	OP	','
1,19-1,28:	STRING	"ur'hello'"
1,28-1,29:	OP	','
1,30-1,31:	NAME	'b'
1,31-1,38:	STRING	"'hello'"
1,38-1,39:	OP	','
1,40-1,42:	NAME	'br'
1,42-1,49:	STRING	"'hello'"
1,49-1,50:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''
$ python3.3 -m tokenize <<<"'hello', u'hello', ur'hello', b'hello', br'hello', rb'hello'"
1,0-1,7: STRING "'hello'" 
1,7-1,8: OP ',' 
1,9-1,17: STRING "u'hello'" 
1,17-1,18: OP ',' 
1,19-1,28: STRING "ur'hello'" 
1,28-1,29: OP ',' 
1,30-1,31: NAME 'b' 
1,31-1,38: STRING "'hello'" 
1,38-1,39: OP ',' 
1,40-1,49: STRING "br'hello'" 
1,49-1,50: OP ',' 
1,51-1,60: STRING "rb'hello'" 
1,60-1,61: NEWLINE '\n' 
2,0-2,0: ENDMARKER ''

Here is a patch (for Python 3.3).

Here is a patch for Python 2.

And here is a better patch for Python 3.

And here is an even better patch for Python 3.

Patches updated with tests.

The Python 3 patch looks OK, except that several of the tests are duplicated. I am looking at the Python 2 patch now.

Nevermind, the tests are OK. I missed the swapped quotes.

> The Python 3 patch looks OK, except that several of the tests are duplicated.
What tests?

Python 2 patch looks OK too. I will commit these later today. Thanks for the patches!

LGTM too. Thanks.

New changeset 35d3a8ed7997 by Meador Inge in branch '2.7':
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
http://hg.python.org/cpython/rev/35d3a8ed7997
New changeset 115b0cb52c6c by Meador Inge in branch 'default':
Issue #15054: Fix incorrect tokenization of 'b' string literals.
http://hg.python.org/cpython/rev/115b0cb52c6c