homepage

Given the way the documentation is written for re.VERBOSE - "Whitespace within the pattern is ignored, except when in a character class or preceded by an unescaped backslash" - I would expect all three of the findall() commands below to return successfully with the same result:
Python 3.2.3 (default, Jun 8 2012, 05:37:15) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.findall('(?x) (?: a | b ) + ', 'abaabc')
['abaab']
>>> re.findall('(?x) (? : a | b ) + ', 'abaabc')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.2/re.py", line 193, in findall
 return _compile(pattern, flags).findall(string)
 File "/usr/lib/python3.2/re.py", line 255, in _compile
 return _compile_typed(type(pattern), pattern, flags)
 File "/usr/lib/python3.2/functools.py", line 184, in wrapper
 result = user_function(*args, **kwds)
 File "/usr/lib/python3.2/re.py", line 267, in _compile_typed
 return sre_compile.compile(pattern, flags)
 File "/usr/lib/python3.2/sre_compile.py", line 491, in compile
 p = sre_parse.parse(p, flags)
 File "/usr/lib/python3.2/sre_parse.py", line 692, in parse
 p = _parse_sub(source, pattern, 0)
 File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
 itemsappend(_parse(source, state))
 File "/usr/lib/python3.2/sre_parse.py", line 627, in _parse
 raise error("unexpected end of pattern")
sre_constants.error: unexpected end of pattern
>>> re.findall('(?x) ( ?: a | b ) + ', 'abaabc')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.2/re.py", line 193, in findall
 return _compile(pattern, flags).findall(string)
 File "/usr/lib/python3.2/re.py", line 255, in _compile
 return _compile_typed(type(pattern), pattern, flags)
 File "/usr/lib/python3.2/functools.py", line 184, in wrapper
 result = user_function(*args, **kwds)
 File "/usr/lib/python3.2/re.py", line 267, in _compile_typed
 return sre_compile.compile(pattern, flags)
 File "/usr/lib/python3.2/sre_compile.py", line 491, in compile
 p = sre_parse.parse(p, flags)
 File "/usr/lib/python3.2/sre_parse.py", line 692, in parse
 p = _parse_sub(source, pattern, 0)
 File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
 itemsappend(_parse(source, state))
 File "/usr/lib/python3.2/sre_parse.py", line 640, in _parse
 p = _parse_sub(source, state)
 File "/usr/lib/python3.2/sre_parse.py", line 315, in _parse_sub
 itemsappend(_parse(source, state))
 File "/usr/lib/python3.2/sre_parse.py", line 520, in _parse
 raise error("nothing to repeat")
sre_constants.error: nothing to repeat
>>> 
The behavior is the same in Python 2.7. Apparently the scan for the special '(?' character sequences happens before the whitespace is stripped out. In my opinion, the behavior should be changed, the documentation should be more clear about the current behavior, or at least the errors given should be more informative (I spent an hour or two debugging the "nothing to repeat" error in my work yesterday.) Thank you.

Ideally, yes, that whitespace should be ignored.
The question is whether it's worth fixing the code for the small case of when there's whitespace within "tokens", such as within "(?:". Usually those who use verbose mode use whitespace as in the first example rather than the second or third examples.

Fair enough, but in that case I still think the current behavior should be documented. Attached is a possible patch. (This is my first interaction with the Python issue tracker, by the way; apologies if I ought to have set some field differently or left some other field alone.)

See also related issue11204.

See also #17184.

Steven, would you mind to update your patch according to review comments and create a pull request on GitHub?

It looks to me like there are more situations than the patch lists where whitespace still separates tokens. For example, *? is a reluctant quantifier and * ? is a syntax error, even in verbose mode.

Steven's patch is outdated since 71a0b43854164b6ada0026d90f241c987b54d019. But that commit missed that spaces are not ignored within tokens. PR 4366 fixes this by using the wording from Ezio's comments.

New changeset b0b44b4b3337297007f5ef87220a75df204399f8 by Serhiy Storchaka in branch 'master':
bpo-15606: Improve the re.VERBOSE documentation. (#4366)
https://github.com/python/cpython/commit/b0b44b4b3337297007f5ef87220a75df204399f8

New changeset 14c1fe682f0086ec28f24fee9bf1c85d80507ee5 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-15606: Improve the re.VERBOSE documentation. (GH-4366) (#4394)
https://github.com/python/cpython/commit/14c1fe682f0086ec28f24fee9bf1c85d80507ee5

New changeset a2f1be0b5ba2bed49b7f94c026b541ff07e52518 by Serhiy Storchaka (Miss Islington (bot)) in branch '2.7':
bpo-15606: Improve the re.VERBOSE documentation. (GH-4366) (#4395)
https://github.com/python/cpython/commit/a2f1be0b5ba2bed49b7f94c026b541ff07e52518