homepage

Given an input module such as
class klass(object):
 """multi line comment
 continued on this line
 """
 """single line comment"""
 """
 Another multi
 line
 comment"""
and implementing a custom ast.NodeVisitor such as
import as
class CustomVisitor(ast.NodeVisitor):
 def visit_ClassDef(self, node):
 for childNode in node.body:
 self.visit(childNode)
 def visit_Expr(self, node):
 print(node.col_offset)
 print(node.value.col_offset)
and feeding it the compiled ast from the module above
f = open('./module.py')
source = f.read()
node = ast.parse(source, mode = 'exec')
visitor = CustomVisitor()
visitor.visit(node)
should yield -1/-1 for the docstring that is the first
child node expression of the classdef body.
it will, however, yield the correct col_offset of 4/4 for
the single line docstring following the first one.
the multi line docstring following that will again
yield a -1/-1 col_offset.
It believe that this behaviour is not correct and instead
the col_offset should be 4 for both the expression node
and its str value.

Please note that, regardless of the indent level, the col_offset for multi line str expressions will always be -1.

In addition, the reported lineno will be set to the last line of the multi line string instead of the first line where parsing the parse began parsing the string.

Please see the attached patch that will resolve the issue. It also includes a test case in test_ast.py.
What the patch does is as follows:
- tok_state is extended by two fields, namely first_lineno
 and multi_line_start
- first_lineno will be set by tok_get as soon as the beginning
 of a STRING is detected and it will be set to the current line
 tok->lineno.
- multi_line_start is the beginning of the first line of a string
- in parsetok we now distinguish between STRING nodes and other
 nodes. in case of STRING nodes, we will use the values of the
 above fields for determining the actual lineno and the col_offset,
 otherwise tok->col_offset and tok->lineno will be used when
 creating the token.
The included test case ensures that the col_offset and lineno of
multi line strings is calculated correctly.

I have created a patch for Python 2.7.3 that fixes the issue for that release, too.

If this is really an 'enhancement', it will only go in 3.4. If it is a bug/behavior issue, then it should be marked as such and 2.7,3.2,3.3 selected. I have not read the doc and messages well enough to know, so I leave that to you and Benjamin.
The patch includes a test. It needs a patch to Misc/ACKS to add Carsten Klein between Reid Kleckner and Bastian Kleineidam

I left comments on Rietveld a few days ago.

Any updates on this? I'm running into this as well (still a problem in 3.4)
```$ python3.4
Python 3.4.2 (default, Oct 11 2014, 17:59:27) 
[GCC 4.4.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse("""'''foo\n'''""").body[0].value.col_offset
-1
```

What's the status on this? Anything preventing it getting fixed? Still the same in 3.6.1:
>>> import ast
>>> ast.parse("""'''foo\n'''""").body[0].value.col_offset
-1

pypy seems to have this right (though I don't know enough about their internals to know if cpython can benefit from their patch)
$ venvpypy/bin/pythonPython 2.7.10 (3260adbeba4a, Apr 19 2016, 17:42:20)
[PyPy 5.1.0 with GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import ast, astpretty
>>>> astpretty.pprint(ast.parse('"""\n"""'))
Module(
 body=[
 Expr(
 lineno=1,
 col_offset=0,
 value=Str(lineno=1, col_offset=0, s='\n'),
 ),
 ],
)

Still a problem in 3.7:
$ python3.7
Python 3.7.0b2 (default, Feb 28 2018, 06:59:18) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse("""x = '''foo\n'''""").body[-1].value
<_ast.Str object at 0x7fcde6898358>
>>> ast.parse("""x = '''foo\n'''""").body[-1].value.col_offset
-1

Should we backport this to 3.7?
AST changes including bugfix affects existing software unexpectedly.

New changeset 995d9b92979768125ced4da3a56f755bcdf80f6e by INADA Naoki (Anthony Sottile) in branch 'master':
bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)
https://github.com/python/cpython/commit/995d9b92979768125ced4da3a56f755bcdf80f6e

I agree -- probably safer to not backport to 3.7 in case someone is relying on this behaviour.

commit 995d9b92979768125ced4da3a56f755bcdf80f6e introduced a regression: bpo-37603: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py.

> commit 995d9b92979768125ced4da3a56f755bcdf80f6e introduced a regression: bpo-37603: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py.
Fixed in https://bugs.python.org/issue39209