homepage

2to3 crashes when run on a doctest which uses reduce(). This happens with both 2.7 and 3.2's 2to3. I have the following code in a compatibility file, but even using that it fails with the same error: 
try:
 from functools import reduce
except ImportError:
 reduce = reduce
This is the traceback produced:
Traceback (most recent call last):
 File "/usr/local/bin/2to3", line 6, in <module>
 sys.exit(main("lib2to3.fixes"))
 File "/usr/local/lib/python2.6/lib2to3/main.py", line 173, in main
 options.processes)
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 620, in refactor
 items, write, doctests_only)
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 275, in refactor
 self.refactor_file(dir_or_file, write, doctests_only)
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 661, in refactor_file
 *args, **kwargs)
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 321, in refactor_file
 output = self.refactor_docstring(input, filename)
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 500, in refactor_docstring
 indent, filename))
 File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 530, in refactor_doctest
 assert clipped == [u"\n"] * (lineno-1), clipped
AssertionError: [u'from functools import reduce\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n']

I cannot reproduce the crash on neither 2.7 nor 3.2. Can you provide more details; attach the exact python source file that crashes 2to3 and give the complete crashing 2to3 command that you're running.

Vladimir will need to confirm how to reproduce this exactly, but here is corresponding SymPy issue: http://code.google.com/p/sympy/issues/detail?id=2605.
The problem is with the sympy/ntheory/factor_.py file at https://github.com/sympy/sympy/blob/sympy-0.7.1.rc1/sympy/ntheory/factor_.py#L453 (linking to the file from our release candidate, as a workaround is likely to be pushed to master soon).
Vladimir, can you confirm that this particular version of the file reproduces the problem?

Still cannot reproduce with the linked file. Setting as pending until we hear the details from Vladimir.

Confirmed with the file Aaron linked to. I'm using "2to3-3.2 -w -n -d sympy/ntheory/factor_.py". This is what Python says about itself:
Python 3.2 (r32:88445, Jun 8 2011, 16:34:06) 
[GCC 4.5.1 20101208 [gcc-4_5-branch revision 167585]] on linux2

Attached a patch that adds a test case reproducing the issue. Hopely someone with more experience with lib2to3 can do the actual fixing...

I looked at this and understood why it's happening. I don't know exactly how to fix it though, so here's what I found out.
When a doctest appears in a docstring at line n in a file, RefactorTool.parse_block will return a tree corresponding to n - 1 newline characters followed by the code in the doctest. That tree is refactored by RefactoringTool.refactor_tree which usually returns n - 1 newline characters followed by the refactored doctest. However, for the reduce fixer, the tree returned by refactor_tree starts with from functools import reduce followed by n - 1 newline characters and then the doctest reduce line. The failing assert happens when stripping those newlines because they are expected to be at the beginning of the output while in reality they're after the import line.
So the problem is a mismatch between the expectations of the doctest machinery (refactoring code that starts with some newlines results in code that starts with the same number of newlines) and the reduce fixer which adds an import, imports are added at the beginning of the file, therefore something appears before the newlines. Other fixers could exhibit the same problem.

The test in issue12611_test.patch is still failing.