homepage

Sporadic issues are difficult to analyze on the buildbots because sometimes a test fails, but when it runs again in verbose mode... it doesn't fail anymore. Typical example:
===============================
[ 99/356/1] test_smtplib
Re-running test test_smtplib in verbose mode
(...)
testTimeoutValue (test.test_smtplib.GeneralTests) ... ok
testBasic (test.test_smtplib.DebuggingServerTests) ... test test_smtplib failed -- multiple errors occurred; run in verbose mode for details
ok
testHELP (test.test_smtplib.DebuggingServerTests) ... ok
testNOOP (test.test_smtplib.DebuggingServerTests) ... ok
testNotImplemented (test.test_smtplib.DebuggingServerTests) ... ok
(...)
----------------------------------------------------------------------
Ran 32 tests in 23.664s
OK
===============================
I propose to always run the tests in verbose mode, write the output into a buffer, but only display the output on failure (or ignore the output on success).
Attached patch simplifies regrtest.py usage: replace 4 verbose levels (no verbose option, -v, -w, -W) by only 2 (no verbose option, -v). Always display the full output in case of an error (as we used -w before): no more "multiple error occured" without output trap, no more "oh, rerunning the test doesn't fail" trap. The -v option works as before (directly write the output in verbose mode).
regrtest is much more verbose than before in case of a failure, but I prefer too much information than no information when I try to debug something. If it is too much verbose by default, I can try to use 3 verbose levels (e.g. don't display the full output if only one test failed).
If the patch is accepted, the devguide should also be patched.

> Typical example: (... smtplib ...)
Another example (yesterday):
------
[355/356/2] test_subprocess
...
Re-running test test_subprocess in verbose mode
...
Ran 228 tests in 322.313s
OK (skipped=20)
------

New patch updated according to bitdancer's comment on IRC: only change the -W option. With the patch, -W only runs the test once but captures the output.
The main difference with my patch is that all output is written to stderr, even if the captured output was written into stdout. I don't want to capture stdout and stderr in two differents stream, I prefer to keep the original output order. In case of a failure, the output only concerns a failure.

New changeset 949e0c0f5a38 by Victor Stinner in branch '3.2':
Issue #12400: regrtest -W doesn't rerun the tests twice anymore, but captures
http://hg.python.org/cpython/rev/949e0c0f5a38
New changeset 0172d40fdcd4 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest -W doesn't rerun the tests twice anymore,
http://hg.python.org/cpython/rev/0172d40fdcd4 

New changeset af799df478c6 by Victor Stinner in branch 'default':
Issue #12400: oops, remove debug code...
http://hg.python.org/cpython/rev/af799df478c6 

New changeset 69e102bfed2b by Victor Stinner in branch '3.2':
Issue #12400: remove unused variable
http://hg.python.org/cpython/rev/69e102bfed2b
New changeset 9d53612333c7 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: remove unused import
http://hg.python.org/cpython/rev/9d53612333c7 

I will not backport the change in Python 2.7, because regrtest already captures stdout (but not in verbose mode), and it checks that the output is empty.

New changeset 80c7d63325f6 by Victor Stinner in branch 'default':
Issue #12400: fix test_faulthandler if regrtest captures sys.stderr
http://hg.python.org/cpython/rev/80c7d63325f6 

New changeset 74c6afbb524a by Victor Stinner in branch '3.2':
Issue #12400: don't use sys.stderr in test_kqueue because it may be replaced by
http://hg.python.org/cpython/rev/74c6afbb524a
New changeset c0afc8d00c0a by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: don't use sys.stderr in test_kqueue because it may be
http://hg.python.org/cpython/rev/c0afc8d00c0a 

New changeset 6c54c334ea7a by Victor Stinner in branch '3.2':
Issue #12400: runtest() reuses the same io.StringIO instance for all calls
http://hg.python.org/cpython/rev/6c54c334ea7a
New changeset 8897c755633b by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: runtest() reuses the same io.StringIO instance for
http://hg.python.org/cpython/rev/8897c755633b 

New changeset 2d07b1b4466a by Victor Stinner in branch '3.2':
Issue #12400: regrtest, force verbose mode to True with option -W
http://hg.python.org/cpython/rev/2d07b1b4466a
New changeset 561974b760eb by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest, force verbose mode to True with option -W
http://hg.python.org/cpython/rev/561974b760eb 

New changeset d8ba59de860a by Victor Stinner in branch '3.2':
Issue #12400: test.support.run_doctest() doesn't change sys.stdout anymore
http://hg.python.org/cpython/rev/d8ba59de860a
New changeset bb7c9308c846 by Victor Stinner in branch 'default':
Issue #12400: test.support.run_doctest() doesn't change sys.stdout anymore
http://hg.python.org/cpython/rev/bb7c9308c846 

Hmm. I thought Antoine got rid of the output checking when he added -j support. Well, he was the one who did it, anyway, so there may be a change set you could backport to get rid of the output checking if you want to backport this to 2.7.

New changeset fdb98a7ef9be by Victor Stinner in branch '3.2':
Issue #12400: Add missing import (os) to test_kqueue
http://hg.python.org/cpython/rev/fdb98a7ef9be
New changeset 37877d19c2c9 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: Add missing import (os) to test_kqueue
http://hg.python.org/cpython/rev/37877d19c2c9 

New changeset 7bd23115b6c4 by Victor Stinner in branch '3.2':
Issue #12400: runtest() truncates the StringIO stream before a new test
http://hg.python.org/cpython/rev/7bd23115b6c4
New changeset fc831c49216d by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: runtest() truncates the StringIO stream before a new
http://hg.python.org/cpython/rev/fc831c49216d 

One more time, commit messages using the present tense are ambiguous: when you write "runtest() truncates the StringIO stream before a new test", it’s not clear at all whether you describe the previous, incorrect behavior or the new, fixed one. These phrasings are better:
"X used to do Y" (old behavior)
"X now does Z" (new)
"make X do Z" (new)

> One more time, commit messages using the present tense are ambiguous:
> when you write "runtest() truncates the StringIO stream before a new
> test", it’s not clear at all whether you describe the previous,
> incorrect behavior or the new, fixed one.
You can guess by reading the patch.

It’s irrelevant, please provide unambiguous commit message. This was discussed two or three times already.

New changeset bc2f3fbda7e3 by Victor Stinner in branch '3.2':
Issue #12400: test_zipimport_support doesn't restore original sys.stdout
http://hg.python.org/cpython/rev/bc2f3fbda7e3
New changeset ff5abf93db80 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: test_zipimport_support doesn't restore original
http://hg.python.org/cpython/rev/ff5abf93db80 

Eric is right. I don't see the patch when reading "hg log" output, or looking up revisions given by "hg annotate".

> Eric is right. I don't see the patch when reading "hg log" output
I'm using hg log -p.
> or looking up revisions given by "hg annotate".
hg annotate doesn't display the changelog, only the commit number or
hash, which option do you use?

New changeset 450209efe272 by Victor Stinner in branch '3.2':
Issue #12400: regrtest.runtest() uses stream.seek(0) before .truncate()
http://hg.python.org/cpython/rev/450209efe272
New changeset 3ce22f1b9540 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest.runtest() uses stream.seek(0) before .truncate()
http://hg.python.org/cpython/rev/3ce22f1b9540 

> Eric is right. I don't see the patch when reading "hg log" output
... or when looking at the issue tracker. Yes, one can probably figure out the intent by looking at the whole patch but it's kind of rude to force people to do that. That's what the commit message is there for: to give a brief, unambiguous summary of what the change is about.

New changeset c7bed90e4f31 by Victor Stinner in branch 'default':
Issue #12400: test_faulthandler now uses sys.__stderr__
http://hg.python.org/cpython/rev/c7bed90e4f31 

New changeset 6edb7c153105 by Victor Stinner in branch 'default':
Issue #12400: test_cprofile now restores correctly the previous sys.stderr
http://hg.python.org/cpython/rev/6edb7c153105 

Ok, no more regression / bug related to this issue, let close it.