homepage

Sporadic issues are difficult to analyze on the buildbots because sometimes a test fails, but when it runs again in verbose mode... it doesn't fail anymore. Typical example:
===============================
[ 99/356/1] test_smtplib
Re-running test test_smtplib in verbose mode
(...)
testTimeoutValue (test.test_smtplib.GeneralTests) ... ok
testBasic (test.test_smtplib.DebuggingServerTests) ... test test_smtplib failed -- multiple errors occurred; run in verbose mode for details
ok
testHELP (test.test_smtplib.DebuggingServerTests) ... ok
testNOOP (test.test_smtplib.DebuggingServerTests) ... ok
testNotImplemented (test.test_smtplib.DebuggingServerTests) ... ok
(...)
----------------------------------------------------------------------
Ran 32 tests in 23.664s
OK
===============================
I propose to always run the tests in verbose mode, write the output into a buffer, but only display the output on failure (or ignore the output on success).
Attached patch simplifies regrtest.py usage: replace 4 verbose levels (no verbose option, -v, -w, -W) by only 2 (no verbose option, -v). Always display the full output in case of an error (as we used -w before): no more "multiple error occured" without output trap, no more "oh, rerunning the test doesn't fail" trap. The -v option works as before (directly write the output in verbose mode).
regrtest is much more verbose than before in case of a failure, but I prefer too much information than no information when I try to debug something. If it is too much verbose by default, I can try to use 3 verbose levels (e.g. don't display the full output if only one test failed).
If the patch is accepted, the devguide should also be patched.