[Python-Dev] Reporting unexpected import failures as test failures in regrtest.py

Wed May 31 12:33:10 CEST 2006

Some background for those not watching python-checkins:
I neglected to do "svn add" for the new functools Python module when 
converting functional->functools. The buildbots stayed green because the 
ImportError triggered by the line "import functools" in test_functools was 
treated as a TestSkipped by regrtest.py.
Georg noticed the file missing from the checkin message, but this is the 
second time I (and the buildbots) have missed a regression due to this 
behaviour. (As I recall, last time I checked in some broken code because I 
didn't notice the additional entry appearing in the list of unexpected skips 
in my local testing)
Tim Peters wrote:
> [Nick Coghlan]
>> ... (we should probably do something about that misleading ImportError ->
>> TestSkipped -> green buildbot behaviour. . . )
>> I looked at that briefly a few weeks back and gave up. Seemed the
> sanest thing was to entirely stop treating ImportError as "test
> skipped", and rewrite tests that legimately _may_ get skipped to catch
> expected ImportErrors and change them to TestSkipped themselves.
>> A bit of framework might help; e.g., a test that expects to get
> skipped due to failing imports on some platforms could define a
> module-level list bound to a conventional name containing the names of
> the modules whose import failure should be treated as TestSkipped, and
> then regrtest.py could be taught to check import errors against the
> test module's list (if any).
>> In the case du jour, test_functools.py presumably wouldn't define that
> list, so that any ImportError it raised would be correctly treated as
> test failure.

What if we appended unexpected skips to the list of bad tests so that they get 
rerun in verbose mode and the return value becomes non-zero?
 print count(len(surprise), "skip"), \
 "unexpected on", plat + ":"
 printlist(surprise)
 # Add the next line after the previous two in regrtest.py
 bad.extend(surprise)
(This happens after the count of failed tests has been printed, so we don't 
affect that output)
Cheers,
Nick.
-- 
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
---------------------------------------------------------------
 http://www.boredomandlaziness.org