Test suite regressions

Mon Mar 19 13:47:00 GMT 2001

Per> By "the testsuite", do you mean anything besides running "make
Per> check" in the libjava build directory?
Nope. I've been running `make check-target-libjava' and then
comparing the output with previous runs using a script I have.
>> I think we should adopt a rule whereby we don't check in any change
>> that causes a test suite regression. At the very least this is
>> what we should do on the gcc3 branch.

Per> Well, that may be going too far: Sometimes a changes may break a
Per> test-case (which often tests less common uses), but allow an
Per> application to build.
If we were building Jacks I would agree. Jacks tests all kinds of
things that ordinary users will never encounter.
In general though I think the libjava test suite is small enough and
mild enough that this requirement wouldn't be a big burden. My recent
experience with hacking the compiler is that often times a regression
will point out some subtle brokenness in the change (or in some other
part of the compiler). So even if we allow regressions we must be
sure to understand why the regression occurs.
Per> I think the rule should be: no changes that causes a test suite
Per> regression *unless* it has been discussed on this list first, and
Per> the consensus is that it is worth it.
This sounds reasonable.
Tom