-
Notifications
You must be signed in to change notification settings - Fork 1.1k
"Pending" tests of "status quo" #23467
-
It's useful to have a test that documents the status quo even if the status quo is undesirable.
For example,
#23276
I chose a random commit to see how tests/pending is used:
3125665 deletes a pos test that had a comment that it should fail and see tests/pending/neg/i5690.scala, which was moved to neg.
I don't think pending is useful if it is used to park tests without testing them. (That is, it is useful as a staging area only.)
I should be alerted if my commit changes behavior in any way, including dubious behavior. (That might be a progression or a regression.)
This question (about testing the status quo) came up again because of
tests/untried/neg/t4989.scala
which generates more errors than Scala 2, but possibly not enough errors. If it were a tried test, even without resolving any behavioral issues, at least the test would have history to demonstrate whether the behavior has changed. (History is easier than bisecting.)
Is there any argument against a test category for status-quo, or perhaps dubious behavior, to signal that a test has problematic behavior? Or an annotation that a test result is known to be dubious, which would have more force than a TODO comment.
My own experience is that particular test categories are useful only for managing how tests are run, rather than characterizing the tests. For example, tests/warn is more useful than pos or neg because it permits zero or more warnings, and not because it counts "warnings" instead of "errors". So I'm not convinced that a separate test kind is desirable, only that such tests are run somehow.
Beta Was this translation helpful? Give feedback.
All reactions
I agree, adding a status-quo is a useful addition. The convention could be that a test that should fail but that compiles should have // should error comments. Likewise a test that fails but should be OK should have // error but should be ok comments.
Replies: 3 comments
-
I agree, adding a status-quo is a useful addition. The convention could be that a test that should fail but that compiles should have // should error comments. Likewise a test that fails but should be OK should have // error but should be ok comments.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
(削除) Two (削除ここまで) (削除) Three (削除ここまで) Four other requests for when I have a free afternoon:
-- testCompilation --branch like partest --branch to run all tests on the branch
-- ignore ignored things
-- improve robustness under catastrophic failure
-- reduce the noise of No files matched "tostring" in test
No errors found when compiling neg test neg/.scala-build
[=================================> ] completed (1718/1996, 1 failed, 92s)
No errors found when compiling neg test neg/.bsp
62 suites passed, 1 failed, 63 total
tests/neg/.scala-build failed
tests/neg/.bsp failed
A test throwing stack overflow can hang:
#23503 (comment)
If that is true, I wonder if there is a way to fail better. "Bye, Vulpix! Hope you fail better soon!"
More on the summary noise, ironically I have to scroll up to see if my test was probably run. I don't know what the number 16 means, my branch has one test, I just want to make sure it ran. I have to scan for completed 1/1, which is more annoying because of checkInit having a creative configuration.
[info] Test dotty.tools.dotc.CompilationTests.checkInit started
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.posTwice started
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsNeg started
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsPos started
[=======================================>] completed (1/1, 0 failed, 0s)
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsRun started
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.checkInitGlobal started
No files matched "tostring" in test
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.fuzzyAll started
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.genericJavaSignatures started
No files matched "tostring" in test
[info] Test dotty.tools.dotc.CompilationTests.explicitNullsWarn started
No files matched "tostring" in test
================================================================================
Test Report
================================================================================
4 suites passed, 0 failed, 4 total
[info] Test run dotty.tools.dotc.CompilationTests finished: 0 failed, 0 ignored, 16 total, 10.156s
[info] Test run dotty.tools.dotc.BootstrappedOnlyCompilationTests started
[info] Test run dotty.tools.dotc.BootstrappedOnlyCompilationTests finished: 0 failed, 0 ignored, 0 total, 0.001s
[info] Test run dotty.tools.dotc.coverage.CoverageTests started
[info] Test run dotty.tools.dotc.coverage.CoverageTests finished: 0 failed, 0 ignored, 0 total, 0.0s
[info] Passed: Total 16, Failed 0, Errors 0, Passed 16
[success] Total time: 17 s, completed Jul 22, 2025, 7:06:55 AM
Beta Was this translation helpful? Give feedback.
All reactions
-
Diffs and summaries at #17784
I wasn't able to turn off the "test feedback" progress bar, which ruins debug output.
+ //val testsInteractive: Boolean = propIsNullOrTrue("dotty.tests.interactive")
+ val testsInteractive: Boolean = false // turn off progress bar that munges output
It would be nice if I could add -Vprint to my test and have it actually printed in my test output. (Like in partest.)
Beta Was this translation helpful? Give feedback.