I would like to add support for mutation testing in the Guile test-driver. The big idea is to introduce programming bug, that is something a developer could introduced by accident. For example, ++k instead of k++. Or a - b instread of a + b. Or a = b instead of a == b.
Rust has mutant.rs for that purpose. I think it's a great feature for a test framework and we could provide it from the core Guile test-driver.
Basically, there are multiple ways of doing so. It could be at the bytecode level, or even machine instructions. But since it is possible to modify bindings at runtime in Guile, I think we could achieve mutation of program without restarting and recompiling by just using the blue/debug/advice.scm module.
One type of easy targets are predicates. All procedures in a module ending with ? is designated as a predicate. Such predicate could be rewrite to always return either #t or #f. The idea is run the test-suite to see if one test now failed. If all tests passed, this mutant is probably something we would like to investigate since our tests are not exhaustive enough.
As an example, if you rewrite the string-version=? procedure under blue/utils/version.scm to:
(define string-version=? (const #t))
you will see that all tests passed. It means we are not testing for the #f case!
I don't think we want this for beta release to end users, but we might start making something for our internal usage.
I would like to add support for [mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) in the Guile test-driver. The big idea is to introduce programming bug, that is something a developer could introduced by accident. For example, `++k` instead of `k++`. Or `a - b` instread of `a + b`. Or `a = b` instead of `a == b`.
Rust has [mutant.rs](https://mutants.rs/) for that purpose. I think it's a great feature for a test framework and we could provide it from the core Guile test-driver.
Basically, there are multiple ways of doing so. It could be at the bytecode level, or even machine instructions. But since it is possible to modify bindings at runtime in Guile, I think we could achieve mutation of program without restarting and recompiling by just using the `blue/debug/advice.scm` module.
One type of easy targets are predicates. All procedures in a module ending with `?` is designated as a predicate. Such predicate could be rewrite to always return either `#t` or `#f`. The idea is run the test-suite to see if one test now failed. If all tests passed, this mutant is probably something we would like to investigate since our tests are not exhaustive enough.
As an example, if you rewrite the `string-version=?` procedure under `blue/utils/version.scm` to:
```scheme
(define string-version=? (const #t))
```
you will see that all tests passed. It means we are not testing for the `#f` case!
I don't think we want this for beta release to end users, but we might start making something for our internal usage.