Testing - How to test if you are actually testing the code?

Question

I was thinking about unit tests and code coverage, and I came up with this thought:

It is possible to have 100% code coverage and not test 100% of your code.

For example:

function myTestedFunction(){
 doSomething();
 doSomethingElse();
}

In this example, if I have a test that myTestedFunction calls doSomething, doSomethingElse(); will count as tested, even though it's not. Is there some way or library that will run your tests with each line removed? I feel like I'm describing it poorly.

I'm using Node.js at the moment, but I'm more curious if something like this exists.

So does anybody know of such a thing?

Thanks!

• Ixrec: There are certainly unit test frameworks like Google Test that will let you write a unit test calling myTestedFunction() and asserting that doSomething() should get called once, which would then would print a warning that doSomethingElse() got called and you didn't say it would be. Is that the sort of thing you're looking for?
• Jörg W Mittag: "It is possible to have 100% code coverage and not test 100% of your code." – Uhm, actually, that's not only possible but completely trivial: just delete all asserts from your tests, the tests will still execute the exact same code as before, so you will get the exact same coverage, but you will test absolutely nothing. Code coverage tells you what's executed, not what's tested. It can only tell you what's definitely not tested, namely what's not being executed.
• Awemo: Another thing to consider are the different types of coverage. Line coverage can be deceptive if not used together with the other types of code coverage like parameter value coverage which ensures that all common values have been considered. Doing so will definitely improve test quality.

Answer 1

What you're talking about is called mutation testing, and there are a number of implementations available. I've not tried either, but there are at least two javascript versions:

• grunt-mutation-testing, and
• mutandis

• Awemo: I was coincidentally just reading up on mutation testing the other day. Dave Marshall gives a very good introduction to mutation testing.
• phw: One should also take a look at the Stryker mutation testing library.