Write tests for unit tests in TDD?

Question

In an answer to another question, I suggested creating a randomized value for input to a specific method. In my experience this has been useful for making tests more readable and it lets you skip the "trivial phase" where you hardcode specific results in a method.

Here's the unit test code pasted from the other answer, for quick reference:

[Test]
public void FullHouseReturnsTrue()
{
 var roll = AnyFullHouse();
 Assert.That(sut.IsFullHouse(roll));
}
[Test]
public void StraightReturnsFalse()
{
 var roll = AnyStraight();
 Assert.That(sut.IsFullHouse(roll), Is.False);
}

A couple of comments in response suggested that this strategy would not work well, because the helper method would need to be tested also. In my experience, I've never had a need to test methods like this, since creating the corresponding production code also tests my test code.

Do AnyStraight() and AnyFullHouse() need to have their own unit tests? If so, how do you solve the chicken-and-egg problem that presents?

EDIT

Would I still want to create dedicated unit tests for the AnyFullHouse algorithm if I inlined the method?

[Test]
public void FullHouseReturnsTrue()
{
 var roll = listOfHardCodedFullHouseRolls[_random.Next(0, listOfHardCodedFullHouseRolls.Length)];
 Assert.That(sut.IsFullHouse(roll));
}

• Erik Dietrich: How would you effectively unit test the result of something with a non-deterministic outcome? (For what it's worth, I share your skepticism on the notion that unit test helper methods need to be tested)
• stonemetal: @ErikDietrich You would end up with something like Haskell's quick check. It is a little more involved than regular unit tests. You have to define invariants that are tested against random input. The library makes the random seed available so you can get repeatable results when necessary.
• Vincent Savard: Your unit tests would not respect the FIRST principle because they're not repeatable. There's nothing worse than your test suite failing, then running it again and everything passes.
• Vincent Savard: In general, you wouldn't test code written for your tests because they're indirectly tested in your tests. If that code doesn't work, then in theory, tests that use it shouldn't pass. For instance, in an integration test, if you create a Builder object then latter your assert that your database contains a row matching what you created with the Builder, you can assert that all the data match correctly, which ensures that your builder is correct.

Answer 1

Why randomize the values? Having helper fields of AnyStraight or AnyFullHouse with hardcoded values is just as readable, still provides the reduction in magic values, and reduces the chance of error in your randomization (as well as guaranteeing that your tests are repeatable).

In my experience, as soon as your tests are not repeatable, people make excuses as to why they fail. Actual bugs are ignored and your tests are quickly ignored for not catching the bugs. Don't start down that slope.

• Kristof Claes: I believe tallseth's motivation for randomizing the values is to test multiple valid and invalid combinations. After a while you have tested [1,1,2,2,2], [1,2,2,2,1], [1,1,3,3,3], [1,1,4,4,4], ..., [5,5,5,6,6].
• Michael Borgwardt: @Kristof Claes: yes, and Telastyn's criticism is fully valid: what do you do if suddenly one test fails, and when you run it again, it's successful?
• Kristof Claes: @MichaelBorgwardt : Maybe you can pass the roll-value to the Assert? Like Assert.IsTrue(sut.IsFullHouse(roll), roll.ToString()); with NUnit. This way you know what values caused the test to fail.
• tallseth: Certainly. One might also implement AnyFullhouse by having 3 hardcoded FullHouse values, and picking one at random. The worst case scenario is some random value hits a test case you haven't thought of, and the test failure is a valid one. The nunit RepeatAttribute has always allowed me to find such failures very quickly.
• tallseth: +1 for "people make excuses as to why they fail" though. I haven't experienced that, but I also didn't think about this problem on the team-psychology dimension.

Answer 2

Test suites like PHPUnit and JUnit, etc, are developed using the same TDD principles (and are usually tested against themselves).

Of course, this means that there is a minimal bootstrap suite required, but by taking an axiomatic approach means you can easily build up from simple assertions like "true === true" and "0 !== false".

In your case, you simply need to isolate the helpers and test them - it's not really a chicken and egg, more of an inception: you need to go deeper.

Alternatively, you could assume that the helpers are correct - but then that limits your assumed confidence in your tests, and therefore limits the value of them.

• tallseth: But it isn't a test framework that is being created here. The biggest reason to make AnyFullHouse a method is so I can give it an explanatory name. if it was inlined, would people react the same way?
• HorusKol: Well, it kind of is a framework - you are putting in helper methods and extending the framework to your needs. If you don't confirm those helper methods independently, then you can't be confident in the tests that use them.

Answer 3

This is an odd one. I've had to struggle with the same thought recently through some of my own tests.

While you really should test everything you write, i tend to have a general rule. If its to assist setup of a test, for example grouping setup methods for a interface using Moq, then no. However, everything else should be tested.

So in short, i think you would need to test those functions. Plus your generating data, which might be problematic (plus you really should know exactly what data your passing in/out while testing).