Is it acceptable to accept multiple results in a unit test?

Question

I've noticed this while implementing a Rest-API today. We first define interfaces for each API method and the api contract for one endpoint allows (amongst others) two status codes: 200 and 204. Both may be returned when the request was successful, but the returned data isn't really important. The implementation might choose to return 204 and no additional data or a 200 if some additional, not really relevant data is returned (might be a string for debugging purposes like "entity 123 persisted").

Now, I'm implementing the API and know I want the implementation to always return such a text. But this might change in the future for whatever reason, so I thought I'll test the result like that (this is Java code but should be easy enough to follow):

assertThat(response.getStatus(), anyOf(is(200), is(204));

Is it acceptable to test my implementation like this, just because the interface allows both values for the same case (and it's not really important)? Or should I stay close to the actual implementation here (and change the test later, if I change the implementation)

• Dan Wilson: Why not simply test for a 200-level status? assertTrue(response.getStatus().toString().startsWith("2"))
• Ewan: if its not important, don't test it. If it is important then I think you need to check the content of the response is correct and the code matches
• looper: @DanWilson because not every 200-level status is documented as valid and thus is allowed to be returned

Answer 1

I prefer to test the interface, not the implementation.

If your consumers have access to your code, tests are a form of documentation that is guaranteed to be up to date. If you only test for 200, consumers may come to rely on that, and changing to 204 (always or sometimes) is more likely to be a breaking change.

From a more selfish point of view, tests should be minimally specific (without compromising on the "guarantee" of correctness) because this makes them less brittle and easier to maintain.

• Nachiappan Kumarappan: I like the idea of testing of the return code is either 200 or 204. This sets the expectation for the client code. Testing the scenario of 200 and 204 separately would make the tests more brittle and reveal implementation details rather than contact

Answer 2

It is not only acceptable: it's recommended! Your tests should be according to the contract that your API specifications promises.

Benefits:

• Separation of concerns: tests and implementation can evolve independently but still tell reliably if the implementation is valid or not;
• Maintainability: You will not have to remember some unnecessary coupling that you have created in a previous version. In big projects, tests could even be written by an independent team;
• Accuracy: Writing tests to reflect the promises made by the API specs documents unambiguously how you interpret the specs (and writing tests that would not be consistent with the specs would only raise unnecessary questions from your peers).
• Reusability of the tests for validating any valid alternative implementation (e.g. if your API becomes a standard or someone else writes a replacmeent component from scratch);

Additional thoughts:

• Imagine your API would be a random generator: It would not even be possible to accept only a single value; you would have to accept the full random range. (This being said, you’d the also make statistics on a high number of returned results for verifying that the distribution of the numbers is not flawed).
• Testing in line with the contract is also more in the spirit of TDD that starts with the tests.

Answer 3

So if you get a status 200, that is correct and your test must pass. And if you get a status 204, that is also correct and your test must pass.

Let’s think hard about this...

Yes, your test must pass in both cases.

(This is ignoring the fact that a server must always be expected to return all kinds of errors due to not functioning correctly right now, which should probably not lead to "test passed" or "test failed" but "test could not be performed").

Answer 4

I think you should have two different tests.

You return a different status code based on what the value is in the database. You have defined two different behaviours: when the database is in one state, you return 200. When the database is in another state, you return 204.

But this is a unit test of your web-service - so the test should be decoupled from the database. You can and should use some form of test double (mock, stub, etc). This way the tests not only document the different possible responses, but give example scenarios of when to expect each response.

If you find you really want to connect to a database, your test specification should set the database state, i.e. delete the table(s), populate some test data for the test, then run the test. But this is not preferred - if your database is unavailable, you won't be able to test/build your API.

When you talk about changing it to always return 200, you're not talking about a refactoring that changes the implementation without changing the behaviour. That's an actual behaviour change, so it's natural that you would update the tests at this point.

Answer 5

Is it acceptable to test my implementation like this, just because the interface allows both values for the same case (and it's not really important)?

So this is a good example of your tests leading development. It is a design smell that the response from the server is not important and 200 and 204 could be returned by the same call to the server.

In the HTTP spec 200 and 204 mean different things to the client and should be used in different circumstances. Those circumstances don't overlap (ie the same call to the API should either produce a 200 or a 204 but not both) and should be known before hand, in which case you should be able to test both cases.

Your tests should be distinguishing between these different circumstances. If the tests are identical it means there is something wrong with your API (perhaps 204 is redundant and shouldn't be used at all, or vice versa)

Listen to your tests. They are telling you something is wrong.