Testing for non-existence of a method in API

Question

I am designing a fluent builder pattern API that branches and allows different options based on what has already been called.

For example: Build().Foo().Bar() is valid code, but Build().Baz().Bar() should not be. This means the option Bar() should exist in the API if called after Foo(), but should not exist if Baz() has been called.

Testing for the existence is easy, since the tests won't compile if I can't call Bar() after Foo(). However, I cant figure out how to approach testing the non-existence. That is, how do I test that Bar() indeed does not exist among the callable API after Baz() has been called?

For some context: I am creating a DI framework, and the builder pattern is used for building the binding infos in a similar way to Ninject, for example: Bind<X>().To<Y>().AsSingle().FromNew().

• Ewan: an interesting problem for TDD advocates
• candied_orange: Why do you feel the need to test for "non-existence"?
• starikcetin: @candied_orange The builder will be complex in order to provide user ergonomics. For example there will be the aiblity to skip options for implicit defaults, which causes a huge inheritance chain. There are options that are incompatible with each other. In my opinion tests exists both for verifying and validating. Testing for non-existent API is part my verification, since it was also a design decision (prohibit at compile-time rather than throw at runtime). However, after calculating the amount of tests required (5k in the optimistic scenario), I decided to bite the bullet and not do this.
• candied_orange: "there will be the aiblity to skip options for implicit defaults, which causes a huge inheritance chain." My god why? If you have known good defaults just init with them and make them overwritable.
• candied_orange: "There are options that are incompatible with each other" I linked this already but maybe you didn't read it. Check out the doAlternatives() bit.

Answer 1

Reflection is the obvious way to do this - have a look at the type of object returned by Build().Baz() and examine what methods are available on it.

I would however question if this is a worthwhile test or not - if somebody does add a Bar() method to that type, there's a fair chance they'll just say "ah ha, that test is obsolete, I'll delete it" rather than "oh, I shouldn't have added that method". This may be better enforced by code review than a test.

• starikcetin: Thank you. I was hoping there would be an equivalent of static asserts from C++ somehow, but reflection makes perfect sense also. My reasoning for having these tests stems from the same exact scenario, but instead of someone thinking the tests are obsolete, I was hoping it would give the idea that the API should not allow for what they are trying to implement due to design decisions, especially if I write big bold comments above the test case to indicate why it exists, preferably with reference to API design docs.
• Philip Kendall: The trouble is that "Baz().Bar() does not exist" is not the same as "a function which provides the same functionality as Foo().Bar() does not exist; if someone creates Baz().DefinitelyNotBarIPromise(), your test passes but the consumer of the API can still do whatever you're trying to prevent them from doing. If forbidding the existence of Bar() is purely ergonomics, that might be okay.
• starikcetin: Good point. I suppose the only way to prevent that would be to do an AST search, which is not worth the effort. Instead I will rely on having all API reviewed and documented to make sure no duplicates happen and hope nothing slips through the cracks. Then it would be trivial to test for the permutations of all available API.

Answer 2

how do I test that Bar() indeed does not exist among the callable API after Baz() has been called?

With the type system. In a statically typed language you get this test for free.

Fluent builders, at least ones that can vary the methods you are allowed to call, have a name. That name is internal Domain Specific Language (iDSL). It works by returning differing types on each link in the chain. Different types will offer different methods to call. Trying to call one that doesn’t exist for this latest type is a compiler error. No need for reflection. No need for unit tests. Compiling is your test. If your IDE has it, this will even work with code completion.

iDSLs are very powerful. But they are a fair bit of work to set up. Use them to solve frequently recurring problems to offset this work.

• user2357112: Trying to write a test for "this doesn't compile" can be awkward, though.
• user2357112: But trying to integrate that into an actual test suite instead of ad-hoc testing can be a real pain.
• candied_orange: @PaŭloEbermann there is no need to write a test for everything that doesn’t exist. Which is good since infinitely more things don’t exist than do. Build().Baz().Bar() doesn’t need an explicit test since it won’t compile. You forbid it when you don’t define it.
• candied_orange: @user2357112 prefer a design that makes a mistake be a compile time error over one that makes it a run time error. Makes testing easy.
• candied_orange: @PaŭloEbermann So we should define what methods a type can’t have in a test suite rather than simply define what it can in the class. All because we don’t want the maintenance programmer adding methods where they don’t belong because we know the future better now then they will know it then. I really hope I never end up maintaining this.

Answer 3

The correct way to test for this is to achieve 100% code and branch coverage with your functional tests.

Then you know there is no extra hidden untested method that shouldn't exist

• candied_orange: For existence you get 100% code coverage when it compiles. You can't call Build().Baz().Bar() because the type Build().Baz() returns doesn't have a Bar(). The fact that Build().Foo().Bar() exists doesn't mean every type that isn't Build().Foo() should be tested to prove it doesn't have a Bar(). If it did that would make every type ridiculously coupled. You can't add or remove a method from a type without needing to update the tests for every type.
• Ewan: not sure what you are getting at here. If you have 100% coverage that shows that all methods have a test. You only write tests for actual working functionality, so ergo you dont have any Foo.Baz.Bar or any other unwanted methods
• Ewan: eg adding foo.baz.definitelynotbar brings your coverage below 100%
• Ewan: a test callBarAssumingFoo passes but doesn't give you 100% branch coverage for the not foo case
• candied_orange: it sounds like you’re trying to move how a class is defined out of the class and into the testing harness.