Scope of integration tests

Question

I'm programming a .NET WebApi application from services.

What is the scope of an integration test within the following schema?

enter image description here

Order creation scenario:

Order is created 
 -> stored in db 
 -> sends message to service bus 
PaymentService responds 
 -> creates payment 
 -> stores in db

Maybe the integration test for OrderService should (1) check for storing in database and sending message to service bus. I would expect the tests to be at the OrderService project (.NET project).

Maybe it should (2) test all the way to PaymentService. That changes the scope and savings of the tests considerably. I would expect the tests to be in a standalone .NET or JMeter project.

I could check the data in each service using the API. Or I could connect directly to the DB of both services and check that it is correct. Because if it's using the API, it's more like E2E testing.

• Christophe: OP's question is summarised in the end. The other questions are only possible alternatives that OP sees. I recommend therefore to leave the question open, since the focus is clear.
• Heinzi: Side note: Getting too hung up on the formal definitions of "unit tests", "integration tests", "end-to-end tests" etc. can be counter-productive. Yes, those differences are relevant for scientific research into testing methods. But for software development teams, the important thing is to write automated tests for whatever makes sense for your product and your workflow, irrespective of whether it "fits nicely" into one of those predefined "testing buckets" or not.
• philipxy: Why are images of text, code and mathematical expressions discouraged?
• Heinzi: @philipxy: To be fair, the image is neither text nor code nor a mathematical expression. It's a diagram - which is something SE markdown simply does not support at the moment.
• philipxy: @Heinzi The entire content is trivially expressible in words. (Although the poster doesn't actually give a legend by which the reader is to interpret the image.)

Answer 1

Integration tests can have different scopes and both of your examples are valid scopes for integration tests.

The aim of an integration test is to verify that a component and the real implementations of its dependencies work together in a correct way.

It is up to you and your team to decide if the PaymentService is a dependency of the OrderService or not. In this decision, you should take into account the risk that the message sent by the OrderService gets changed in a way that it breaks the PaymentService (i.e., an order can be created without an associated payment).

If you decide to test the interaction between OrderService and PaymentService, then the next question is if you want to explicitly test the interaction with the Service Bus or if you leave that implicit (due to the Service Bus being used in the communication between the services). This again should be a risk analysis based on what errors the explicit test can find that cannot be (conveniently) found with other tests.

• Christophe: I fully agree with this answer. But I also observe that in complex scenarios, some dependencies might be hidden and not obvious. So someone should do an end-to-end test to assure the quality of the system as a whole ;-)
• Petr Klekner: I understand. Thank you for your reply. And when these integration tests should run, do they check the Order and Payment service afterwards? I.e. do they really look into the database etc.? I.e. do I really need to have a separate project for this type of tests and is there actually a connection to everything in it? Or should it be checked through the api?
• Bart van Ingen Schenau: @PetrKlekner, if you want to verify the correct information has been written to the database, then the test code should indeed read out the database independently. If you don't care as much about the exact storage format, you could also use a reporting or read feature from the services (after a restart to clear any RAM caches) to see what they stored.

Answer 2

Picking the right scope and granularity of automated tests is a complex and opinionated problem.

The trade-offs are generally between the speed and stability of tests, and how trust-worthy the tests are. For option 1, I would expect the tests to be fast. But they wouldn't tell you if the database and collaboration between components is correct. For option 2, the tests would give you confidence that the application works as a whole, but these tests would be slow and fragile.

This is why testing pyramid exists. Most of your tests should be fast, test only in-process and limit their impact. Even better is if you can architect your solution so those tests can represent meaningful business behavior. After that, you should have some tests that verify that pieces of your application can interact with outside of the application. Databases and external services are best to be tested like this. At the top, have a few end-to-end tests to ensure your critical application flows work correctly across the whole application and its external services. But don't expect those tests to cover edge-cases.

• Petr Klekner: Actually, that's exactly how we designed it. But now that I'm writing them, I'm so confused. As you write, we have unit tests that test each method, mocking, etc. Then we have class integration tests that really test saving to the database, pulling data from file storage, etc. But we don't test calling our services via the service bus and what happens next in the Payment service. At the end, of course, we have E2E tests and those test our whole scenario. However, I started reading about integration tests and I don't really know where the scope should be.
• Euphoric: @PetrKlekner The design of your services must be such that each service handles a meaningful piece of buiness behavior and it should be possible to test this behavior without access to other services. If you need other services, then the separation between thsoe services is wrong.
• Petr Klekner: make sense. Which we do. Services don't affect us. Then it's actually true that this way the tests are probably fine.

Answer 3

So what is the scope of the integration tests?

I'm personally a fan of how Google defines the scope of tests (though I haven't worked for them). They define test sizes as "small", "medium", and "large":

in brief, small tests run in a single process, medium tests run on a single machine, and large tests run wherever they want

[...]

We make this distinction, as opposed to the more traditional "unit" or "integration," because the most important qualities we want from our test suite are speed and determinism, regardless of the scope of the test. Small tests, regardless of the scope, are almost always faster and more deterministic than tests that involve more infrastructure or consume more resources. Placing restrictions on small tests makes speed and determinism much easier to achieve. As test sizes grow, many of the restrictions are relaxed. Medium tests have more flexibility but also more risk of nondeterminism. Larger tests are saved for only the most complex and difficult testing scenarios. Let’s take a closer look at the exact constraints imposed on each type of test.

Here is a convenient table from their blog post:

Feature	Small	Medium	Large
Network access	No	localhost only	Yes
Database	No	Yes	Yes
File system access	No	Yes	Yes
Use external systems	No	Discouraged	Yes
Multiple threads	No	Yes	Yes
Sleep statements	No	Yes	Yes
System properties	No	Yes	Yes
Time limit (seconds)	60	300	900+

So how would I check the data in each service? Using the API? Or would I connect directly to the db of both services and check that it is correct? Because if it's using the API, it's more like E2E testing to me.

Because specifying the scope of a test is a small part of your testing strategy in general, I personally found reading the whole of "Testing Overview" to be very helpful in having a more comprehensive understanding of developing a testing strategy, such as "what to test?" and "how to test?". It may not all directly apply, but is very informative and can help you make decisions about how you want to do testing for your project.

Based on what you've described in your post, if your API, DB, and service bus can all be accessed locally but still relies on multiple processes, then it would be considered a "medium" sized test.

If you limit your testing to just individual services and fake the DB/networking/service bus, etc. then it would turn into a "small" test making it more deterministic.

Or if you set up separate instances for the DB, API, etc. then it can turn into a "large" test because now you are dependent on other variables such as firewall rules, networking reliability, etc. which makes it less deterministic.