Improvements in java based test framework to reduce execution time

Question

Let me give a bit of background on the problem.

We have a java application which has MQ integration and an Integration Test pack which is also written in java.

The test pack runs all scenarios (positive and negative) by running the test cases (methods) and changing the parameters depending on the scenario. The way it does the assertions is to put a message on the input queues of the application and expects a message in the output queues of the application. The message body is checked because it differs depending on the test scenario.

There is only 1 input queue and 1 output queue in the main application. The number of test cases (methods) are about 300 and due to running all the known scenarios, the number of test cases actually executed are about 2500. Due to this, the test pack takes around 6 hours to execute fully.

Since there is only 1 queue on which the test pack asserts the output message, we have to clear the queue each time before we run the next test case. Otherwise it may lead to mixing of the output messages. For e.g. method call#2 might end up consuming output message for method call#1. Hence we can't run the test cases in parrallel too.

I want to improve this process and would be highly grateful for any suggestions to optimize the test pack. Right now I am thinking to separate out the input and output queues of the main application as per business function and thus assert on multiple queues in parallel. But this approach means I have to change the original architecture so that my Test cases run faster and this is not easy since there are already clients using this application and interacting using the 2 queues and they will have to change their applications too.

Apologies for the long post but I would love to hear inputs of so many creative people in SO community.

Thanks!

• Bart van Ingen Schenau: Can you spin up multiple instances of the application and queues on separate (virtual) machines? Then you could run tests in parallel.
• phoenixSid: Thanks, that's a valid suggestion. But it would require changes in the infrastructure to spin up so many instances of the application, which would not be easy from a finance/approval perspective of our organization.
• phoenixSid: @BartvanIngenSchenau so after trying a lot of different options, we have finally decided to try spinning up different containers. To avoid the finance/approval perspective, we would look at dockerizing the test pack to spin up these many containers. Thanks for your comment.
• Carra: For my unit tests, I mock out the Servicebus queue and replace with an in-memory queue. Runs a lot faster for testing purposes.
• phoenixSid: @Carra Could you elaborate a bit more on the in-memory queue? How exactly do you do that?

Answer 1

Filters on message queues exist. You can throw them into the same message queue, and then assign a listener for messages based on a specific criteria. This criteria can be anything, to thread id to an arbitrary message name.

The end result is that messages not having that specific criteria don't get thrown away, they merely wait to be processed by handlers without that criteria, effectively allowing you to perform checks on specific aspects pertaining to specific messages without removing messages other than the ones you wish to check.

• phoenixSid: Yes you are correct and that is one way we wanted to do this. But the problem is the test pack runs the application which doesn't use the filters on the message queue. So if we were to do it specifically in our test pack, it wouldn't mirror the real work usage. Thanks for the answer though.

Answer 2

One of the things that used to drive me nuts was when eclipse would run my tests in parallel when I hadn't expected it. This vexed me because I'm an old school tester who still thinks writing to the console is useful. Two tests would write to the console and sometimes everything was fine. Sometimes one's text would be mixed halfway into the others.

Then I realized how simple the solution was. Just let each test build it's own string and output that one string at one time. Once I did that I could let the tests run in parallel just fine.

If you don't have sequential dependencies, shared memory, or multiple outputs from each job to keep in order I really don't see why you can't let them run in parallel.

• phoenixSid: There are no sequential dependencies, but there is only one output assertion queue where all tests will send the messages. If the tests are running in parallel, test#3 might read messages from test #2 etc.
• candied_orange: You just described a sequential dependency.
• phoenixSid: no, I thought you were referring to sequential dependencies in the test methods itself for e.g. test#3 must always run after test#2. There is no such dependency.

Answer 3

Since there is only 1 queue on which the test pack asserts the output message, we have to clear the queue each time before we run the next test case. Otherwise it may lead to mixing of the output messages. For e.g. method call#2 might end up consuming output message for method call#1. Hence we can't run the test cases in parallel too.

Assign unique ids to the messages and listen for message with expected id in output queue with an upper time limit. This should get you testing parallel executions.

You can wrap this publish/listen logic in a generic request/response style implementation. From there on, use something like RxJava to compose scenarios of multiple such calls.

• phoenixSid: we already have unique ids. But the problem is, once a message is picked up from a queue it is gone. So for e.g. if test#3 consumes a message and realises it was test#2 message, nothing can be done now. test#2 will never get the message.
• candied_orange: So stop consuming messages that aren't yours.

Answer 4

So the solution we have come up with is this. We will Dockerize the test pack so that we have multiple containers running multiple servers and multiple message queues. We will utilize the maven profiles to deploy the same war but run different test packs in the separate containers. The test pack would be modularized in such a way that the tests within each module do no overlap each other in terms of the queues/messages. Thanks to everyone for their time and their valuable comments.