How to test Web app for concurrency problems?

Question

Recently we found a concurrency problem in our Web app. It came out that a class in a third-party library is not thread-safe and when it is shared between multiple threads, then access to it must be somehow synchronized. This finding was a by-product of load testing of our Web application with many concurrent users.

I wonder if for finding concurrency issues I should run more focused tests in addition to load tests? Or maybe it is enough to run traditional performance tests? Or maybe I should change something in performance tests so they detect such issues better? E.g.,

• make test scenarios more complex, including more endpoints in one scenario.
• run threads with random delays to increase the chance of interesting interleavings between accessing the shared resource.

Answer 1

This is a black art. It is hard to test for concurrency problems in a methodical way. In other words, it is hard to construct functional test cases that will verify whether concurrency works correctly. We tend to find concurrency problems as a side-effect of load testing.

I think it is a good idea to combine as many kinds of operations as you can in the same test, i.e. instead of all threads doing the same thing, different threads do different things.

The time intervals during which concurrency problems occur may be very small. Factors such as network latency and thread contention will already introduce random delays. I'm not sure that adding intentional delays will matter.

• dzieciou: I like the wording "black art" :-)

Answer 2

I'm a strong believer of the approach of running performance tests on production environment complete replica as it will allow to cover the following cases:

• scalability issues
• interoperability issues
• network issues
• etc.

Once you encounter a performance problem, bottleneck, degradation additional investigation will be required to get to the bottom of the problem, identify point of failure and report or fix associated problem.

• dzieciou: Thank you, Dmitri. I miss, however, "concurrency" word in your answer. How does it answer my question? Do you mean that only on production-like environment I will be able to discover those concurrency issues that may appear in production? If I have limited budget then what aspects of prod infrastructure I should copy into my test environment?
• Dmitri T: I meant that your load scenario should be as realistic as possible. Like 20% of users are just browsing having reasonable think times, 30% are search something, 15% are checking something out, etc. I don't know specifics of you application to provide more or less exact values. Usually it's enough to simulate the peak expected load and see how it goes.