15

Our organization is considering integrating unit testing into our software development workflow. I've heard lots of anecdotal stories about how it encourages better, easy to maintain, and well-planned code. As a programmer, I understand the benefits.

However, in order to build consensus around using unit testing in projects that might affect customers, I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.

Does anyone in the community have any historical data or experience they would be willing to share as quantitative evidence in my case for unit testing?

asked Sep 7, 2011 at 23:10
5
  • 1
    I'm not 100% sure you will find what you are looking for. The act of performing unit testing doesn't necessarily save any developer time, as it takes time to write the tests and then fix defects found. However, what you will find is data relating to the cost savings of finding and fixing defects early and improved product quality (and customer satisfaction). Unit testing is just one step a series of processes designed to improve product quality. I think you should refocus your efforts - sell product quality, with unit testing as a single tool among many to begin to move toward that goal. Commented Sep 7, 2011 at 23:40
  • 1
    "I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.". And then, everyone will simply dispute your numbers. This is always doomed from the start. Are you open to suggestions for techniques that work? Or are you stuck on arguing about the numbers instead of doing something? Commented Sep 8, 2011 at 1:04
  • 1
    All of these are duplicates: programmers.stackexchange.com/search?q=justify+unit+testing. What's wrong with all of the other questions just like yours? Commented Sep 8, 2011 at 1:06
  • Historical data? What about 3 days of searching to find a hard to locate bug that was introduced by a simple "no-risk change"? A bug that would have been caught by a very simple unit test? A unit test that would have taken at most 5 minutes to code? Commented Sep 8, 2011 at 6:51
  • How do you quantify leaving out features because you're afraid to break the application? Commented Sep 8, 2011 at 12:10

4 Answers 4

12

Unit testing is cheap.

Each time you run a unit test, you didn't have to pay a tester to perform that test manually, or perform the test yourself, by hand. A computer can perform the same tests over and over tirelessy, with perfect fidelity (executing the test in exactly the same way every time).

Each time an automated test catches a bug you missed during refactoring, you save money by preventing QA people from having to report it, and untold money and reputation capital by preventing that bug from going out the door to a customer.

Automated testing, especially unit testing, saves money because it catches bugs early. The later a bug is found, the more expensive it is (by orders of magnitude):

enter image description here

answered Sep 7, 2011 at 23:12
9
  • Thanks, Robert. I'm certainly in agreement, but it's going to be hard to sell this argument without some sort of quantitative evidence to back it up. Commented Sep 7, 2011 at 23:15
  • Where's this chart from? Commented Sep 7, 2011 at 23:21
  • 1
    @Aaronaught: jonkruger.com/blog/2008/11/20/… Commented Sep 7, 2011 at 23:23
  • 2
    Huh... doesn't seem to be based on any actual data. Not that I disagree with the general idea but it'd be nice to see something a bit more material. Commented Sep 7, 2011 at 23:26
  • @Aaronaught: It is from a Google image search for "Cost of Unit Testing." There are a bazillion such graphs; they all look the same. The original was probably from "The Mythical Man Month," but I don't have a copy in front of me right now to verify that. Commented Sep 7, 2011 at 23:29
10

Here is a chart from my experiences with and without unit testing. How many of the problems on the right-hand column apply to your project?

 With Unit Tests Without Unit Tests
----------------- ----------------------------- -----------------------------
Development Code/Test/Refactor Cut and Paste/Hack 
Process or Test/Code/Refactor to minimize code change 
 and avoid complete retest
Module Coupling Limited by continuous Tangled due to need to
 refactoring add features with minimal
 code change
Program Design Evolving through Refactoring Frozen due to risk of change
Velocity Increasing as higher-level Decreasing as code volume and
over Time functions are factored out coupling increase 
Bugs detected Mostly by developer Mostly by QA or Customer
Cost of bugs Lower Higher

You should be able to point to specific instances where the code volume was increased unnecessarily, or the module coupling was increased to minimize the need for retest. However, if they don't believe they have a problem, and are not interested in improvement, there may be little you can do. Even when continued maintenance becomes uneconomical, they may believe that is simply the normal end of all software projects.

I have been successful in improving testing only when management knew they should be testing, but weren't sure how to do it economically. When an organization is uninterested in process improvement, great patience is needed to effect change. How patient are you?

answered Sep 8, 2011 at 0:22
5

I once did a statistic over a number of versions and various modules of a large system, measuring basically the number of unit test LOC per functional code LOC, plotted against the number of bugs found during manual testing per functional LOC. The result looked something like this:

More Bugs
^
|
| +
|+ + + +
| + + + + 
| + + + + + + 
|+ + + + + + + +
| + + + + + +
---------------------------------------> More Unit tests

A linear interpolation said that there were nearly 50% less bugs in the code with the most unit tests, compared to that with none. But as you see, there is a large variance, most likely related to complexity of the modules, experience and competence of the developers and maturity of the code.

answered Sep 8, 2011 at 7:32
3

Unit testing is invaluable.

I have many experiences with this.

When we first didn't adopt Unit testing, we spent the majority of the time solving bugs and dealing with mad customers because things didn't work as expected. Debugging code when you have no idea of where the bug may be is a hassle, especially because you first have to reproduce it in order to solve it.

When we decided it was time to start working with UT, it all changed. We got no more calls from mad customers at all. It may sound excessive but it went exactly like this.

With UT you find the bugs before your customers or QA. This is the big deal.

answered Sep 8, 2011 at 7:07

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.