InfoQ Homepage News Opinion: Code Coverage Stats Misleading
Opinion: Code Coverage Stats Misleading
Aug 30, 2006 1 min read
Write for InfoQ
Feed your curiosity. Help 550k+ globalsenior developers
each month stay ahead.Get in touch
John Casey,a key contributer to the Apache Software Foundation's Maven Project, recently spent some time refactoring Maven's assembly plugin. He thought he'd use coverage reporting to mark his testing progress, and to make sure he didn't break anything as he went. Well, at very least it was a learning experience.
He constructed a completely new suite of tests to focus on small, specific units of the new plugin's implementation. His refactoring went well, as did the testing. He had nearly perfect coverage numbers on the parts he felt were critical and felt fairly confident that this plugin would almost drop in as a replacement of the old incarnation. That's when things started to fall apart.
You can read the details on Casey's blog, along with his reasons for saying: when you're seeking confidence through testing, perhaps the worst thing you can do is to look at a test coverage report.
His conclusion: coverage reporting is dangerous. It has a tendency to distract from the use cases that should drive the software development process. It also tends to give the impression that when the code is covered, it is tested and ready for real life. However, it won't tell you that you've forgotten to test the contract on null handling. It won't tell you that you were supposed to check that String for a leading slash, and trim it off if necessary. To achieve real confidence in your tests, you would have to achieve multiple coverage for most of your code, in order to test each line under various conditions... And, since there is no hard-and-fast rule about how many test passes it will take to test any given line of code adequately, the concept of a coverage report is itself fatally flawed.
He constructed a completely new suite of tests to focus on small, specific units of the new plugin's implementation. His refactoring went well, as did the testing. He had nearly perfect coverage numbers on the parts he felt were critical and felt fairly confident that this plugin would almost drop in as a replacement of the old incarnation. That's when things started to fall apart.
You can read the details on Casey's blog, along with his reasons for saying: when you're seeking confidence through testing, perhaps the worst thing you can do is to look at a test coverage report.
His conclusion: coverage reporting is dangerous. It has a tendency to distract from the use cases that should drive the software development process. It also tends to give the impression that when the code is covered, it is tested and ready for real life. However, it won't tell you that you've forgotten to test the contract on null handling. It won't tell you that you were supposed to check that String for a leading slash, and trim it off if necessary. To achieve real confidence in your tests, you would have to achieve multiple coverage for most of your code, in order to test each line under various conditions... And, since there is no hard-and-fast rule about how many test passes it will take to test any given line of code adequately, the concept of a coverage report is itself fatally flawed.
This content is in the Agile topic
Related Topics:
-
Related Editorial
-
Related Sponsors
-
Popular across InfoQ
-
Microsoft Patches Critical ASP.NET Core Vulnerability with 9.9 Severity Score
-
GitHub Expands Copilot Ecosystem with AgentHQ
-
Redis Critical Remote Code Execution Vulnerability Discovered after 13 Years
-
Monzo’s Real-Time Fraud Detection Architecture with BigQuery and Microservices
-
AWS Launches Capabilities by Region Tool
-
Architecture Should Model the World as it Really is: a Conversation with Randy Shoup
-
Related Content
The InfoQ Newsletter
A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example