Which test execution framework for Java has the highest test coverage?

Question

I found TestNG failing occasionally and when I started to investigate I found it has little test coverage. It looks like the test framework itself is poorly tested.

Which other test execution frameworks for Java are better tested with unit tests?

• FDM: Where exactly does TestNG go wrong? Next best known alternative springing to mind is JUnit.
• dzieciou: @FDM E.g.github.com/cbeust/testng/issues/452#issuecomment-63526282
• dzieciou: @FDM what's the test/code coverage for JUnit?
• corsiKa: Correct me if I'm wrong, but this question is all about "eating your own dogfood" right? What test-frameworks are, themselves, actually tested and how much so, right?
• dzieciou: @corsiKa. Yes, initially it was my way to relieve frustration after debugging TestNG, RestAssured and other test frameworks code. They contain bugs, often they are open source code created by passionate programmers and testers. I was trying to understand how they are tested and which one I can trust.

Answer 1

I have never used it but I would be surprised to find out that the popular test framework TestNG is actually the source of your intermittent results.

That being said, if you want to use a different one I would recommend JUnit. The interface is similar to TestNG (Annotation based) and I've never had a problem with it.

• dzieciou: This does not answer the question. The question was about which test framework is better tested.
• dzieciou: If if you don't believe that TestNG and test frameworks in general contain bugs, check issues reported for TestNG here: github.com/cbeust/testng/issues.

Answer 2

When talking about unit testing with java, then no one beats JUnit.

JUnit is the awesome unit testing framework and widely used in software companies. As compare to TestNG, JUnit is lightweight and faster. But almost same as TestNG.

TestNG itself is extension of JUnit, and it has just added new features to JUnit. But much powerful then Junit.

I will suggest to go for JUnit, but still it is as same as TestNG.

Another framework combination can be using JTest with JUnit.

Jtest complements and extends JUnit. Jtest not only runs JUnit test cases, but also automatically designs and executes additional test cases that verify the code and increase test coverage. Moreover, Jtest automatically creates JUnit test class templates into which you can easily enter test cases, and exports Jtest test cases in JUnit-compatible format. It even enforces best practices for JUnit test classes.

• corsiKa: By "no one beats JUnit", in terms of the question, this would imply that when JUnit is built that it goes through a series of unit tests and if what you say is true that "no one beats JUnit" that those tests have 100% code coverage and that none of the tests ever fail on the production build. Is this your claim? Because that's what the question appears to be asking.
• Jeevan Bhushetty: @corsiKa - my answer was for "Which other test execution frameworks for Java are better tested with unit tests?"
• dzieciou: To be precisely, TestNG introduced many features before JUnit 3 had them. Then they were added in JUnit4, so functionally JUnit and TestNG are very comparable now.
• dzieciou: What is test/code coverage of JUnit framework itself?