What are the Qualities of a Good Test?
What is a good test? How do we know if we're writing good tests?
Kent Beck posited, that tests should be:
- Isolated (unaffected by the presence, absence, or results of other tests)
- Quick to write
- Quick to run
- Unique (providing confidence not provided by other tests/uncorrelated
with other tests)
Roy Osherove adds that good tests have three fundamental properties: maintainable, trustworthy and readable.
Mike Hill has much longer list:
- It is short, typically under a dozen lines of code.
- It does not test the object inside the running app, but instead in a purpose-built testing application.
- It invokes only a tiny portion of the code, most usually a single branch of a single function.
- It is written gray-box, i.e. it reads as if it were black-box, but sometimes takes advantage of white-box knowledge. (Typically a critical factor in avoiding combinatoric issues.)
- It is coded to the same standard as shipping code, i.e. the team's best current understanding of coding excellence.
- In combination with all other microtests of an app, it serves as a 'gateway-to-commit'. That is, a developer is encouraged to commit anytime all microtests run green, and discouraged (strongly, even nastily) to commit otherwise.
- It takes complete control of the object-under-test and is therefore self-contained, i.e. running with no dependencies on anything other than the testing code and its dependency graph.
- It runs in an extremely short time.
- It is generally written before the code-change it is meant to test.
- It avoids most or all usage of 'awkward' collaborators via a variety of slip-and-fake techniques.
Mike and Ron Jeffries remind us that the key value of TDD is simplifying design and improving productivity. Improvements in code quality and a reduction in bugs are an important side effect.
Jeremy Miller adds that good unit tests should be:
- Order Independent and Isolated - it should be possible to run the tests in whatever order the test runner chooses.
- Intention Revealing - the best unit tests make it clear to the reader how an objects API is intended to be used.
- Easy to Setup
Finally Ed Burnette writes: Make your unit tests repeatable in every aspect; Test your boundary conditions and Always keep your tests passing at 100%.
Kevin E. Schlabach
Since testing is more than TDD or other automated testing approaches, part of quality in testing is tied to when you run the tests.
And the test shouldn't suddenly appear green the moment right before a sprint review.
Re: What the qualities of proper English?
Two more requirements
2. If logic is changed, only few (ideal: one) tests fail.
(1) provides the security net for refactoring.
(2) ensures that the security net will not hinder refactorings since you have to modify hundereds or thousands of tests for one logic change.
Jester uses the approach (1) to find missing test cases.
Brandon Holt, Preston Briggs, Luis Ceze, Mark Oskin May 21, 2015