The Unit in Unit Testing

Developers write tests to increase confidence that production code works correctly, to document intentions, and to aid in application design. Lately, we’ve seen developers trend toward heavy use of test doubles, especially mocks, in unit tests. This is done to improve the speed of tests, decrease test dependencies on infrastructure, or reduce the number of objects that a test depends on. However, it often comes at an unacceptably high cost of lower confidence, unclear documentation, and high coupling between implementation and test code.

To avoid these drawbacks, developers should consider using fake objects instead of mock objects, as fake objects offer similar isolation benefits while driving high confidence, clear documentation, and loose coupling between implementation and test code.

Background

We label lower-level tests unit tests to describe that these tests are somehow isolated from the code around them. Because of this isolation, unit tests should be fast to run, simple to write, and easy to understand and maintain.

Developers often use test doubles as a way to facilitate this isolation. A test double is any object used in a test that takes the place of a collaborator. In his book xUnit Test Patterns, Gerard Meszaros defines several categories of test doubles: dummies, spies, stubs, fakes, and mocks. For our purposes, we’ll focus on the last two:

• Mock objects are preprogrammed with specifications of the calls they expect to receive and their response to these calls. They have a mechanism to verify that they receive the correct calls during a test and will fail the tests if the calls don’t match their expectation. People often use frameworks like Mockito, Mockk, or GoMock to create mock objects.
• Fake objects are functioning implementations of collaborators that take some kind of shortcut to make them more suitable to run in a test environment. For example, a developer might create an in-memory datastore to take the place of an object that saves data to S3  to run the tests locally.

It’s common for virtually everything to be mocked in the test suite of a modern codebase and for the suite to run without any supporting services. In such a case, the test suite provides a high degree of confidence that each piece of the system works correctly in isolation but little confidence that they work correctly together. Later, we’ll discuss when mocks are inappropriate to use.

For example, many test suites mock the database layer during testing. Tests check that the right calls are made to the database and use pre-programmed responses. It’s hard for such a test suite to give us confidence that the code will behave correctly in production since the database is never exercised, and the expected calls programmed into the mocks might not be correct, not to mention that with a mock-only approach, SQL statements go untested.

Isolation

There’s general agreement that the word unit in unit testing refers to units of isolation,. That is, unit tests are isolated, at some level, from the rest of the codebase. However, opinions differ when defining what the unit is that is being isolated.

This definition is essential. The unit of isolation shapes the scope of each test, the relationship between test code and production code, and ultimately the application architecture. Historically, there have been widely accepted definitions of a unit, which we will discuss below.

Test isolation

Kent Beck, representing the classical approach to testing, argues that

“Unit tests are completely isolated from each other, creating their test fixtures from scratch each time.”

In this approach, the word unit in unit testing refers to the test itself: unit tests are isolated from other tests. Beck argues that “tests should be coupled to the behavior of code and decoupled from the structure of code.” 

Tests written using this approach tend to have few mocks. They instead use instances of collaborating objects and even real supporting infrastructure (such as databases) to support each test run. 

For example, a classical test whose subject makes database calls would use the actual database during tests. The tests would ensure the database is in the right state before running and assert that the resulting database state matches expectations.

A classical test with a subject that makes external HTTP calls would make HTTP calls when running the test. Since external calls often decrease test reliability, the author might start an HTTP server locally that behaves similarly to the external service.

Classical-style tests provide a high degree of confidence that the tested code behaves correctly. When the code is refactored, the tests tend not to change since they know little about the external interface of collaborators.

Subject isolation

Steve Freeman and Nat Pryce, representing the mockist approach to testing, argue that

“[unit tests] exercise objects, or small clusters of objects, in isolation.”

Freeman argues that unit tests are “important to help us design classes and give us confidence that they work, but they don’t say anything about whether they work together with the rest of the system.” In this approach, the word unit in unit testing refers to the subject being tested.

Tests written using this approach must use test doubles to stand in for collaborators and tend to have many mocks. They rarely use real supporting infrastructure, favoring mocks or fakes. The idea is that we should isolate test subjects from the behavior of their collaborators during tests; a change in the behavior of one object should not impact the tests of another object. Developers also use mocks to increase the speed and reliability of their tests, using mocks to take the place of slow or unreliable collaborators. 

For example, a mockist test whose subject makes database calls would mock the database layer when running the test. The subject would interact with the mock database object, recording calls as it returned canned responses during a test, and checking expectations at the end.

A mockist test with a subject that makes external HTTP calls would use a mock HTTP client when running the test. This client would return pre-programmed responses to HTTP calls during the test. After the test, the test author would use the mock to check that the right HTTP calls had been made.

These tests tend to run quickly and reliably, but they provide a lesser degree of confidence that the behavior under test functions correctly. When the code is changed or refactored, the tests tend to require significant changes since they have an intimate knowledge of the external interface of collaborators.

Additionally, using mocks increases the amount of test code. In many languages, like Go, mockist must write or generate all mocks and keep the code in their codebase. This effectively doubles the size of the test suite. Even in languages like Kotlin and Java, where mocks are generated at runtime, the mocks must be programmed before each test and verified after each test, which results in more test code to maintain.

In practice

To determine the approach to use in practice, we must first enumerate the goals we are trying to achieve by writing tests. We want to:

• Increase confidence that our code works correctly.
• Document how our code should function.
• Aid in designing loosely coupled, highly cohesive software.

With these goals in mind, I’d argue to start with the test isolation approach to unit tests. If each test runs reliably in isolation while using as many real collaborators as possible, we achieve the following.

Confidence, since our tests run in a production-like environment. We can be confident that our test subjects function correctly in isolation and in concert. Our tests also give us confidence that our subjects behave correctly in concert with their external collaborators. When using a mockist approach, we don’t have the same confidence that our test subjects work well together.

Clear documentation, since readers can see how our code should be used in the production-like environment. Developers who read our tests can simply, for example, examine the expected database state meant to result from a given operation to see what will happen in production. Developers who read mockist tests must translate each mock’s responses and expectations into the operations on the actual collaborator, which significantly reduces clarity and readability.

Thoughtful design. Refactorings happen independent of test code, so they are relatively easy to perform and thus happen frequently. If using a mockist approach, refactoring an object by changing its external interface, for example, requires re-writing or re-generating all the mocks for this object as well. There are no mocks to rewrite when using the classical approach, so refactoring requires fewer changes in test code. This makes refactorings easier to perform, which means it happens more often, and the design of the codebase improves over time.

Be flexible

In practice, I advise a test isolation approach that starts with the classical style and falls back on the mockist style when necessary. Martin Fowler says, "I don’t treat using doubles for external resources as an absolute rule. If talking to the resource is stable and fast enough for you, then there's no reason not to do it in your unit tests. [...] Indeed when xunit testing began in the ’90s, we made no attempt to go solitary unless communicating with the collaborators was awkward (such as a remote credit card verification system).”

As long as we’re using fast, reliable collaborators (which should be our goal anyway), testing with real collaborators doesn’t negatively impact the speed and reliability of our tests. When this is not the case (when collaborating with an external service over HTTP, for example), test doubles are a great way to increase a test’s speed and reliability while sacrificing a bit of confidence, clarity, and flexibility.

When considering the type of test double to use, favor fakes over mocks. Fakes have a few key advantages over mocks:

• A fake behaves more similarly to the real collaborator than a mock, giving us more confidence.
• We interact with a fake in the same way that we interact with a real collaborator, so it provides better documentation.
• A fake must be updated whenever the real collaborator is updated, just like a mock. However, with fakes, no expectations or verifications need to be updated, so refactoring a codebase with fakes tends to be easier than refactoring a codebase with mocks.

Conclusion

When determining your testing approach, think carefully about your approach to unit isolation, so you’re aware of the benefits and tradeoffs of starting with a classical or a mockist approach. Always be willing to adapt your approach depending on the nature of your collaborators. Ultimately we all want a fast, reliable test suite that gives us the confidence to ship, documents our intentions clearly, and helps us to design an extensible system.