BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

Writing a Comprehensive Unit Test

Posted by Jonathan Allen on May 24, 2012 |

A common theme amongst people professing "best practices" for unit tests is that you should only write a single assertion, maybe two, for each test. People who make these proclamations rarely show any unit test and those that do only show one. Yet if you follow their advice, a dozen other unit tests may be required to ensure quality for even a trivial operation. This article intends to demonstrate, by example, that multiple assertions per test are both necessary and beneficial.

Let us consider a fairly typical Person object that you would find in a data-binding scenario:

Testing FirstName

The first thing we want to test is setting the FirstName property. So to begin:

[TestMethod]
public void Person_FirstName_Set()
{
      varperson = new Person("Adam", "Smith");
      person.FirstName = "Bob";
      Assert.AreEqual("Bob", person.FirstName);
}

Next we want to test the property changed notification for first name.

[TestMethod]
public void Person_FirstName_Set_PropertyChanged()
{
      var person = new Person("Adam", "Smith");
      var eventAssert = new Granite.Testing.PropertyChangedEventAssert(person);
      person.FirstName = "Bob";
      eventAssert.Expect("FirstName");
}

When we run this test we will get a failure with the message "expected PropertyName ‘FirstName’ but received ‘IsChanged’". Apparently setting the first name property tripped the IsChanged flag and we need to account for it. So we add that into the mix:

[TestMethod]
public void Person_FirstName_Set_PropertyChanged()
{
      var person = new Person("Adam", "Smith");
      var eventAssert = new Granite.Testing.PropertyChangedEventAssert(person);
      person.FirstName = "Bob";
      eventAssert.SkipEvent(); //this was IsChanged
      eventAssert.Expect("FirstName");
}

With two passing tests, we consider what else needs to change when FirstName is altered. Looking over the API, the properties IsChanged and FullName took promising.

[TestMethod]
public void Person_FullName_Changed_By_Setting_FirstName()
{
      var person = new Person("Adam", "Smith");
      person.FirstName = "Bob";
      Assert.AreEqual("Bob Smith", person.FullName);
}

[TestMethod]
public void Person_IsChanged_Changed_By_Setting_FirstName()
{
      var person = new Person("Adam", "Smith");
      person.FirstName = "Bob";
      Assert.IsTrue(person.IsChanged);
}

And of course if these properties have changed we need to capture their property change notifications:

[TestMethod]
public void Person_IsChanged_Property_Change_Notification_By_Setting_FirstName()
{
      var person = new Person("Adam", "Smith");
      var eventAssert = new PropertyChangedEventAssert(person);
      person.FirstName = "Bob";
      eventAssert.Expect("IsChanged");
}
[TestMethod]
public void Person_FullName_Property_Change_Notification_By_Setting_FirstName()
{
      var person = new Person("Adam", "Smith");
      var eventAssert = new PropertyChangedEventAssert(person);
      person.FirstName = "Bob";
      eventAssert.SkipEvent(); //this was IsChanged
      eventAssert.SkipEvent(); //this was FirstName
      eventAssert.Expect("FullName");
}

Our next two tests address the HasErrors property and the ErrorsChanged event.

[TestMethod]
public void Person_FirstName_Set_HasErrorsIsFalse()
{
          var person = new Person("Adam", "Smith");
          person.FirstName = "Bob";
          Assert.IsFalse(person.HasErrors);
}
[TestMethod]
public void Person_FirstName_Set_ErrorsChanged_Did_Not_Fire()
{
          var person = new Person("Adam", "Smith");
          var errorsChangedAssert = new ErrorsChangedEventAssert(person);
          person.FirstName = "Bob";
          errorsChangedAssert.ExpectNothing();
}

We have eight tests so far, which means we have accounted for everything that should change when we alter the FirstName property. But that doesn’t mean we are done yet. We also need to make sure nothing else was changed accidentally. In theory that means several more assertions and an equal number of tests, but we’ll cheat and replace the HasErrors test with a utility called ChangeAssert.

[TestMethod]
public void Person_FirstName_Set_Nothing_Unexpected_Changed()
{
     var person = new Person("Adam", "Smith");
     var changeAssert = new ChangeAssert(person);
     person.FirstName = "Bob";
     changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
}

ChangeAssert simply captures the state of the object using reflection. Then later you can assert that nothing changed expect the specific properties you indicate.

Congratulations, you are done with your first test case. One down, many, many more to go.

What Do You Mean "One" Test Case?

The eight tests that we ended up keeping only cover the scenario in which FirstName is changed from "Adam" to "Bob" when all other fields are not in an error state and LastName is not null or empty. Let’s look at a more complete list of test cases:

  1. Setting the FirstName to "Adam"
  2. Setting the FirstName to null
  3. Setting the FirstName to an empty string
  4. Cases 1 thru 3, but when LastName is null.
  5. Cases 1 thru 3, but when LastName is an empty string.
  6. Cases 1 thru 5, but with FirstName starting with null.
  7. Cases 1 thru 5, but with FirstName starting with an empty string.

So far we are looking at 27 different scenarios. With up to 8 individual test per scenario we get an upper bound of 216 tests just for this one property. And in the grand scheme of things, this is a fairly trivial piece of code. So what are we to do?

Tests Can Have Code Smells Too

Looking back at our eight tests for the first test case, we see that each test has exactly the same setup and exactly the same operation. The only difference is the assertions we wrote. That’s what is known in the industry as a "Code smell". In fact there are two code smells according to the list in Wikipedia:

  • Duplicated code
  • Excessively long identifiers

We can eliminate both of these code smells simply by combining the assertions into one test:

[TestMethod]
public void Person_FirstName_Set()
{
     var person = new Person("Adam", "Smith");
     var eventAssert = new PropertyChangedEventAssert(person);
     var errorsChangedAssert = new ErrorsChangedEventAssert(person);
     var changeAssert = new ChangeAssert(person);
     person.FirstName = "Bob";
     Assert.AreEqual("Bob", person.FirstName, "FirstName setter failed");
     Assert.AreEqual("Bob Smith", person.FullName, "FullName not updated with FirstName changed");
     Assert.IsTrue(person.IsChanged, "IsChanged flag was not set when FirstName changed");
     eventAssert.Expect("IsChanged");
     eventAssert.Expect("FirstName");
     eventAssert.Expect("FullName");
     errorsChangedAssert.ExpectNothing("Expected no ErrorsChanged events");
     changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
}

Since it is still important to know what causes a test to fail, we make sure the assertions include a message.

Unit Tests and Code Reuse

Looking back over our 27 test cases, we may decide that setting the first name to null or empty string should have the same net effect. So we generalize it:

[TestMethod]
public void Person_FirstName_Set_Empty()
{
     Person_FirstName_Set_Invalid(String.Empty);
}
[TestMethod]
public void Person_FirstName_Set_Null()
{
     Person_FirstName_Set_Invalid(null);
}
public void Person_FirstName_Set_Invalid(string firstName)
{
     var person = new Person("Adam", "Smith");
     var eventAssert = new PropertyChangedEventAssert(person);
     var errorsChangedAssert = new ErrorsChangedEventAssert(person);
     var changeAssert = new ChangeAssert(person);
     Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
     Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
     Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");
     person.FirstName = firstName;
     Assert.AreEqual(firstName , person.FirstName, "FirstName setter failed");
     Assert.AreEqual("Smith", person.FullName, "FullName not updated with FirstName changed");
     Assert.IsTrue(person.IsChanged, "IsChanged flag was not set when FirstName changed");
     eventAssert.Expect("IsChanged");
     eventAssert.Expect("FirstName");
     eventAssert.Expect("FullName");
     Assert.IsTrue(person.HasErrors, "HasErrors should have remained false");
     errorsChangedAssert.ExpectCountEquals(1, "Expected an ErrorsChanged event");
     changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged", "HasErrors");
}

Seeing the differences between Person_FirstName_Set and Person_FirstName_Set_Invalid are fairly minor, we may seek to generalize it further.

[TestMethod]
public void Person_FirstName_Set_Valid()
{
     Person_FirstName_Set("Bob", false);
}
[TestMethod]
public void Person_FirstName_Set_Empty()
{
     Person_FirstName_Set(String.Empty, true);
}
[TestMethod]
public void Person_FirstName_Set_Null()
{
     Person_FirstName_Set(null, true);
}
public void Person_FirstName_Set(string firstName, bool shouldHaveErrors)
{
     var person = new Person("Adam", "Smith");
     var eventAssert = new PropertyChangedEventAssert(person);
     var errorsChangedAssert = new ErrorsChangedEventAssert(person);
     var changeAssert = new ChangeAssert(person);
     Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
     Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
     Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");
     person.FirstName = firstName;
     Assert.AreEqual(firstName, person.FirstName, "FirstName setter failed");
     Assert.AreEqual((firstName + " Smith").Trim(), person.FullName, "FullName not updated with FirstName changed");
     Assert.AreEqual(true, person.IsChanged, "IsChanged flag was not set when FirstName changed");
     eventAssert.Expect("IsChanged");
     eventAssert.Expect("FirstName");
     eventAssert.Expect("FullName");
     if (shouldHaveErrors)
     {
          Assert.IsTrue(person.HasErrors, "HasErrors should have remained false");
          errorsChangedAssert.ExpectCountEquals(1, "Expected an ErrorsChanged event");
          changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged", "HasErrors");
     }
     else 
     {
          errorsChangedAssert.ExpectNothing("Expected no ErrorsChanged events");
          changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
     }
}

There is definitely a limit as to how much you can generalize your test code before it becomes confusing. However, a meaningful test name paired with a good description on each assertion will go a long way towards keeping your tests understandable.

Controlling the Variables

So far all of our assertions considered only the outcomes of the test case. They assumed each person object started in a known-good state and went from there. But if we want to be scientific in our testing we need to ensure that we "control the variables". Or in other words, we need to ensure that the state of the world actually matches what we expect.

This leads to the following set of assertions.

Assert.IsFalse(person.HasErrors, "Test setup failed, HasErrors is not false");
Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");

Since we don’t want to repeat those assertions at the beginning of each test, we may choose to move it into a factory method that we can trust to always give us a clean object. This is especially useful if we end up reusing the same setup for testing other properties.

[TestMethod]
public void Person_FirstName_Set()
{
     var person = GetAdamSmith();
     ...

Table Style Testing

What it really comes down to is the number of "test methods" you have really has no bearing on how comprehensive your tests are. They are just a convenient way to organize and run your test cases.

Another way to organize a lot of test cases is to use a single table-driven test method. You lose the ability to run individual tests, but you gain the ability to add new test cases with a single line of code. The "table" in table style testing may come from a XML file, a database table, hard coded in arrays, or simply be the same function repeated called with different values. Some frameworks such as MBTest even allow you to specify the test cases using attributes, but to keep the examples portable we’ll stick to the lowest common denominator.

[TestMethod]
public void Person_FullName_Tests()
{     
     Person_FullName_Test("Bob", "Jones", "Bob Jones");
     Person_FullName_Test("Bob ", "Jones", "Bob Jones");
     Person_FullName_Test(" Bob", "Jones", "Bob Jones");
     Person_FullName_Test("Bob", " Jones", "Bob Jones");
     Person_FullName_Test("Bob", "Jones ", "Bob Jones");
     Person_FullName_Test(null, "Jones", "Jones");
     Person_FullName_Test(string.Empty, "Jones", "Jones");
     Person_FullName_Test("      ", "Jones", "Jones");
     Person_FullName_Test("Bob", "", "Bob");
     Person_FullName_Test("Bob", null, "Bob");
     Person_FullName_Test("Bob", string.Empty, "Bob");
     Person_FullName_Test("Bob", "      ", "Bob");
}
private void Person_FullName_Test(string firstName, string lastName, string expectedFullName)
{
     var person = GetAdamSmith();
     person.FirstName = firstName;
     person.LastName = lastName;
     Assert.AreEqual(expectedFullName, person.FullName,
          string.Format("Incorrect full name when first name is '{0}' and last name is '{1}'"
          firstName ?? "<NULL>", lastName ?? "<NULL>"));
}

It is very important that you use error messages that include the parameters when using this technique. If you don’t, you’ll find yourself having to step through code when trying to determine which particular combination isn’t correct.

Conclusion

When writing units of any variety one should try to maximize these factors:

  • Meaningful test coverage per unit of effort
  • Maintainability in the face of a changing code base
  • Performance of the test suite
  • Clarity in terms of what is being tested and why

While these factors are often in conflict, the careful use of multiple assertions per test can serve to improve all four factors by:

  • Reducing the amount of boilerplate code to be written
  • Reducing the amount of boilerplate code to be updated when the API changes
  • Reducing the physical number of boilerplate code to be executed per assertion
  • Documenting in one place all of the assertions that apply to a given operation

About the Author

Jonathan Allen has been writing news report for InfoQ since 2006 and is currently the lead editor for the .NET queue. If you are interested in writing news or educational articles for InfoQ please contact him at jonathan@infoq.com.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Wrong title by Joe Eames

Thanks for the time and effort you spent writing this article. It's obvious you put quite a bit of effort into it.

I do have a couple constructive criticisms for you:

The title of your article is a little misleading. You don't mention the fact that what you are testing is obviously some very painful and difficult to work with legacy code that you don't dare make changes to because of it's brittleness.

A more appropriate title would be "When unit testing painful legacy code, one assert per test is the least of your worries"

Also, it would behoove you to put in a disclaimer at the top about how if someone is writing new code, and using TDD, then none of the advice in this article applies. It's purely for testing legacy code.

In your section on Controlling The Variables, you imply that testing the setup over and over is a good thing. It is not. If you are going to test the construction of your person class, do that in one test. At that point, you can trust that your construction works correctly, and have no need to repeat those assertions. Repeating those assertions on each test is not only wasted code, but also unnecessary noise. In short, any given state, only requires a given assertion once.

Also, you should mention in your section on "Table Style Testing" that this is a solved issue with NUnit, which point could lead into a note about how developers should avoid using MSTest whenever possible, and in organizations that DO use MSTest, they should push for change to a more mature unit testing framework.

And finally, your point about code smells in tests is a bit wrong.

Overly long identifiers really don't apply to unit test names. They should read more like a sentence than typical code does. In fact the one code smell (which is testing specific) I see would be your sample unit test method name. Person_FirstName_Set just isn't expressive enough. It gives no indication as to the point of the test. Given that you unfortunately can't keep to one assert per test, you can't actually put in a truly expressive method name, but perhaps something like "Setting_FirstName_Runs_All_FirstName_Validations" or something along those lines.

Also, DRY is actually much less important in unit tests than expressiveness. An adage that "tests can be wet, just not soaking wet" applies here. A couple good articles about this:
blogs.msdn.com/b/matt/archive/2009/07/12/dry-an...
codebetter.com/karlseguin/2009/09/12/unit-testi...
on-agile.blogspot.com/2007/06/are-your-unit-tes...

Besides these points, I do appreciate the effort you spent writing this article.

Code correctness is crucial, but... by Guillaume L

It holds together, but do you seriously write this huge mass of comprehensive tests for every property of each of your objects ?

I consider unit tests are needed when there is minimum complexity and decision making/conditional logic involved. I never test getters/setters or code that only does mere platform calls for instance.

There are a number of other things that you seem to test while I typically wouldn't, don't know if you really do that day to day or if it's just for the sake of the demonstration :

- testing that "no other thing has changed than what we wanted to change" - that could take a long time ;)
- checking for HasErrors all the time.
- caring about variables being null or empty when it has no consequence on the program's correct execution and no special behavior to it.
- caring about the state of other variables not directly connected to the one under test.

Finally, if you really want to go for that level of extremeness in testing, wouldn't it be a typical job for automated test generation using metaprogramming/templating ? After all, these are just stupid properties that fall remarkably uniformly under the same basic patterns (PropertyChanged, HasErrors...)

Re: Wrong title by Jonathan Allen

There is nothing "legacy" about the code being tested. This is just a normal model with typical validation, change tracking, and property change notifications. You see the same thing in any XAML based user interface.

And really it isn't even that complicated. Anything simpler probably isn't worth testing.

Re: Code correctness is crucial, but... by Jonathan Allen

Well I did say "comprehensive" unit tests. To be totally honest I've only used this level of testing once. It was for an electronic medical records application for a nursing station, which means even a minor UI glitch could cause someone to get a wrong, and potentially lethal, dose of medication.

Unit testing properties is strange to me. Looking back at my notes, I see 2.7 bugs per 100 property tests in a project that had over 1400 tests at nearly this level of quality. There is no way we could have justified 100% hand written unit tests for that project, the time per bug ratio was way too low. But we couldn’t allow the bugs to go uncaught either.

So yes, we did end up using quite lot of machine generated test code. Unfortunately it could only go so far because of calculated fields.

That code generator is owned by the medical network, but I've started working on a general purpose unit test generator for .NET applications. It's called Test Monkey and it is primarily interface based. For example, if it sees that a class implemented the IEditableObject interface then it will run through a series of tests to ensure that BeginEdit, EndEdit, and CancelEdit perform correctly across all properties in the class.

Re: Wrong title by Joe Eames

en.wikipedia.org/wiki/Legacy_code see the section "modern interpretations".

Unit testing properties by Michael Perry

I realise that it's not your main point, but I would not test properties at all. Instead of automation or code reuse, just don't write unnecessary tests. There are plenty of frameworks that automate property changed notification. Once you trust the framework, you don't need to test it anymore. Save your tests for things that are hard to get right.

Re: Unit testing properties by Jonathan Allen

I would recommend this level of testing on a couple classes to verify the framework. I've seen the frameworks do silly things like fire change events when nothing has changed.

But in general I agree that property testing is usually not worth the effort. Your higher level tests should catch most of the bugs.

unit testing does not improve poor design by phloid domino

Alas, the example is all too common -- this is not an object, it is a collection of publicly exposed values with psuedo-methods that do not really offer any meaningful service(s).

Even then, the 'properties' are not in the same semantic realm (HasErrors and IsChanged are about something completely different than the attributes of the thing being modeled, a 'person').

Now, I'm ok with using a simple example to make a point, but unit tests for this? really! And a lot of code for what? Testing getters and setters, and some very trivial constraints.

If all you got is a glorified 'struct' or 'record', you're focusing the on the wrong level.

The real sad news is that sooo many projects and developers revel in this type of misguided unit testing, get a false sense of well-being, and miss the important stuff.

Unless, of course, you're working in some Dilbert-esque environment where 'productivity' is measured in lines of code....

Test patterns by Matt Honeycutt

The "one assert per test" guideline exists so that you can easily tell more about *why* a test fails. In your example, if the "Person_FullName_Tests" fails, why did it fail? It will tell you the first failure, but that's all. Maybe the other cases are fine, but maybe they aren't.

Concerning method names, the Gerkin-style naming that is prevalent these days is far better than your "Person_FullName_Tests" style. What does that spec test? It's impossible to tell without fully reading the code. Tests named BDD-style are highly descriptive, so you can tell *what* they are testing without having to dive in to *how* they're testing.

There are actually test patterns that help keep things semi-DRY, and modern frameworks like SpecsFor (yes, I created it, yes, I'm certainly biased) help you keep your tests simple *and* readable while promoting code reuse where it makes sense.

Re: Wrong title by John Goodsen

I pretty much agree with Joe. I am concerned somebody new to testing will read this article and walk away with the idea that refactoring many asserts into a single test method is a good idea. Please add some points to this article to warn readers away from that conclusion. I just waded through a bunch of those kinds of tests today with a client and it makes me cringe to see that in an InfoQ article ...

Re: Test patterns by Jonathan Allen

The "one assert per test" guideline exists so that you can easily tell more about *why* a test fails. In your example, if the "Person_FullName_Tests" fails, why did it fail? It will tell you the first failure, but that's all. Maybe the other cases are fine, but maybe they aren't.


Your test runner will tell you which assertion was violated. For more information you can, and should, put a message in each assertion saying why that assertion exists.

As for your other test cases, are they really that important at this point? I say no. Once you reach the red light/code is broken condition it is time to put down the testing tools and pick up the debugger.


What does that spec test? It's impossible to tell without fully reading the code.


Good.

Tests are not specifications. I shouldn't have to read a dozen or more tests to figure out how something is supposed to work from a business standpoint. Especially since those tests are necessarily going to be a mixture of business concerns and technological concerns and it's important to know which is which.

Writing comprehensive tests for the purpose of validating our code is correct is hard enough. Trying to make those same tests act as specifications, design documents, a debugger, and everything else under the sun turns hard into impossible and, in the end, usually leaves you without good tests or good specifications.

Re: Unit testing properties by Jonathan Allen

The property we are looking at here, while fairly simple, is still many times more complicated than most examples of unit testing you’ll see in magazines and blog posts. Yet those same-said articles still fail to have what I would consider a significant amount of test coverage.

You are right about one thing; this is far less complicated than the kind of stuff that you really want to focus on testing. But how can we expect someone to write a solid set of tests for something hard if they don’t understand how to test something that is easy?

Re: Unit testing properties by Russell East

Jonathan, thanks for writing this article. I understand why you challenge the "One assert per test" theory. I have been doing TDD for about 10 years starting with JUnit and mainly NUnit for most of these 10 years. I first heard of this "One assert per test" theory about 4 years ago. At the time, i believed its was made up by people trying to flex there agile muscles at an agile talk somewhere so make them sound like a super star programmer. Or just that a statement that was made being "one test per context" and was blurred as it traveled mouth to mouth.

I do agree with all the other comments so far that has been added. But I would highlight the essence of TDD. Its a design process not primarily for testing code. The fact that you have tests after you have designed your code is a bonus. But TDD is about design. Hence "Red, Green", "Refactor". I have, and so do many believe that code coverage is a good measure but not that important. I myself write more BDD style test these days and i believe that the BDD is TDD done right. But i stopped using all the BDD frameworks because they end up writing long methods that are over specified and just take to long to craft. Although I do you StoryQ for acceptance tests only (which is very different thing all together).

I would not test properties as many has mentioned, but i understand from your example that you are ensuring that the right events and flags are set. I would to be honest also do this if i was designing this observable behavior for objects and would most likely place this logic into a base class.

For me TDD/BDD is about designing behavior and getting quick feedback. having a suite of tests to ensure that my design and my behaviors still work is a bonus. Having the ability to refactor my code with confidence is also a nice thing to have.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

13 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT