xUnit.net - Next Generation of Unit Testing Frameworks?

Posted by Hartmut Wilms on Sep 25, 2007

Sections

Process & Practices,

Development

Topics

Agile ,

Unit Testing ,

Agile Techniques ,

.NET ,

Delivering Quality

Tags

Testing ,

NUnit ,

TDD

Jim Newkirk, creator of NUnit, has announced a new Unit Testing Framework called xUnit.net. The proclaimed successor to NUnit is supposed to get rid of NUnit's mistakes and shortcomings and add some best practices and extensibility to the framework.

Jim Newkirk and Brad Wilson, the creators of xUnit.net, have identified the following bad practices, shortcomings and improvements from the experience with NUnit and other Unit Testing frameworks:

• Single Object Instance per Test Method.
• No [SetUp] or [TearDown].
• No [ExpectedException].
• Aspect-Like Functionality.
• Reducing the Number of Custom Attributes.
• Use of Generics.
• Anonymous Delegates.
• Assert extensibility.
• Test method extensibility.
• Test class extensibility.

xUnit.net reduces the amount of .NET Attributes, which control the tests and their execution. There is a single Attribute [Test], which marks a test method. Unlike NUnit, MbUnit, or MSTest, Test classes are not marked. XUnit simply looks for all public test methods in all public classes within the assembly. [SetUp] and [TearDown] methods have been abandoned as being "generally bad" practice:

The xUnit.net team feels that per-test setup and teardown creates difficult-to-follow and debug testing code, often causing unnecessary code to run before every single test is run.

Jim Newkirk has written about "Why you should not use SetUp or TearDown in NUnit on his blog:

The problem that I have with SetUp in this case is twofold. The first and primary complaint is that when I am reading each test I have to glance up to BeforeTest() to see the values that are being used in the test. Worse yet if there was a TearDown method I would need to look in 3 methods. The second issue is that BeforeTest() initializes member variables for all 3 tests which complicates BeforeTest() and makes it violate the single responsibility pattern.

The [ExpectedException] Attribute, which expects the declared Exception to be thrown by the test code, has been replaced by the Assert.Throws assertion. TestFixtures are introduced by an ITestFixture interface, which declares two methods: BeforeAllTests() and AfterAllTests(). Test timeouts and temporarily skipping a test are implemented by parameters of the [Test] attribute instead of full fledged attributes. The very popular [RowTest] and [Row] test patterns of MbUnit have been included as [Theory] and [DataViaXxx]:

The xunit.extensions.dll ships with support for data-driven tests call Theories. Mark you test with [Theory] (instead of [Test]), and then mark it with one of the [DataVia...] attributes to indicate where the data will come from.

Assertions have also been reduced in xUnit.net. Every assertion, whose functionality can be achieved by one of the basics, has been left out. In addition the prefixes "is" and "are", e.g. "AreEqual" or "IsEmpty", have been removed. The xUnit.net site provides an extensive comparison of NUnit, MbUnit, MSTest, and xUnit.net attributes and assertions.

xUnit.net also makes use of the new language features of .NET 2.0 and 3.5. It supports the use of generics, which for instance provide type-safety in comparers, such as Equal and NotEqual assertions. The Assert.Throws() method, which replaces the [ExpectedException] attribute, expects anonymous delegates or lambda expressions, which encapsulate the functionality, which is expected to throw the declared exception. The code gets more compact and more readable:

Assert.Throws(delegate { operation(); }); // .NET 2.0
Assert.Throws(() => operation()); // .NET 3.5

Test classes, test methods, and assertions can easily be extended. The IComparer interface allows to extend the functionality of Equals, NotEqual and suchlike. xUnit.net supports the creation of test patterns, which control how tests are invoked and executed. Finally the execution of a test suite or test class can be controlled by extending existing or creating new test runners.

The creators of xUnit.net clearly think of their open source framework as the successor to NUnit. Roy Osherove thinks that xUnit.net is still premature and has some doubts about its future.

1. Back to top

   "You can misuse it, so it's bad"

   by J. B. Rainsberger

   I'm so tired of this fallacious reasoning. Brad Wilson wrote this as a reason to avoid set up and tear down in tests:
   
   "3. Inflexibility
   
   By using a fixed SetUp and TearDown, there is a presumption that every single test needs the exact same execution environment. There is a tendency over time to continue to pile things into SetUp and TearDown that are only needed for a few tests. This has the potential to impact readability and performance."
   
   For years we have been telling people to organize tests around fixture objects. As a result, if you have fixture objects being set up for only 3 out of 8 tests in your test fixture, then split the fixture in two! If you choose not to do this, you're the problem, not fixtures.
   
   My test fixtures call into three categories, and it works just fine:
   
   1. Common fixture objects with common values. Declaration and initialization happens in set up/before.
   
   2. Common fixture objects without common values. Declaration happens in set up/before, but initialization happens in the test/spec method.
   
   3. No common fixture objects. Each test is an island and there are no fields in the test case class. This is the "kitchen sink" test fixture.
   
   Please don't tell me something is a bad idea because some people use it poorly. That argument is old, tired, and bullshit.

   Reply

2. Back to top

   Re:

   by J. B. Rainsberger

   My test fixtures call into three categories, and it works just fine:

   
   
   Of course, that should read "...fall into...". I mishit Post Message when I wanted to hit Preview.

   Reply

3. Back to top

   RE: "You can misuse it, so it's bad"

   by Jim Newkirk

   I think the point that is attempting to be made is that there is a difference between trying to prevent misuse (which you cannot do) and the design of the feature which causes people to make bad decisions. My experience with NUnit test code leads me to believe that the design of SetUp and TearDown is flawed and needs to be changed.
   
   J.B. you are correct that smaller more focused fixtures is an appropriate answer. However, most of the code that I see and have attempted to help people with is not factored into the 3 categories that you describe. I could continue to provide advice to people on how best to use what is there or I could try to do something at the framework level to address the issue.

   Reply

4. Back to top

   Re: RE:

   by J. B. Rainsberger

   Don't get me wrong, Jim; I respect the goal to help people make better decisions, but I see two problems with what Brad wrote: it's too easy to conclude that the tool is at fault, when it's people at fault; and it's another example of trying to solve an obvious people problem with a technical solution.
   
   I disagree that the feature's design causes people to make bad decisions. I concede that it doesn't stop people from making bad decisions, which is a subtle but important difference.

   Reply

• Older >