James Newkirk

Announcing xUnit.net

In the 5 years since the release of NUnit 2.0, there have been millions of lines of code written using the various unit testing frameworks for .NET. About a year ago it became clear to myself and Brad Wilson that there were some very clear patterns of success (and failure) with the tools we were using for writing tests. Rather than repeating guidance about "do X" or "don't do Y", it seemed like it was the right time to reconsider the framework itself and see if we could codify some of those rules.

 

Additionally, the .NET framework itself has evolved a lot since its v1 release in early 2002. Being able to leverage some of the new framework features can help us write clearer tests.

 

Another aspect of change that we wanted to affect was bringing the testing framework more closely in line with the .NET platform. Many of the decisions we made, which we enumerate below, were driven by this desire. We wanted an architecture which is built specifically for programmer testing (specifically Test-Driven Development), which can also be very easily extended to support other kinds of testing (like automated acceptance tests).

 

Finally, there have been advances in other unit test library implementations that have not really surfaced in the .NET community.

 

While any one of these reasons would not necessarily have been sufficient to create a new testing framework, the combination of them all made us want to undertake a new project: xUnit.net.

 

Lessons Learned

 

   
• Single Object Instance per Test Method. Much has been written about why this improves test isolation. In xUnit.net we create a new instance per test. For more information, see http://blogs.msdn.com/jamesnewkirk/archive/2004/12/04/275172.aspx and http://martinfowler.com/bliki/JunitNewInstance.html.
   
• No [SetUp] or [TearDown]. I blogged recently about some of the problems related to SetUp/TearDown. xUnit.net does not have any built-in support for this capability. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html.
   
• No [ExpectedException]. Rather that decorating a method with an attribute, we have returned to the old JUnit style of Assert.Throws for expected exceptions. This helps two major issues: 1. With [ExpectedException] it's possible to hide real errors when the wrong method call throws an exception, and 2. Allows your tests to continue to obey the Arrange-Act-Assert pattern (as coined "3A" by William Wake).
   
• Aspect-Like Functionality. End users extended NUnit and MbUnit with cross-cutting concerns that could be attached to test methods (an example is automatically rolling back changes made to a database during the test). This made the tests simpler to write and allowed more consistent usage of the cross-cutting operations. xUnit.net makes it very simple to create such operations and attach them to test methods.
   
• Reducing the Number of Custom Attributes. Sometimes, the excessive use of attributes can make you feel like you've diverged far from the underlying language. xUnit.net removed some attributes from the framework, instead relying on language features to provide similar functionality:      
        
  • [TestFixture] was removed entirely, so tests can be anywhere.
         
  • [Ignore] is expressed using the Skip= parameter on [Test].
         
  • [SetUp] and [TearDown] are removed.
         
  • [ExpectedException] was replaced with Assert.Throws.
         
  • [TestFixtureSetup] and [TestFixtureTearDown] are instead expressed as implementations of an interface (ITestFixture).
         
  • Support for IDisposable was added for tests.
      
     

 

Language Features

 

   
• Use of Generics. The addition to generics to .NET 2.0 allowed much more concise assertions, even allowing some compile-time errors (mismatched types) to be caught. It also allowed us to add type-specific comparer support for the more common asserts (like Equal and NotEqual).
   
• Anonymous Delegates. Support for anonymous delegates in .NET 2.0 made the syntax for Assert.Throws much more compact and readable. Here are two examples of Assert.Throws:      
        
  Assert.Throws<InvalidOperationException>(delegate { operation(); });  // .NET 2.0      
        
  Assert.Throws<InvalidOperationException>(() => operation());  // .NET 3.5

 

Test Runner

 

   
• For v1, we are shipping a console-based test runner, but not a GUI-based test runner. Based on user feedback, we will determine which other runners xUnit.net should support in future releases.

 

Extensibility

 

   
• Assert extensibility. Through the use of custom comparers (that implement IComparer<T>), you can extend the concepts of Equal, NotEqual, InRange, and NotInRange for your tests. There are two examples of this in the Samples project (one which does case-insensitive comparisons and one which does date-only DateTime comparisons).
   
• Test method extensibility. The definition of how to run a test method can be extended. There are two example of this: the first, in the extensions DLL, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see http://shareandenjoy.saff.net/2006/12/new-paper-practice-of-theories.html.
   
• Test class extensibility. The definition of run to run a test class can be extended. There is an example of this in the extensions DLL, the [RunWithNUnit] attribute which allows you to have mixed xUnit.net and NUnit tests in the same assembly, all executable by any xUnit.net runner.

Ridiculous Legalese

I don't know what it is but every time I have my car in for service I get the urge to blog something. This time something was fixed on my car and the person at the cashier wanted me to sign something inidcating that I understood what the Limited Warranty was for the part was fixed. An excerpt from the Limited Warranty is as follows:

Dealer warrants all labor with respect to the repairs performed under the original repair order against defects in workmanship under normal use for a period of 24 months or unlimited miles, whichever comes first. ...

It seems to me that it just could have said 24 months because I believe 24 months will be here before unlimited miles.

Why you should not use SetUp and TearDown in NUnit

When I lead the team that built NUnit we implemented SetUp and TearDown similar to the way they were implemented in JUnit. For those who do not know SetUp and TearDown are attributes on methods in a TestFixture that perform common initialization and destruction.  Here is an example that demonstrates how and when SetUp and TearDown are called,

[TestFixture]
public class TestFixtureLifetime
{
    [SetUp]
    public void BeforeTest()
    { Console.WriteLine("BeforeTest"); }

    [TearDown]
    public void AfterTest()
    { Console.WriteLine("AfterTest"); }

    [Test]
    public void Test1()
    { Console.WriteLine("Test1"); }

    [Test]
    public void Test2()
    { Console.WriteLine("Test2"); }
}

When I run this test  I get the following output: .

BeforeTest
Test1
AfterTest
BeforeTest
Test2
AfterTest

I used to think that factoring out duplicated code into a common SetUp and TearDown method was good idea. My reasoning was based on one of the tenants of Simple Design; Remove Code Duplication, even from your test code. In order to better demonstrate why my thoughts have changed let's look at a slightly modified MoneyTest from the NUnit samples. MoneyTest has 3 Test methods (IsZero, BagAdd, and BagSubtractIsZero) and a Setup method, (BeforeTest). 

[TestFixture]
public class MoneyTest
{
    private Money f12CHF;
    private Money f14CHF;
    private Money f7USD;

    private MoneyBag moneyBag;

    [SetUp]
    public void BeforeTest()
    {
        f12CHF = new Money(12, "CHF");
        f14CHF = new Money(14, "CHF");
        f7USD = new Money(7, "USD");

        moneyBag = new MoneyBag(f12CHF, f7USD);
    }

    [Test]
    public void IsZero()
    {
        Money[] bag = { new Money(0, "CHF"), new Money(0, "USD") };
        Assert.IsTrue(new MoneyBag(bag).IsZero);
    }

    [Test]
    public void BagAdd()
    {
        Money[] bag = { new Money(26, "CHF"), new Money(7, "USD") };
        MoneyBag expected = new MoneyBag(bag);
        Assert.AreEqual(expected, f14CHF.Add(moneyBag));
    }

    [Test]
    public void BagSubtractIsZero()
    {
        Assert.IsTrue(moneyBag.Subtract(moneyBag).IsZero);       
    }
}

The problem that I have with SetUp in this case is twofold. The first and primary complaint is that when I am reading each test I have to glance up to BeforeTest() to see the values that are being used in the test. Worse yet if there was a TearDown method I would need to look in 3 methods. The second issue is that BeforeTest() initializes member variables for all 3 tests which complicates BeforeTest() and makes it violate the single responsibility pattern.  In order to fix this I would remove the BeforeTest() method and place the specific variables in each test - here is the refactored code:

[TestFixture]
public class NoSetUpMoneyTest
{
    [Test]
    public void IsZero()
    {
        Money[] bag = { new Money(0, "CHF"),
                        new Money(0, "USD") };
        Assert.IsTrue(new MoneyBag(bag).IsZero);
    }

    [Test]
    public void BagAdd()
    {
        Money f7USD = new Money(7, "USD");
        Money f12CHF = new Money(12, "CHF");
        Money f14CHF = new Money(14, "CHF");
        Money f26CHF = new Money(26, "CHF");

        Money[] bag = { f26CHF, f7USD };
        MoneyBag expected = new MoneyBag(bag);

        MoneyBag moneyBag = new MoneyBag(f12CHF, f7USD);

        Assert.AreEqual(expected, f14CHF.Add(moneyBag));
    }

    [Test]
    public void BagSubtractIsZero()
    {
        MoneyBag moneyBag = new MoneyBag(new Money(7, "USD"),
                                         new Money(12, "CHF"));
        Assert.IsTrue(moneyBag.Subtract(moneyBag).IsZero);
    }
}

In the refactored code the BeforeTest() is removed and each test is forced to do the initialization for what it needs to run. This makes it very easy to read each test as a single entity. Also, it has the side benefit of making it very clear which tests are more complicated than others. Compare the initialization code needed for BagAdd compared to IsZero. In addition the new class has no member variables. This is critical for test isolation. Without member variables there is no mechanism contained within the class for one test to mess with another.

These reasons lead me to never want to use common setup and tear down methods. In fact, I believe that this feature should be removed from the tool. Some people will argue that you cannot build a tool that stops people from writing poor code. However, you can write a tool that prevents people from shooting themselves in the foot.