Announcing xUnit.net

In the 5 years since the release of NUnit 2.0, there have been millions of lines of code written using the various unit testing frameworks for .NET. About a year ago it became clear to myself and Brad Wilson that there were some very clear patterns of success (and failure) with the tools we were using for writing tests. Rather than repeating guidance about "do X" or "don't do Y", it seemed like it was the right time to reconsider the framework itself and see if we could codify some of those rules.

 

Additionally, the .NET framework itself has evolved a lot since its v1 release in early 2002. Being able to leverage some of the new framework features can help us write clearer tests.

 

Another aspect of change that we wanted to affect was bringing the testing framework more closely in line with the .NET platform. Many of the decisions we made, which we enumerate below, were driven by this desire. We wanted an architecture which is built specifically for programmer testing (specifically Test-Driven Development), which can also be very easily extended to support other kinds of testing (like automated acceptance tests).

 

Finally, there have been advances in other unit test library implementations that have not really surfaced in the .NET community.

 

While any one of these reasons would not necessarily have been sufficient to create a new testing framework, the combination of them all made us want to undertake a new project: xUnit.net.

 

Lessons Learned

 

   
• Single Object Instance per Test Method. Much has been written about why this improves test isolation. In xUnit.net we create a new instance per test. For more information, see http://blogs.msdn.com/jamesnewkirk/archive/2004/12/04/275172.aspx and http://martinfowler.com/bliki/JunitNewInstance.html.
   
• No [SetUp] or [TearDown]. I blogged recently about some of the problems related to SetUp/TearDown. xUnit.net does not have any built-in support for this capability. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html.
   
• No [ExpectedException]. Rather that decorating a method with an attribute, we have returned to the old JUnit style of Assert.Throws for expected exceptions. This helps two major issues: 1. With [ExpectedException] it's possible to hide real errors when the wrong method call throws an exception, and 2. Allows your tests to continue to obey the Arrange-Act-Assert pattern (as coined "3A" by William Wake).
   
• Aspect-Like Functionality. End users extended NUnit and MbUnit with cross-cutting concerns that could be attached to test methods (an example is automatically rolling back changes made to a database during the test). This made the tests simpler to write and allowed more consistent usage of the cross-cutting operations. xUnit.net makes it very simple to create such operations and attach them to test methods.
   
• Reducing the Number of Custom Attributes. Sometimes, the excessive use of attributes can make you feel like you've diverged far from the underlying language. xUnit.net removed some attributes from the framework, instead relying on language features to provide similar functionality:      
        
  • [TestFixture] was removed entirely, so tests can be anywhere.
         
  • [Ignore] is expressed using the Skip= parameter on [Test].
         
  • [SetUp] and [TearDown] are removed.
         
  • [ExpectedException] was replaced with Assert.Throws.
         
  • [TestFixtureSetup] and [TestFixtureTearDown] are instead expressed as implementations of an interface (ITestFixture).
         
  • Support for IDisposable was added for tests.
      
     

 

Language Features

 

   
• Use of Generics. The addition to generics to .NET 2.0 allowed much more concise assertions, even allowing some compile-time errors (mismatched types) to be caught. It also allowed us to add type-specific comparer support for the more common asserts (like Equal and NotEqual).
   
• Anonymous Delegates. Support for anonymous delegates in .NET 2.0 made the syntax for Assert.Throws much more compact and readable. Here are two examples of Assert.Throws:      
        
  Assert.Throws<InvalidOperationException>(delegate { operation(); });  // .NET 2.0      
        
  Assert.Throws<InvalidOperationException>(() => operation());  // .NET 3.5

 

Test Runner

 

   
• For v1, we are shipping a console-based test runner, but not a GUI-based test runner. Based on user feedback, we will determine which other runners xUnit.net should support in future releases.

 

Extensibility

 

   
• Assert extensibility. Through the use of custom comparers (that implement IComparer<T>), you can extend the concepts of Equal, NotEqual, InRange, and NotInRange for your tests. There are two examples of this in the Samples project (one which does case-insensitive comparisons and one which does date-only DateTime comparisons).
   
• Test method extensibility. The definition of how to run a test method can be extended. There are two example of this: the first, in the extensions DLL, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see http://shareandenjoy.saff.net/2006/12/new-paper-practice-of-theories.html.
   
• Test class extensibility. The definition of run to run a test class can be extended. There is an example of this in the extensions DLL, the [RunWithNUnit] attribute which allows you to have mixed xUnit.net and NUnit tests in the same assembly, all executable by any xUnit.net runner.

Comments

[Haacked]

I don't think people are getting the fact that this bullet point:

* Single Object Instance per Test Method

Negates the need for Setup and TearDown. How do you setup and tear down a normal CLR object? Constructor and Dispose method (assuming IDisposable).

[Mike Gale]

What happens with existing projects that include large numbers of tests? When it comes time to enhance the project, I think most development teams will stick with the existing tests. The development time that goes into these tests is likely a big part of the effort spent.

Backward compatibility seems like a really good idea to help transition. Else we are creating "legacy tests".

I know that tests can be more important than the "conventional code base".

[yup]

Nothing says both NUnit and xUnit can't both be used per solution, right?

[Brad Wilson]

That is correct, yup. You can have both NUnit and xUnit.net tests in the same assembly.

You can either "run" the assembly twice, one with each runner, or you can add "[RunWithNUnit]" to your NUnit fixtures so that the xUnit.net runner will actually run your NUnit tests. The [RunWithNUnit] attribute comes from xunit.extensions.dll.

[Steve Freeman]

Interesting proposal and nice to see the shared instance stuff go.

Have you also looked at jUnit4's adoption of Hamcrest (a matcher library) for implementing assertions? It's working pretty well for us and allows people to write custom assertions. I believe Charlie Poole included something equivalent in NUnit.

S.

[Brad Wilson]

Steve, I've looked at the BDD style assertions in the past, and I can see there is value for both camps. We didn't provide that style of assertion, primarily for simplicity reasons. Speaking for myself here (and not Jim), I'm not a huge fan of giving people multiple ways to do something, because it really just confuses the issue of "which one is better".

Writing BDD style assertions that work with xUnit.net should be relatively trivial (after all, test frameworks are really just about throwing exceptions when things go wrong). It would be interesting to see if the community would be willing to contribute a "standard" extension to xUnit.net for BDD style assertions. For that matter, such a library should work with any of the .NET unit testing frameworks.

I feel the same way about mock object frameworks, personally. It doesn't have to be part of the unit testing framework; one should be able to pick and choose the pieces they want (NMock with NUnit, Rhino Mocks with xUnit.net, etc.) without feeling like one choice locks you into another.

jm2c :)

[Yves Hanoulle]

@Dan:
>But how about in the next version, a .NET plugin which can be >launched from a button click.
you can already do this with all console or gui testers:
you can confiure your visual studio to run any application when you run a project. (right click project/ select properties/ ...)

[kuldip]

yeah. I don't think gui runner is necessary. Console runner is good enough.

[Kevin Dente]

The removal of the TestFixture attribute is a drag for those of us who like to keep their test fixture classes in the same assembly as the classes under test. Seems like it would make finding all the test methods much slower.

[James Newkirk]

Kevin,

With regard to speed, are you concerned about the runners ability to find the tests or for the people looking through the code?

JIm