Announcing xUnit.net

In the 5 years since the release of NUnit 2.0, there have been millions of lines of code written using the various unit testing frameworks for .NET. About a year ago it became clear to myself and Brad Wilson that there were some very clear patterns of success (and failure) with the tools we were using for writing tests. Rather than repeating guidance about "do X" or "don't do Y", it seemed like it was the right time to reconsider the framework itself and see if we could codify some of those rules.

 

Additionally, the .NET framework itself has evolved a lot since its v1 release in early 2002. Being able to leverage some of the new framework features can help us write clearer tests.

 

Another aspect of change that we wanted to affect was bringing the testing framework more closely in line with the .NET platform. Many of the decisions we made, which we enumerate below, were driven by this desire. We wanted an architecture which is built specifically for programmer testing (specifically Test-Driven Development), which can also be very easily extended to support other kinds of testing (like automated acceptance tests).

 

Finally, there have been advances in other unit test library implementations that have not really surfaced in the .NET community.

 

While any one of these reasons would not necessarily have been sufficient to create a new testing framework, the combination of them all made us want to undertake a new project: xUnit.net.

 

Lessons Learned

 

   
• Single Object Instance per Test Method. Much has been written about why this improves test isolation. In xUnit.net we create a new instance per test. For more information, see http://blogs.msdn.com/jamesnewkirk/archive/2004/12/04/275172.aspx and http://martinfowler.com/bliki/JunitNewInstance.html.
   
• No [SetUp] or [TearDown]. I blogged recently about some of the problems related to SetUp/TearDown. xUnit.net does not have any built-in support for this capability. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html.
   
• No [ExpectedException]. Rather that decorating a method with an attribute, we have returned to the old JUnit style of Assert.Throws for expected exceptions. This helps two major issues: 1. With [ExpectedException] it's possible to hide real errors when the wrong method call throws an exception, and 2. Allows your tests to continue to obey the Arrange-Act-Assert pattern (as coined "3A" by William Wake).
   
• Aspect-Like Functionality. End users extended NUnit and MbUnit with cross-cutting concerns that could be attached to test methods (an example is automatically rolling back changes made to a database during the test). This made the tests simpler to write and allowed more consistent usage of the cross-cutting operations. xUnit.net makes it very simple to create such operations and attach them to test methods.
   
• Reducing the Number of Custom Attributes. Sometimes, the excessive use of attributes can make you feel like you've diverged far from the underlying language. xUnit.net removed some attributes from the framework, instead relying on language features to provide similar functionality:      
        
  • [TestFixture] was removed entirely, so tests can be anywhere.
         
  • [Ignore] is expressed using the Skip= parameter on [Test].
         
  • [SetUp] and [TearDown] are removed.
         
  • [ExpectedException] was replaced with Assert.Throws.
         
  • [TestFixtureSetup] and [TestFixtureTearDown] are instead expressed as implementations of an interface (ITestFixture).
         
  • Support for IDisposable was added for tests.
      
     

 

Language Features

 

   
• Use of Generics. The addition to generics to .NET 2.0 allowed much more concise assertions, even allowing some compile-time errors (mismatched types) to be caught. It also allowed us to add type-specific comparer support for the more common asserts (like Equal and NotEqual).
   
• Anonymous Delegates. Support for anonymous delegates in .NET 2.0 made the syntax for Assert.Throws much more compact and readable. Here are two examples of Assert.Throws:      
        
  Assert.Throws<InvalidOperationException>(delegate { operation(); });  // .NET 2.0      
        
  Assert.Throws<InvalidOperationException>(() => operation());  // .NET 3.5

 

Test Runner

 

   
• For v1, we are shipping a console-based test runner, but not a GUI-based test runner. Based on user feedback, we will determine which other runners xUnit.net should support in future releases.

 

Extensibility

 

   
• Assert extensibility. Through the use of custom comparers (that implement IComparer<T>), you can extend the concepts of Equal, NotEqual, InRange, and NotInRange for your tests. There are two examples of this in the Samples project (one which does case-insensitive comparisons and one which does date-only DateTime comparisons).
   
• Test method extensibility. The definition of how to run a test method can be extended. There are two example of this: the first, in the extensions DLL, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see http://shareandenjoy.saff.net/2006/12/new-paper-practice-of-theories.html.
   
• Test class extensibility. The definition of run to run a test class can be extended. There is an example of this in the extensions DLL, the [RunWithNUnit] attribute which allows you to have mixed xUnit.net and NUnit tests in the same assembly, all executable by any xUnit.net runner.

Comments

[Joshua Flanagan]

I think a console runner is sufficient. Don't waste resources on a GUI runner - just make sure to give Jamie Cansdale any help he needs adding TestDriven.NET support.

[Brad Wilson]

Joshua, we're already shipping support for TD.NET :)

[Haacked]

Could you post some code snippets of tests. For example, one with the rollback you mentioned. Also, is there a way to do parameterized tests ala MbUnit's RowTest?

I like the reduction of attributes. It's a great idea.

[Chris R. Chapman]

Seriously good work! Glad to see some rational, experience-driven advances made in the TDD sphere with this release.

Kudos!

[Troy DeMonbreun]

No SetUp or TearDown is throwing out the baby with the bathwater, IMHO, FWIW, YMMV. My point being that it will be hard to supplant NUnit without it. Maybe some optional/non-default/buried approach (not sure what that would look like, OTTOMH) would encourage users to go to xUnit.net.

[Herb]

I love seeing new, fresh products that look to improve on what I'm already using. However, from just reading this page I'm not interested in your application. It seems like every bullet point is either about a feature you don't support or about a feature that requires extra coding from me to get working. I'm sure your product offers something that nunit does not, otherwise you wouldn't have written it. But I don't see much on this page that explains what those advantages are. Please take this as helpful feedback. I'm really not trying to be a jerk here. I just think you need to do a better job of selling your application here. Good luck!

[Brad Wilson]

Troy, you can use constructor/IDisposable.Dispose() as a direct replacement for SetUp and TearDown.

Haacked, in xunit.extensions is a new test attribute called [Theory], which allows data driven tests. We need to publish some samples of it, but for now you can see examples within the unit/acceptance tests in Tests.xunit.extensions.

[Dan]

This all sounds great... As a long time user of NUnit (and when I was coding Java last 2-3years back, JUnit), its great to see the framework 'evolve'...

Whilst I share some of the concerns of loosing [SetUp] and [TearDown], Im fairly confident that once i see example code of the newer syntax, it'll all be clear...

I do tho, have one suggestion.

The initial version will ship with just a console-runner, which is fair enough. But how about in the next version, a .NET plugin which can be launched from a button click. The button click will compile the test code and the targetted .dll/.dll's and present a 'results window' or even present the results in the workspace. Will just save developers some slight hassle of having to jump back and forth between windows; we can just stay in the .NET IDE...

Just my 2cents.

[Keith Reichert]

Would adding SetUp and TearDown properties to the Test attribute be beneficial? You could have tests in a fixture that don't need to have SetUp and TearDown methods. There could also be sets of tests that run completely different Setup and TearDown processes in the same fixture. Not to mention that you could look at any test and see what sort of SetUp and TearDown methods are required.

[DM]

This sounds great. The only thing that is kind of annoying is the name. I've always heard "xUnit" used to describe any of the unit testing frameworks. See: http://en.wikipedia.org/wiki/XUnit

Again, it really sounds great. Just nitpicking on the name.