Announcing xUnit.net

In the 5 years since the release of NUnit 2.0, there have been millions of lines of code written using the various unit testing frameworks for .NET. About a year ago it became clear to myself and Brad Wilson that there were some very clear patterns of success (and failure) with the tools we were using for writing tests. Rather than repeating guidance about "do X" or "don't do Y", it seemed like it was the right time to reconsider the framework itself and see if we could codify some of those rules.

 

Additionally, the .NET framework itself has evolved a lot since its v1 release in early 2002. Being able to leverage some of the new framework features can help us write clearer tests.

 

Another aspect of change that we wanted to affect was bringing the testing framework more closely in line with the .NET platform. Many of the decisions we made, which we enumerate below, were driven by this desire. We wanted an architecture which is built specifically for programmer testing (specifically Test-Driven Development), which can also be very easily extended to support other kinds of testing (like automated acceptance tests).

 

Finally, there have been advances in other unit test library implementations that have not really surfaced in the .NET community.

 

While any one of these reasons would not necessarily have been sufficient to create a new testing framework, the combination of them all made us want to undertake a new project: xUnit.net.

 

Lessons Learned

 

   
• Single Object Instance per Test Method. Much has been written about why this improves test isolation. In xUnit.net we create a new instance per test. For more information, see http://blogs.msdn.com/jamesnewkirk/archive/2004/12/04/275172.aspx and http://martinfowler.com/bliki/JunitNewInstance.html.
   
• No [SetUp] or [TearDown]. I blogged recently about some of the problems related to SetUp/TearDown. xUnit.net does not have any built-in support for this capability. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html.
   
• No [ExpectedException]. Rather that decorating a method with an attribute, we have returned to the old JUnit style of Assert.Throws for expected exceptions. This helps two major issues: 1. With [ExpectedException] it's possible to hide real errors when the wrong method call throws an exception, and 2. Allows your tests to continue to obey the Arrange-Act-Assert pattern (as coined "3A" by William Wake).
   
• Aspect-Like Functionality. End users extended NUnit and MbUnit with cross-cutting concerns that could be attached to test methods (an example is automatically rolling back changes made to a database during the test). This made the tests simpler to write and allowed more consistent usage of the cross-cutting operations. xUnit.net makes it very simple to create such operations and attach them to test methods.
   
• Reducing the Number of Custom Attributes. Sometimes, the excessive use of attributes can make you feel like you've diverged far from the underlying language. xUnit.net removed some attributes from the framework, instead relying on language features to provide similar functionality:      
        
  • [TestFixture] was removed entirely, so tests can be anywhere.
         
  • [Ignore] is expressed using the Skip= parameter on [Test].
         
  • [SetUp] and [TearDown] are removed.
         
  • [ExpectedException] was replaced with Assert.Throws.
         
  • [TestFixtureSetup] and [TestFixtureTearDown] are instead expressed as implementations of an interface (ITestFixture).
         
  • Support for IDisposable was added for tests.
      
     

 

Language Features

 

   
• Use of Generics. The addition to generics to .NET 2.0 allowed much more concise assertions, even allowing some compile-time errors (mismatched types) to be caught. It also allowed us to add type-specific comparer support for the more common asserts (like Equal and NotEqual).
   
• Anonymous Delegates. Support for anonymous delegates in .NET 2.0 made the syntax for Assert.Throws much more compact and readable. Here are two examples of Assert.Throws:      
        
  Assert.Throws<InvalidOperationException>(delegate { operation(); });  // .NET 2.0      
        
  Assert.Throws<InvalidOperationException>(() => operation());  // .NET 3.5

 

Test Runner

 

   
• For v1, we are shipping a console-based test runner, but not a GUI-based test runner. Based on user feedback, we will determine which other runners xUnit.net should support in future releases.

 

Extensibility

 

   
• Assert extensibility. Through the use of custom comparers (that implement IComparer<T>), you can extend the concepts of Equal, NotEqual, InRange, and NotInRange for your tests. There are two examples of this in the Samples project (one which does case-insensitive comparisons and one which does date-only DateTime comparisons).
   
• Test method extensibility. The definition of how to run a test method can be extended. There are two example of this: the first, in the extensions DLL, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see http://shareandenjoy.saff.net/2006/12/new-paper-practice-of-theories.html.
   
• Test class extensibility. The definition of run to run a test class can be extended. There is an example of this in the extensions DLL, the [RunWithNUnit] attribute which allows you to have mixed xUnit.net and NUnit tests in the same assembly, all executable by any xUnit.net runner.

Comments

[Joshua Flanagan]

I think a console runner is sufficient. Don't waste resources on a GUI runner - just make sure to give Jamie Cansdale any help he needs adding TestDriven.NET support.

[Brad Wilson]

Joshua, we're already shipping support for TD.NET :)

[Haacked]

Could you post some code snippets of tests. For example, one with the rollback you mentioned. Also, is there a way to do parameterized tests ala MbUnit's RowTest?

I like the reduction of attributes. It's a great idea.

[Chris R. Chapman]

Seriously good work! Glad to see some rational, experience-driven advances made in the TDD sphere with this release.

Kudos!

[Troy DeMonbreun]

No SetUp or TearDown is throwing out the baby with the bathwater, IMHO, FWIW, YMMV. My point being that it will be hard to supplant NUnit without it. Maybe some optional/non-default/buried approach (not sure what that would look like, OTTOMH) would encourage users to go to xUnit.net.

[Herb]

I love seeing new, fresh products that look to improve on what I'm already using. However, from just reading this page I'm not interested in your application. It seems like every bullet point is either about a feature you don't support or about a feature that requires extra coding from me to get working. I'm sure your product offers something that nunit does not, otherwise you wouldn't have written it. But I don't see much on this page that explains what those advantages are. Please take this as helpful feedback. I'm really not trying to be a jerk here. I just think you need to do a better job of selling your application here. Good luck!

[Brad Wilson]

Troy, you can use constructor/IDisposable.Dispose() as a direct replacement for SetUp and TearDown.

Haacked, in xunit.extensions is a new test attribute called [Theory], which allows data driven tests. We need to publish some samples of it, but for now you can see examples within the unit/acceptance tests in Tests.xunit.extensions.

[Dan]

This all sounds great... As a long time user of NUnit (and when I was coding Java last 2-3years back, JUnit), its great to see the framework 'evolve'...

Whilst I share some of the concerns of loosing [SetUp] and [TearDown], Im fairly confident that once i see example code of the newer syntax, it'll all be clear...

I do tho, have one suggestion.

The initial version will ship with just a console-runner, which is fair enough. But how about in the next version, a .NET plugin which can be launched from a button click. The button click will compile the test code and the targetted .dll/.dll's and present a 'results window' or even present the results in the workspace. Will just save developers some slight hassle of having to jump back and forth between windows; we can just stay in the .NET IDE...

Just my 2cents.

[Keith Reichert]

Would adding SetUp and TearDown properties to the Test attribute be beneficial? You could have tests in a fixture that don't need to have SetUp and TearDown methods. There could also be sets of tests that run completely different Setup and TearDown processes in the same fixture. Not to mention that you could look at any test and see what sort of SetUp and TearDown methods are required.

[DM]

This sounds great. The only thing that is kind of annoying is the name. I've always heard "xUnit" used to describe any of the unit testing frameworks. See: http://en.wikipedia.org/wiki/XUnit

Again, it really sounds great. Just nitpicking on the name.

[Haacked]

I don't think people are getting the fact that this bullet point:

* Single Object Instance per Test Method

Negates the need for Setup and TearDown. How do you setup and tear down a normal CLR object? Constructor and Dispose method (assuming IDisposable).

[Mike Gale]

What happens with existing projects that include large numbers of tests? When it comes time to enhance the project, I think most development teams will stick with the existing tests. The development time that goes into these tests is likely a big part of the effort spent.

Backward compatibility seems like a really good idea to help transition. Else we are creating "legacy tests".

I know that tests can be more important than the "conventional code base".

[yup]

Nothing says both NUnit and xUnit can't both be used per solution, right?

[Brad Wilson]

That is correct, yup. You can have both NUnit and xUnit.net tests in the same assembly.

You can either "run" the assembly twice, one with each runner, or you can add "[RunWithNUnit]" to your NUnit fixtures so that the xUnit.net runner will actually run your NUnit tests. The [RunWithNUnit] attribute comes from xunit.extensions.dll.

[Steve Freeman]

Interesting proposal and nice to see the shared instance stuff go.

Have you also looked at jUnit4's adoption of Hamcrest (a matcher library) for implementing assertions? It's working pretty well for us and allows people to write custom assertions. I believe Charlie Poole included something equivalent in NUnit.

S.

[Brad Wilson]

Steve, I've looked at the BDD style assertions in the past, and I can see there is value for both camps. We didn't provide that style of assertion, primarily for simplicity reasons. Speaking for myself here (and not Jim), I'm not a huge fan of giving people multiple ways to do something, because it really just confuses the issue of "which one is better".

Writing BDD style assertions that work with xUnit.net should be relatively trivial (after all, test frameworks are really just about throwing exceptions when things go wrong). It would be interesting to see if the community would be willing to contribute a "standard" extension to xUnit.net for BDD style assertions. For that matter, such a library should work with any of the .NET unit testing frameworks.

I feel the same way about mock object frameworks, personally. It doesn't have to be part of the unit testing framework; one should be able to pick and choose the pieces they want (NMock with NUnit, Rhino Mocks with xUnit.net, etc.) without feeling like one choice locks you into another.

jm2c :)

[Yves Hanoulle]

@Dan:
>But how about in the next version, a .NET plugin which can be >launched from a button click.
you can already do this with all console or gui testers:
you can confiure your visual studio to run any application when you run a project. (right click project/ select properties/ ...)

[kuldip]

yeah. I don't think gui runner is necessary. Console runner is good enough.

[Kevin Dente]

The removal of the TestFixture attribute is a drag for those of us who like to keep their test fixture classes in the same assembly as the classes under test. Seems like it would make finding all the test methods much slower.

[James Newkirk]

Kevin,

With regard to speed, are you concerned about the runners ability to find the tests or for the people looking through the code?

JIm

[Kevin Dente]

Jim,
The runner's ability to find the tests. I always separate the test fixture classes out into a separate folder, so reading the code isn't really an issue.

[Jebs House]

I'm not sure what the hold-up is... maybe they have re-thought their stance on how this is going to actually make the company any money. Or perhaps their lawyers pointed out the liability of providing agents a platform to stick their feet in their mouth. Whatever it is, it's hardly something I'd claim as being "Well done".
www.jebshouse.com