NUnit

June 29, 2008

Agile Conference in Toronto

I will be attending the agile conference in Toronto August 4th – 8th. If you are interested in the details click here. The last information that I received stated that they are expecting almost 1600 people. A far cry from the first XP Universe in 2001. I think we were lucky if we had a 100.

This year I will be doing 2 sessions. Here are the details:

  • Rethinking Unit Testing: xUnit.net with Brad Wilson, Tuesday August 5, 2008 at 10:45am – Brad and I wrote xUnit.net and this session will detail the motivations and decisions that went into the design and implementation of the tool.
  • TDD Clinic: NUnit, Friday August 8, 2008 at 8:30am – This session is about using NUnit when doing test-first programming.

If your attending and want to chat please contact me. 

June 28, 2008

Lessons Learned in Programmer Testing

I have just finished the early summer conference circuit. I  spoke at the p&p Summit in Quebec City in early May, TechEd Developer in Orlando, and the Better Software Development Conference in Las Vegas.

The session that I gave at all of the conferences was about lessons that I have learned writing programmer tests over the years. Some of these lessons are now embodied in xUnit.net. I have attached the presentation (2135.7K) to this blog post. You can also download all of the sample code  (214.0K) from the sessions as well. The code is licensed under the Microsoft Public License.

Joe White has a detailed write-up on his blog outlining the TechEd session. He takes me to task about being dogmatic in a couple of places. I will admit to being dogmatic when it comes to test readability and maintenance. The specific lessons “Don’t use Setup and Teardown” and “Don’t use abstract base test classes” are discussed with readability and maintainability as the primary goals. This does potentially increase code duplication but I am willing to do that to improve the communication of the code.

March 16, 2008

xUnit.net v1.0(RC2) Released

Brad Wilson and I released xUnit.net V1.0(RC2) today. Brad has a great summary on his blog.

You can download the release from CodePlex. Click here for the release page.

December 18, 2007

xUnit.net RC1 Released Today

Brad and I just put the finisihing touches on xUnit.net RC1 today. See Brad's blog post for details.

If all goes well we expect to release v1 sometime in January.

October 27, 2007

xUnit.net Beta 2 released

On Thursday Brad and I released Beta 2 of xUnit.net. Brad wrote a forum post highlighting the changes that we made going from Beta 1 to Beta2.

In addition we also added support for Resharper in addition to TestDriven.net.

[Test] becomes [Fact]
The rationale for making this change stretches back to earlier this year when I attended the Agile Open Northwest conference. Kent Beck was discussing the paper from David Saff about theories and someone in the audience said that if you were going to call "for-all" tests "Theory" then you should call a single test "Fact". Since then I have had a number of conversations with people and Fact seems to fit.

I would like to thank all the people who have provided us feedback since we launched.

October 21, 2007

xUnit.net Beta 2 arriving shortly

Brad and I have been busy incorporating the initial feedback from the first beta of xUnit.net. Some of the highlights include support for Resharper 3.0 and a better way to do Fixture SetUp and Teardown. Brad has written up the the details here. We should be releasing the new beta sometime in the next week.

October 20, 2007

The CodePlex Team and the patterns & practices Summit

The CodePlex team will be well represented again at the patterns & practices Summit in Redmond, WA - November 5-9, 2007 . We will be presenting or co-presenting the following sessions:

  • "Yet Another Talk on Agility" - Brad Wilson and Peter Provost
  • Dependency Injection Frameworks - Brad Wilson and Scott Densmore
  • EntLib Devolved - Scott Densmore
  • Make It Your Own - Scott Densmore
  • Lessons Learned in Programmer Testing - Jim Newkirk
  • CodePlex is sponsoring an evening event on Tuesday titled "Open Source in the Enterprise".

Peter's not on the CodePlex team but we like him just the same. For additonal information on the summit and all of the sessions please click here.

September 20, 2007

Announcing xUnit.net

In the 5 years since the release of NUnit 2.0, there have been millions of lines of code written using the various unit testing frameworks for .NET. About a year ago it became clear to myself and Brad Wilson that there were some very clear patterns of success (and failure) with the tools we were using for writing tests. Rather than repeating guidance about "do X" or "don't do Y", it seemed like it was the right time to reconsider the framework itself and see if we could codify some of those rules.

 

Additionally, the .NET framework itself has evolved a lot since its v1 release in early 2002. Being able to leverage some of the new framework features can help us write clearer tests.

 

Another aspect of change that we wanted to affect was bringing the testing framework more closely in line with the .NET platform. Many of the decisions we made, which we enumerate below, were driven by this desire. We wanted an architecture which is built specifically for programmer testing (specifically Test-Driven Development), which can also be very easily extended to support other kinds of testing (like automated acceptance tests).

 

Finally, there have been advances in other unit test library implementations that have not really surfaced in the .NET community.

 

While any one of these reasons would not necessarily have been sufficient to create a new testing framework, the combination of them all made us want to undertake a new project: xUnit.net.

 

Lessons Learned

 
       
  • Single Object Instance per Test Method. Much has been written about why this improves test isolation. In xUnit.net we create a new instance per test. For more information, see http://blogs.msdn.com/jamesnewkirk/archive/2004/12/04/275172.aspx and http://martinfowler.com/bliki/JunitNewInstance.html.
  •    
  • No [SetUp] or [TearDown]. I blogged recently about some of the problems related to SetUp/TearDown. xUnit.net does not have any built-in support for this capability. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html.
  •    
  • No [ExpectedException]. Rather that decorating a method with an attribute, we have returned to the old JUnit style of Assert.Throws for expected exceptions. This helps two major issues: 1. With [ExpectedException] it's possible to hide real errors when the wrong method call throws an exception, and 2. Allows your tests to continue to obey the Arrange-Act-Assert pattern (as coined "3A" by William Wake).
  •    
  • Aspect-Like Functionality. End users extended NUnit and MbUnit with cross-cutting concerns that could be attached to test methods (an example is automatically rolling back changes made to a database during the test). This made the tests simpler to write and allowed more consistent usage of the cross-cutting operations. xUnit.net makes it very simple to create such operations and attach them to test methods.
  •    
  • Reducing the Number of Custom Attributes. Sometimes, the excessive use of attributes can make you feel like you've diverged far from the underlying language. xUnit.net removed some attributes from the framework, instead relying on language features to provide similar functionality:      
            
    • [TestFixture] was removed entirely, so tests can be anywhere.
    •        
    • [Ignore] is expressed using the Skip= parameter on [Test].
    •        
    • [SetUp] and [TearDown] are removed.
    •        
    • [ExpectedException] was replaced with Assert.Throws.
    •        
    • [TestFixtureSetup] and [TestFixtureTearDown] are instead expressed as implementations of an interface (ITestFixture).
    •        
    • Support for IDisposable was added for tests.
    •     
       
 

Language Features

 
       
  • Use of Generics. The addition to generics to .NET 2.0 allowed much more concise assertions, even allowing some compile-time errors (mismatched types) to be caught. It also allowed us to add type-specific comparer support for the more common asserts (like Equal and NotEqual).
  •    
  • Anonymous Delegates. Support for anonymous delegates in .NET 2.0 made the syntax for Assert.Throws much more compact and readable. Here are two examples of Assert.Throws:      
          
    Assert.Throws<InvalidOperationException>(delegate { operation(); });  // .NET 2.0      
          
    Assert.Throws<InvalidOperationException>(() => operation());  // .NET 3.5
 

Test Runner

 
       
  • For v1, we are shipping a console-based test runner, but not a GUI-based test runner. Based on user feedback, we will determine which other runners xUnit.net should support in future releases.
 

Extensibility

 
       
  • Assert extensibility. Through the use of custom comparers (that implement IComparer<T>), you can extend the concepts of Equal, NotEqual, InRange, and NotInRange for your tests. There are two examples of this in the Samples project (one which does case-insensitive comparisons and one which does date-only DateTime comparisons).
  •    
  • Test method extensibility. The definition of how to run a test method can be extended. There are two example of this: the first, in the extensions DLL, is the [Theory] attribute which allows data-driven tests; the second, in the samples, is the [RepeatTest] attribute which runs a test method multiple times in a row. For more information on data theories, see http://shareandenjoy.saff.net/2006/12/new-paper-practice-of-theories.html.
  •    
  • Test class extensibility. The definition of run to run a test class can be extended. There is an example of this in the extensions DLL, the [RunWithNUnit] attribute which allows you to have mixed xUnit.net and NUnit tests in the same assembly, all executable by any xUnit.net runner.

September 15, 2007

Why you should not use SetUp and TearDown in NUnit

When I lead the team that built NUnit we implemented SetUp and TearDown similar to the way they were implemented in JUnit. For those who do not know SetUp and TearDown are attributes on methods in a TestFixture that perform common initialization and destruction.  Here is an example that demonstrates how and when SetUp and TearDown are called,

[TestFixture]
public class TestFixtureLifetime
{
    [SetUp]
    public void BeforeTest()
    { Console.WriteLine("BeforeTest"); }

    [TearDown]
    public void AfterTest()
    { Console.WriteLine("AfterTest"); }

    [Test]
    public void Test1()
    { Console.WriteLine("Test1"); }

    [Test]
    public void Test2()
    { Console.WriteLine("Test2"); }
}

When I run this test  I get the following output: .

BeforeTest
Test1
AfterTest
BeforeTest
Test2
AfterTest

I used to think that factoring out duplicated code into a common SetUp and TearDown method was good idea. My reasoning was based on one of the tenants of Simple Design; Remove Code Duplication, even from your test code. In order to better demonstrate why my thoughts have changed let's look at a slightly modified MoneyTest from the NUnit samples. MoneyTest has 3 Test methods (IsZero, BagAdd, and BagSubtractIsZero) and a Setup method, (BeforeTest). 

[TestFixture]
public class MoneyTest
{
    private Money f12CHF;
    private Money f14CHF;
    private Money f7USD;

    private MoneyBag moneyBag;

    [SetUp]
    public void BeforeTest()
    {
        f12CHF = new Money(12, "CHF");
        f14CHF = new Money(14, "CHF");
        f7USD = new Money(7, "USD");

        moneyBag = new MoneyBag(f12CHF, f7USD);
    }

    [Test]
    public void IsZero()
    {
        Money[] bag = { new Money(0, "CHF"), new Money(0, "USD") };
        Assert.IsTrue(new MoneyBag(bag).IsZero);
    }

    [Test]
    public void BagAdd()
    {
        Money[] bag = { new Money(26, "CHF"), new Money(7, "USD") };
        MoneyBag expected = new MoneyBag(bag);
        Assert.AreEqual(expected, f14CHF.Add(moneyBag));
    }

    [Test]
    public void BagSubtractIsZero()
    {
        Assert.IsTrue(moneyBag.Subtract(moneyBag).IsZero);       
    }
}

The problem that I have with SetUp in this case is twofold. The first and primary complaint is that when I am reading each test I have to glance up to BeforeTest() to see the values that are being used in the test. Worse yet if there was a TearDown method I would need to look in 3 methods. The second issue is that BeforeTest() initializes member variables for all 3 tests which complicates BeforeTest() and makes it violate the single responsibility pattern.  In order to fix this I would remove the BeforeTest() method and place the specific variables in each test - here is the refactored code:

[TestFixture]
public class NoSetUpMoneyTest
{
    [Test]
    public void IsZero()
    {
        Money[] bag = { new Money(0, "CHF"),
                        new Money(0, "USD") };
        Assert.IsTrue(new MoneyBag(bag).IsZero);
    }

    [Test]
    public void BagAdd()
    {
        Money f7USD = new Money(7, "USD");
        Money f12CHF = new Money(12, "CHF");
        Money f14CHF = new Money(14, "CHF");
        Money f26CHF = new Money(26, "CHF");

        Money[] bag = { f26CHF, f7USD };
        MoneyBag expected = new MoneyBag(bag);

        MoneyBag moneyBag = new MoneyBag(f12CHF, f7USD);

        Assert.AreEqual(expected, f14CHF.Add(moneyBag));
    }

    [Test]
    public void BagSubtractIsZero()
    {
        MoneyBag moneyBag = new MoneyBag(new Money(7, "USD"),
                                         new Money(12, "CHF"));
        Assert.IsTrue(moneyBag.Subtract(moneyBag).IsZero);
    }
}

In the refactored code the BeforeTest() is removed and each test is forced to do the initialization for what it needs to run. This makes it very easy to read each test as a single entity. Also, it has the side benefit of making it very clear which tests are more complicated than others. Compare the initialization code needed for BagAdd compared to IsZero. In addition the new class has no member variables. This is critical for test isolation. Without member variables there is no mechanism contained within the class for one test to mess with another.

These reasons lead me to never want to use common setup and tear down methods. In fact, I believe that this feature should be removed from the tool. Some people will argue that you cannot build a tool that stops people from writing poor code. However, you can write a tool that prevents people from shooting themselves in the foot.

June 30, 2007

NUnit Converter V1.1 - Moved from GotDotNet

Due to the shutdown of GotDotNet I have moved the NUnit Converter V1.1 to here. Note: This is not a new version. If you have comments or questions please let me know.