James Newkirk

Assert.Single - A new assert for xUnit.net

A couple of weekends ago Brad Wilson and I added a new Assert to xUnit.net 

Recently I was doing the first cut of refactoring the console program and I noticed that we wrote the following test code quite often: 

IResultXmlTransform transform = transforms.Single();    

var transform = Assert.IsType<XslStreamTransformer>(...);

As I looked around we wrote this same type of code over and over. What I proposed was to write a new Assert named Assert.Single() which would take an IEnumerable<T> (and reluctantly its non-generic equivalent) and Brad suggested if there was only one element then the Assert would return that element. After introducing this new Assert the above code likes like the following: 

IResultXmlTransform transform = Assert.Single(transforms);

Not only is the more explicit it provides a better message. If the previous one had failed you would have received a LINQ exception. When we made the changes to our code we found that it affected about 3.5% of the Asserts in the xUnit.net code. Note: xUnit.net works with .NET Framework 2.0 and higher.

I ran this by Kent Beck and we discussed the idea of having Asserts return useful values. His concern was that it was unexpected which can be a problem because its not discoverable. However, he said (I am paraphrasing)  that if that many asserts were changed then the code is telling you something and its good to listen. 

p&p Summit - Redmond

The patterns & practices summit is coming to Redmond, November 3-7, 2008. You can get details at http://tinyurl.com/4raw7l.

There is quite a lineup this time. Kent Beck will be there along with Scott Guthrie, Pat Helland, Billy Hollis, Brad Wilson, Rocky Lhotka, Peter Provost and more.

The CodePlex team will present the following sessions:

• Evolutionary Design - Jim Newkirk - 11:00 - 12:00 November 5
• Web Apps at Large: Distributed Caching - Drew Miller - 12:45 - 1:45 November 6
• Tuesday Evening Kickback - sponsored by CodePlex.

Hope to see you there.

Simple Design & Testing Conference

The Simple Design & Testing Conference is next weekend (September 12-14) at Microsoft's office in Downers Grove, IL. This is the third year of the conference. It is an open space conference and the price is right (free). You do have to register and write a short position paper. For more details see: http://sdtconf.com.

I have found that this conference has great discussions and with regards to this topic better content than the Agile Conference. If you have time please come by and check it out. See you there.

Support Visual Studio Test Runner

We started investigating adding support for running tests with Visual Studio’s built-in test runner. It would require Visual Studio 2008 SP1, and would require Professional edition or higher.

In order to support running xUnit.net tests in Visual Studio, we discovered that the project needs to be marked as one of Visual Studio’s Test projects (essentially, it’s a C#/VB.NET project that has extra bits inside it that tell Visual Studio the project contains tests). Projects which are marked this way cannot be opened in versions of Visual Studio that do not have unit testing support installed (this includes Standard and Express editions, which do not offer unit testing, as well as any user who opts not to install the unit testing support).

Practically speaking, this causes the following to be true:

• You must have a separate project for tests. You cannot place your tests inside your main project.
• If you have an existing project with tests in it, you need to edit the .csproj file to add the extra bits into it. (You can also start over with a new project and port the code, but hand editing the .csproj seems easier and more success-prone.)
• If you create a new Unit Test project, you will need to remember to remove any references to the Visual Studio Test DLLs, since these will not be present unless Visual Studio is installed (for example, on your continuous integration server).
• If you forget to change the project type, the support simply won’t work. There will not be any feedback as to why; the tests simply won’t show up in Visual Studios Test views.
• If your project is consumed by many users with various versions of Visual Studio (for example, if you’re running an open source project), you are more likely to run into the version limitations than if you’re using xUnit.net for an internal project with a consistent developer platform.

Given this context, if you have an opinion about how to proceed could you voice your opinion here.

Agile Conference in Toronto

I will be attending the agile conference in Toronto August 4th – 8th. If you are interested in the details click here. The last information that I received stated that they are expecting almost 1600 people. A far cry from the first XP Universe in 2001. I think we were lucky if we had a 100.

This year I will be doing 2 sessions. Here are the details:

• Rethinking Unit Testing: xUnit.net with Brad Wilson, Tuesday August 5, 2008 at 10:45am – Brad and I wrote xUnit.net and this session will detail the motivations and decisions that went into the design and implementation of the tool.
• TDD Clinic: NUnit, Friday August 8, 2008 at 8:30am – This session is about using NUnit when doing test-first programming.

If your attending and want to chat please contact me. 

Lessons Learned in Programmer Testing

I have just finished the early summer conference circuit. I  spoke at the p&p Summit in Quebec City in early May, TechEd Developer in Orlando, and the Better Software Development Conference in Las Vegas.

The session that I gave at all of the conferences was about lessons that I have learned writing programmer tests over the years. Some of these lessons are now embodied in xUnit.net. I have attached the presentation (2135.7K) to this blog post. You can also download all of the sample code  (214.0K) from the sessions as well. The code is licensed under the Microsoft Public License.

Joe White has a detailed write-up on his blog outlining the TechEd session. He takes me to task about being dogmatic in a couple of places. I will admit to being dogmatic when it comes to test readability and maintenance. The specific lessons “Don’t use Setup and Teardown” and “Don’t use abstract base test classes” are discussed with readability and maintainability as the primary goals. This does potentially increase code duplication but I am willing to do that to improve the communication of the code.

Replacing ExpectedException in NUnit

Many people have heard and read my rants on ExpectedException in the past few months. If you have not heard it  in short I don't believe you should use it. In fact, in xUnit.net there is no such attribute. However, people tell me great; I am not using xUnit.net and I still want to adopt the approach. Well there is good news on a couple of fronts. First, NUnit 2.5 which is currently in alpha has implemented something similar to Assert.Throws. Secondly, if you don't want to wait for NUnit 2.5 or you don't want to upgrade I have provided an implementation of Assert.Throws, Assert.DoesNotThrow, and Record.Exception that you can download from here.

 

One of the biggest flaws of ExpectedException at the method level is that it can often succeed for the wrong reasons. For example, read through the following code and think about whether the test passes or not.

 

[Test]
[ExpectedException(typeof(ArgumentException))]
public void DepositThrowsArgumentExceptionWhenZero()
{
    CheckingAccount account = new CheckingAccount(0.00);

    account.Deposit(0.00);
}

Here is the relevant CheckingAccount code:

public CheckingAccount(double balance)
{
    if (balance == 0) 
       throw new ArgumentException("Initial balance cannot be zero");

    this.balance = balance; 
}

public void Deposit(double amount)
{
    if (amount == 0) 
       throw new ArgumentException("Deposit amount cannot be zero");

    balance += amount;
}

If you were to run this test in NUnit it would succeed. However if you look at the name of the test, DepositThrowsArgumentExceptionWhenZero it succeeds because the creation of the CheckingAccount class with an initial deposit of $0.00 also throws ArgumentException. A better way, in my opinion, uses the XunitAssert.Throws method to isolate the specific piece of code that throws the exception. Here is the same example using XunitAssert.Throws:

[Test]
public void DepositThrowsArgumentExceptionWhenZero()
{
    CheckingAccount account = new CheckingAccount(0.00);

    XunitAssert.Throws<ArgumentException>(
        () => account.Deposit(0.00));
}

If you were to run this test the test would fail on construction of the CheckingAccount object. The only “flaw” in this approach and I mean flaw lightly is that XunitAssert.Throws does violate the 3A pattern. In the downloaded code there is an alternative called Record.Exception which facilitates the writing of the tests using 3A. Here is the same example using Record.Exception.

[Test]
public void DepositThrowsArgumentExceptionWhenZero()
{
    CheckingAccount account = new CheckingAccount(0.00);

    Exception exception = 
        Record.Exception(() => account.Deposit(0.00));

    Assert.IsInstanceOfType(typeof(ArgumentException), exception);
}

I have described the above solutions using the .NET 3.5 Lambda syntax but all of the methods will work with a delegate and there are non-generic implementations as well. The sample code is licensed under the Microsoft Public License.

Download assert.zip (17.2K). I have tested the code with NUnit 2.4.6 as well at the testing tool in Visual Studio 2008 and the sample code includes the tests as well.

xUnit.net 1.0 Released Today

Brad Wilson and I released xUnit.net 1.0 RTM today.

Acknowledgements

Many people have helped us get to our 1.0 release.

First, we want to acknowledge the inventors of the core ideas behind xUnit frameworks, especially Kent Beck and Eric Gamma, without whom we probably wouldn't have anything to talk about today. :)

Second, we want to thank the people who hatched the idea of a new framework with us, including Peter Provost, Brian Button, Scott Densmore, Jonathan Wanagel, Jamie Cansdale, and Drew Miller.

Lastly, our community provided significant feedback to us during the development process by providing feedback on the web site, opening feature requests, reporting bugs, and generally helping guide the design of the framework. We especially want to thank Matt Podwysocki, Ben Hall, Harry Pierson, Scott Hanselman, Kirk Viehland, Daniel Cazzulino, Oren Eini, Jeff Brown, and Charlie Poole.

Looking Back

We release Beta 1 of our framework September 19, 2007, almost exactly 7 months ago. It took a lot longer than we thought it would to get to 1.0, especially given that we had been using the framework extensively internally for a while before that.

Our original release plans called for just a console runner, but our users quickly educated us on the value of variety. We personally are TestDriven.net users, so that became our second runner; our community quickly followed on asking us to support ReSharper and to write a stand-alone GUI runner. Our time spent with the console runner in an automated environment then drove us to write an MSBuild task which provides much better feedback on the testing process.

Extensibility was a very important feature for us. We felt that there was a lot to be desired in the landscape of extensibility for test frameworks. An important early decision we made was to push as much of our "functionality" as we possibly could out into extensions so that we were forced to exercise and improve our extensibility points.

Another unexpected issue was around test runners and their tight coupling to the version of xUnit.net that they were built against. We made a decision to push out the 1.0 release in order to better support the idea of having test runners that were independent from any specific version of xUnit.net. We succeeded for all of our runners, save the ReSharper runner, for reasons we've already documented.

Looking Forward

Our near-term focus is going to be on refining the process of actually using the framework for Test Driven Development, especially in conjunction with Visual Studio. While tools like TestDriven.net provide a lower friction environment than others, we still believe that there is room for improvement in the day to day usage for TDD.

A lot of this work will be centered around the GUI runner. What we released in 1.0 is essentially a bare-bones runner that we labeled experimental. While we would like to know if you have issues with the runner, please be aware that we are intended to make dramatic changes to it to help support our idea of zero-friction TDD.

It is our intention, therefore, to release interim drops of the GUI runner as we make progress, hopefully on a fairly regular schedule.

Additionally, there are several releases on the horizon (especially ReSharper 4.0 and ASP.NET MVC) which will likely also result in new point releases of the framework.

For the longer term (version 2.0), what we really want now is to let the framework get greater adoption so that many more users can help provide feedback on the things they love and the things they don't, the things that just work and the things that could be tweaked. If you download and use 1.0, please visit our forums and tell us about your experiences!

Brad Wilson and I released xUnit.net 1.0 RTM today.

Acknowledgements

Many people have helped us get to our 1.0 release.

First, we want to acknowledge the inventors of the core ideas behind xUnit frameworks, especially Kent Beck and Eric Gamma, without whom we probably wouldn't have anything to talk about today. :)

Second, we want to thank the people who hatched the idea of a new framework with us, including Peter Provost, Brian Button, Scott Densmore, Jonathan Wanagel, Jamie Cansdale, and Drew Miller.

Lastly, our community provided significant feedback to us during the development process by providing feedback on the web site, opening feature requests, reporting bugs, and generally helping guide the design of the framework. We especially want to thank Matt Podwysocki, Ben Hall, Harry Pierson, Scott Hanselman, Kirk Viehland, Daniel Cazzulino, Oren Eini, Jeff Brown, and Charlie Poole.

Looking Back

We release Beta 1 of our framework September 19, 2007, almost exactly 7 months ago. It took a lot longer than we thought it would to get to 1.0, especially given that we had been using the framework extensively internally for a while before that.

Our original release plans called for just a console runner, but our users quickly educated us on the value of variety. We personally are TestDriven.net users, so that became our second runner; our community quickly followed on asking us to support ReSharper and to write a stand-alone GUI runner. Our time spent with the console runner in an automated environment then drove us to write an MSBuild task which provides much better feedback on the testing process.

Extensibility was a very important feature for us. We felt that there was a lot to be desired in the landscape of extensibility for test frameworks. An important early decision we made was to push as much of our "functionality" as we possibly could out into extensions so that we were forced to exercise and improve our extensibility points.

Another unexpected issue was around test runners and their tight coupling to the version of xUnit.net that they were built against. We made a decision to push out the 1.0 release in order to better support the idea of having test runners that were independent from any specific version of xUnit.net. We succeeded for all of our runners, save the ReSharper runner, for reasons we've already documented.

Looking Forward

Our near-term focus is going to be on refining the process of actually using the framework for Test Driven Development, especially in conjunction with Visual Studio. While tools like TestDriven.net provide a lower friction environment than others, we still believe that there is room for improvement in the day to day usage for TDD.

A lot of this work will be centered around the GUI runner. What we released in 1.0 is essentially a bare-bones runner that we labeled experimental. While we would like to know if you have issues with the runner, please be aware that we are intended to make dramatic changes to it to help support our idea of zero-friction TDD.

It is our intention, therefore, to release interim drops of the GUI runner as we make progress, hopefully on a fairly regular schedule.

Additionally, there are several releases on the horizon (especially ReSharper 4.0 and ASP.NET MVC) which will likely also result in new point releases of the framework.

For the longer term (version 2.0), what we really want now is to let the framework get greater adoption so that many more users can help provide feedback on the things they love and the things they don't, the things that just work and the things that could be tweaked. If you download and use 1.0, please visit our forums and tell us about your experiences!

patterns & practices Summit

It time for another patterns & practices Summit. This time the summit will be in Quebec City, Canada, May 6-8th. If you are interetsed in attending click here for details. I will be presenting two sessions.

• Day 3 Keynote - I have some thoughts percolating on what to talk about. The one that keeps coming up the most is around simple design.
• Lessons Learned in Programmer Testing - This session covers some of the key lessons learned using NUnit over the past 5 years.

I will be posting the presentations and any source code here once the event is completed. I hope to see you there.

xUnit.net v1.0(RC2) Released

Brad Wilson and I released xUnit.net V1.0(RC2) today. Brad has a great summary on his blog.

You can download the release from CodePlex. Click here for the release page.