Testing for Exceptions in Unit Test Frameworks

In MSTest and nUnit, the typical way to test for exceptions is by using the [ExpectedException] attribute on a test method.  I’ve always found this to be a little annoying, but it was just recently that I decided to do something better.  Unsurprisingly, I’m not the first one to have this thought; xUnit uses this method of exception testing by default and a Google search revealed that a few others had posted about this as well.  It still makes for a decent blog post, though, so I figured that I would toss this up for everyone to check out.

The problem

Pure unit test philosophy says that each test method should test one and only one condition.  I tend to agree with this, but sometimes, it can be a bit cumbersome.  For example.  Let’s say we’re testing the indexer on a 2D array.  An obvious test would be to make sure that an IndexOutOfRangeException is thrown when a user tries to use an out of range index.  For a 2D array, though, there are quite a few ways for this to happen.  Let’s take the lower boundaries to start:

[TestMethod]
[ExpectedException(typeof(IndexOutOfRangeException))]
public void TestIndexerWithExpectedException()
{
    var obj = new int[5, 5]; 
    var tmp = obj[-1, -1]; 
}

[TestMethod] 
[ExpectedException(typeof(IndexOutOfRangeException))] 
public void TestIndexerWithExpectedException1() 
{ 
    var obj = new int[5, 5]; 
    var tmp = obj[0, -1]; 
}

[TestMethod] 
[ExpectedException(typeof(IndexOutOfRangeException))] 
public void TestIndexerWithExpectedException2() 
{ 
    var obj = new int[5, 5]; 
    var tmp = obj[-1, 0]; 
}

As you can see, this is actually quite a bit of code to test something that should be pretty simple.  We just wanted to test to make sure that the indexer throws an exception if you go off the low end, but we had to write 3 functions to do this, all with a lot of repeated code (even if we extracted the construction of obj out of each method).  Now think about what happens when we try to test all of the boundaries of the array to make sure that they throw exceptions.  If you think of the array as a square grid, each corner has 3 different ways to go off the edge and cause an IndexOutOfRangeException.  That means that we have 12 different ways to cause an error, so at the very least, we have 12 different test methods.  Yuck.

The ExpectedException approach also has another obvious problem:

[TestMethod] 
[ExpectedException(typeof(IndexOutOfRangeException))] 
public void UnfortunateResultFromUsingUnexpectedException() 
{ 
    var obj = new int[5,5];

    //initialize array... 
    obj[5, 5] = -1;//random mistake while initializing the array 

    //more initialization code would go here...

    // this code was the real test, 
    // but it doesn't get executed...            
    var tmp = obj[-1, -1];
}

Though the above code is a small example, it illustrates the danger of using the ExpectedException approach.  ExpectedException says that somewhere in your test method, the test will throw an exception, but it doesn’t specify where that will happen.  That’s not fine grained enough for a good unit test.  As you can see above, fairly innocent initialization code can cause your well intentioned test to pass or fail for the wrong reasons.

A Better Approach?

One approach to remedy this problem is to use try/catch blocks with some sort of "exception thrown" flag:

[TestMethod] 
public void TestIndexerWithTryCatch() 
{ 
    var obj = new int[5,5];

    // lots of ugly try/catches... 
    // not very nice to look at, but
    // provides more fine grain control within a test. 
    bool exceptionThrown = false; 
    try 
    { 
        var tmp = obj[-1, -1]; 
    } 
    catch (IndexOutOfRangeException) 
    { 
        exceptionThrown = true; 
    } 
    Assert.IsTrue(exceptionThrown);

    exceptionThrown = false; 
    try 
    { 
        var tmp = obj[-1, 0]; 
    } 
    catch (IndexOutOfRangeException) 
    { 
        exceptionThrown = true; 
    } 
    Assert.IsTrue(exceptionThrown);

    exceptionThrown = false; 
    try 
    { 
        var tmp = obj[0, -1]; 
    } 
    catch (IndexOutOfRangeException) 
    { 
        exceptionThrown = true; 
    } 
    Assert.IsTrue(exceptionThrown); 
}

This approach gives you the benefit of allowing you to control where you are testing for the exception to occur.  You get the fine grained control you want in a unit test, but unfortunately, this makes your code ugly and bloated.  It’s not a bad compromise, but we can definitely do better.

A Better Approach 2.0

While it was nice to have fine grained control, it was a pain to write.  What would really be nice is if our unit test library provided something like this:

[TestMethod] 
public void TestIndexerUsingExceptionAssert() 
{ 
    var obj = new int[5, 5];

    // assertion version of the try/catch syntax...
    // much less repeated code, no need for individual functions. 
    MyAssert.ThrowsException<IndexOutOfRangeException>(() => { var tmp = obj[-1, -1]; }); 
    MyAssert.ThrowsException<IndexOutOfRangeException>(() => { var tmp = obj[0, -1]; }); 
    MyAssert.ThrowsException<IndexOutOfRangeException>(() => { var tmp = obj[0,0]; }); 
}

Now we have what we really want.  We have the fine grained control that the "try/catch with a flag" approach gave us, but it’s much easier to read/write, and we don’t have to repeat so much code.  xUnit supports this style of exception testing by default (see Assert.Throws), but if you don’t want to switch to xUnit, it’s very easy to implement yourself:

/// <summary> 
/// Contains assertion types that are not provided with the standard MSTest assertions. 
/// </summary> 
public static class MyAssert 
{ 
    /// <summary> 
    /// Checks to make sure that the input delegate throws a exception of type exceptionType. 
    /// </summary> 
    /// <typeparam name="exceptionType">The type of exception expected.</typeparam> 
    /// <param name="blockToExecute" />The block of code to execute to generate the exception.</param> 
    public static void ThrowsException<exceptiontype>(Action blockToExecute) where exceptionType : System.Exception 
    { 
        try 
        { 
            blockToExecute(); 
        } 
        catch (exceptionType) 
        { 
            return; 
        } 
        catch (Exception ex) 
        { 
            Assert.Fail("Expected exception of type " + typeof(exceptionType) + " but type of " + ex.GetType() + " was thrown instead."); 
        }

        Assert.Fail("Expected exception of type " + typeof(exceptionType) + " but no exception was thrown."); 
    } 
}

And there you have it- a better way to test for exceptions.  Nothing revolutionary, I know, but it never hurts to have more example code for folks to look over.

This entry was posted in C#, Unit Testing. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

One Comment

  1. Lee
    Posted October 28, 2008 at 8:32 pm | Permalink

    Thanks Chris, this is exactly the example I needed!

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*
*