.NET’s version 5 of the C# compiler introduced support for an interesting feature related to multithreaded programming, available via the new async and await keywords.

This post is not much about the feature itself as plenty of information is available online, but rather the support that was introduced for it in NUnit. Here is a simple NUnit test:

1
2
3
4
5
6
7
8

[Test]
public void OneSimpleTest()
{
    var eightBall = new EightBall();
    var answer = eightBall.ShouldIChangeJob();

    Assert.That(answer, Is.True);
}

So far so good, but let’s assume that invoking that method on the 8 ball is really expensive in terms of execution time and you would like your production code invoking it to not just sit and wait until it completes. That’s where the async feature fits, and you could change the method to be an async, Task-returning one instead. How to make that specific method asynchronous is definitely out of scope here though.

Testing asynchronous code

In order to call the new method asynchronously the test code would need to be adapted as shown here:

1
2
3
4
5
6
7
8
9
10

[Test]
public async void OneSimpleTest()
{
    var eightBall = new EightBall();
    var answer = await eightBall.ShouldIChangeJob();

    Assert.That(answer, Is.True);

    // why am I still here?
}

Awaiting the result of the method implies that the test method has to be marked as async, too. Now, assuming that you’re not planning to change job you would really expect the test to fail, which it instead doesn’t if run with a version of NUnit older than 2.6.2.

The reason behind this incorrect behavior is that asynchronous methods are rewritten by the C# compiler in a way that they return control to the caller as soon as they await on an operation which has not completed yet, as is the case with the invocation of the ShouldIChangeJob method. In this case therefore the control is returned to the code which called the OneSimpleTest method, which turns out to be NUnit itself.

Up to version 2.6.2 of NUnit the test runner always assumed that as soon as a test method returned then its execution was complete, and thus if no assertion failures were reported the test was successful. This is no longer the case with asynchronous methods, as we just found out, because the now asynchronous test returns immediately after invoking ShouldIChangeJob on line 5.

Without any support from NUnit there was a simple fix available already, which was to Wait on the Task returned by the asynchronous method rather than await it.

1
2
3
4
5
6
7
8
9

[Test]
public void OneSimpleTest()
{
    var eightBall = new EightBall();
    Task<bool> answer = eightBall.ShouldIChangeJob();
    answer.Wait();

    Assert.That(answer.Result, Is.True);
}

This would work perfectly, although it’s not as nice as being able to write the test code in the same way you would write the production code, that is, by using the await keyword. This has the additional disadvantage of exception handling behaving differently between await and Wait. When using await the code looks like everything is happening sequentially, and you can catch exceptions in the same way you do with sequential code. If ShouldIChangeJob threw an exception of the fictitious type TimeoutException you would see exactly that exception bubbling up from your code. When using Wait instead a System.AggregateException would bubble up, which is a sort of container for multiple exceptions which might have happened during an asynchronous operation, and would wrap the real TimeoutException. This of course makes await more intuitive and preferable over Wait.

Just a matter of Waiting

Introducing this support in NUnit doesn’t necessarily mean that NUnit has now become asynchronous or that it allows to run tests in parallel. This is something which is being worked on for the next 3rd major release of NUnit though.

What we did in NUnit 2 was to allow users to write async tests without worrying about tests completing before their assertions were even evaluated, in addition to supporting thorough use of asynchronous methods in some specific framework use cases. In other words, when invoking asynchronous test methods NUnit will “sit and wait” on your behalf, until the test method, along with all the asynchronous operations it invokes, has finally completed.

Behind the scenes

As should be clear by now NUnit’s support for async methods is mainly a matter of detecting async methods, calling Wait on the Task returned by them and handling exceptions accordingly. This is true in most cases, although NUnit’s need to be compatible with .NET 2.0 means that all of this logic needs to be implemented without referencing any .NET > 2.0 assemblies.

Yet this was still not as straightforward, but we really strived to open up as many intuitive usages of await/async as possible so to relieve the users from having to even think about it.

Although you might not have noticed it you’ve already encountered one example of why this is not trivial. In the asynchronous sample above you can see that the test method is void. What does it mean exactly? Well, it means that there is no Task on which to call Wait, and that really waiting for it to complete means setting up a new SynchronizationContext to hook into the .NET implementation of the async/await feature.

Although asynchronous void methods were not designed for this purpose and Microsoft itself discourages using them besides in event handlers, we really wanted to support them for a couple of reasons:

1. Not supporting them would mean that NUnit would throw an exception at runtime every time a test is written with an async void signature
2. Most users would probably repeatedly commit the same mistake of writing asynchronous test methods as void async because no one really cares about their return value

We thought that the two reasons above would lead to a frustrating user experience and decided to support async void tests as well as Task-returning ones. This choice came with some drawbacks too:

1. The ways async void methods interact with the SynchronizationContext in which they run is undocumented and might change in the future, which opens up the possibility of breaking NUnit’s implementation
2. Switching the SynchronizationContext in which the test method runs (the way NUnit supports async void tests, that is) might affect even up to a great extent the flow of the code wrapped by the new SynchronizationContext

We though that these two issues alone did not justify the poor user experience and we opted to fully support async void tests.

If not void, what?

Now, although it seems quite reasonable to write the above asynchronous test as one returning void it is equally valid to write it so that it returns a Task instead.

1
2
3
4
5
6
7
8

[Test]
public async Task OneSimpleTest()
{
    var eightBall = new EightBall();
    var answer = await eightBall.ShouldIChangeJob();

    Assert.That(answer, Is.True);
}

It is actually more advisable to do so if you are especially interested about forwards compatibility of your code for the reasons explained above. If you do this NUnit will not have to jump through hoops to run it correctly and will simply fall back to waiting on the returned task.

async, where else

As briefly mentioned earlier support for await/async in NUnit 2 goes beyond test methods. The good thing about it is that in most cases you won’t have to think about whether it is supported or not, because it most likely is.

In any case here’s a by-no-means complete overview of places where we had to do some relevant work.

Test cases checking results via Task return values

Methods marked with TestCaseAttribute can return values, which are then checked against the Result property of the test case. They now support returning result asynchronously:

1
2
3
4
5
6
7
8
9
10

[TestCase(1, 2, Result = 3)]
public async Task<int> TestAddAsync(int a, int b)
{
    return await SumAsync(a, b);
}	

public async Task<int> SumAsync(int a, int b)
{
	return await Task.FromResult(a) + await Task.FromResult(b);
}

Async lambdas

The async support went slightly beyond test methods, and extended to framework features which accept lambda expression as their arguments:

1
2
3
4
5
6
7
8
9
10
11
12

[Test]
public async Task AsyncLambaSupport()
{
    // throwing asynchronously
    Assert.That(async () => await ThrowAsync(), Throws.TypeOf<InvalidOperationException>());

    // returning values asynchronously
    Assert.That(async () => await ReturnOneAsync(), Is.EqualTo(1));

    // "After" works with async methods too
    Assert.That(async () => await ResutnOneAsync(), Is.EqualTo(1).After(100));
}

Framework support is not yet available in NUnit 2.6.2, it will be in the next build.

Test context availability

If you don’t know about TestContext I suggest you check it out as it might come handy in a bunch of scenarios. If you’re already using it, just be aware that it is now accessible anywhere inside the body of asynchronous tests, which is how you would expect it to be.

What else?

This feature has been ported to the upcoming major release 3.0 of NUnit as well as to NUnitLite to make it available consistently throughout the whole testing platform.

If there’s anything we’ve missed feel free to let us know on the NUnit’s mailing list.