Mark’s Testblog

It’s a fairly common occurrence — somebody wants to use NUnit’s data driven testing, but they want to vary either the action under test, or the expectation.  I.e. they’re not parametrising simple data, they’re parametrising the actions.

You cannot encode these things via normal data-driven testing (short of doing really nasty things like passing string names of methods to be invoked or using enums and a dictionary of methods) and even if you use a hackish workaround, it’s unlikely to be flexible or terse.

Test readability is paramount, so if you have some tests written in an unfamiliar style, it is very important to express the intent clearly, too.

NUnit’s data-driven testing

NUnit uses a few mechanisms to parametrise tests.  Firstly, for simple test cases, it offers the [TestCase] attribute which takes a params object[] array in its constructor.  Each argument passed to the TestCaseAttribute’s constructor is stored, ready for retrieval by the framework.  NUnit does the heavy lifting for us and casts/converts each argument to the test method’s parameter types.  Here’s an example where three ints are passed, then correctly mapped to a test method:

The main limitation here is that we can only store intrinsic types.  Strings, ints, shorts, bools etc.  We can’t new up classes or structs because .NET doesn’t allow it.  How the devil do we do something more complicated?

Passing more complicated types

It would appear we’re screwed, but fortunately, we can use the [TestCaseSource] attribute.  There are numerous options for yielding the data, and one of them is to define an IEnumerable<TestCaseData> as a public method of your test class (it works if it’s private, but since it’s accessed via reflection it’s a good idea to keep it public so that ReSharper or other tools do not flag it as unused).  You can then fill up and yield individual TestCaseData instances in the same fashion as before.  Once again, NUnit does the mapping and the heavy lifting for us.

If you do not require any of the fancy SetDescription, ExpectedException etc. stuff associated with the TestCaseData type, you can skip one piece of ceremony by simply yielding your own arbitrary type instead (i.e. change the IEnumerable<TestCaseData> to IEnumerable<MyType> and then simply yield return new MyType()).

Passing a delegate as a parameter (simple)

The simplest case is that you want to vary which methods are called.  For example, if you have multiple types implementing the same interface or multiple static methods, encoding which method to call is very simple.

Here’s an example from Stackoverflow that I answered recently where the author wanted to call one of three different static methods, each with the same signature and asserts.  The solution was to examine the method signature of the call and then use the appropriate Func<> type (Funcs and Actions are convenience delegates provided by the .NET framework).  It was then easy to parametrise the test by passing in a delegates targeting the appropriate methods.

More advanced applications

Beyond calling simple, stateless methods via delegates or passing non-intrinsic types, you can do a lot of creative and cool stuff.  For example, you could new up an instance of a type T in the test body and pass in an Action<T> to call.  The test body would create an instance of type T, then apply the action to it.  You can even go as far as expressing Act/Assert pairs via a combination of Actions and mocking frameworks.  E.g. you could say “when I call method X on the controller, I expect method Y on the model to be called”, and so forth.

The caveat is that as you do use more and more ‘creative’ types of data-driven testing, it gets less and less readable for other programmers.  Always keep checking what you’re doing and determine whether there is a better way to implement the type of testing you’re doing.  It’s easy to get carried away when applying new techniques, but it’s often the case that a more verbose but familiar pattern is a better choice.

When the logic of a test method remains constant but the data varies, data-driven testing is a great tool.  It allows you, the test author, to write compact code and to add new test cases rapidly.  Unfortunately, data-driven tests have a disadvantage: The inputs are often less readable.

A simple example

Let’s take an example; testing Rob Conery’s PagedList implementation.  A page is basically a slice of the data returned by a linq query.  If more data exists beyond the ‘slice’ represented by the PagedList<T> instance, its “HasNextPage” property should return true to indicate that it is available.  Now, suppose we want to test whether a particular page has a next available page.  Three things spring to mind that can influence the result: The page size, the current page index and the number of items in the list.

Here’s a quick data-driven test for HasNextPage:

As you can see, the method itself is readable, but the parametrised values fed in via the [TestCase] attribute are not.  It’s really hard to keep everything in your head and remember what each number maps to in the function, especially when the method parameter types are all identical.  If you have a list of 20 [TestCase] attributes, you start wondering what’s for your tea and forget that the second value is the (checks image) page size.  Mince.  Mince for tea.

Hmm, if only we could those TestCases more readable; something like object initializers would be ideal.

A simple trick: Subclass TestCase

My friend Hughel helped me come up with this one and it works quite well.  Simply subclass the TestCaseAttribute class and add your own properties to represent the test parameters.  It gets a little bit hairy when you have to access the Arguments array directly (especially since Attributes can be weird), but in practice, it works fine.   In most of the tests, we’re only interested in parametrising three things, so it’s simple to add them as properties.

The end result

Finally, we apply these attributes to our data-driven test, significantly improving the readability!

I would hasten to add that I don’t recommend using this approach willy-nilly — only when you have a large amount of tests that are parametrised by the same data types, causing the test cases to become hard to follow.  I’ve used the [TestCaseSource] attribute and the [TestCase] attribute a lot in the past and most of the time it’s not a problem.

I’ve recently been tinkering with WCF and, as many people already know, writing data transfer objects is a pain in the balls.  Nobody likes writing repetitive, duplicate and tedious code, so I was delighted when I read about AutoMapper.  It works really nicely;  with convention over configuration, you can bang out the entity => data transfer code in no time, the conversions are less error prone, the tests stay in sync and you’re left to concentrate on more important things.

Anyway, I immediately realised that I’ve used the same pattern in testing — with property initializers & test data builders.  I’ve posted before about Test Data Builders and I’d recommend you read that post first.

For small test data builder classes, it’s really not that big a deal.  For larger classes, using AutoMapper is quite useful.  For example, for testing purposes we’ve got an exception details class that is sent over to an exception logging service.

Every time the app dies, we create an exception data transfer object, fill it out and then send it over the wire.  When unit testing the service, I use a Test Data Builder to create the exception report so that I can vary its properties easily.  Guess what?  The test data builder’s properties map 1:1 with the exception report — hmm!

So, rather than create the same boilerplate code to map the 10+ properties on the exception builder => data transfer object, I just used AutoMapper to handle the mapping for me :)

public class ExceptionReportDto
{
    public string ExceptionType { get; set; }
    public string StackTrace { get; set; }
    public string AssemblyName { get; set; }
    public string EntryPoint { get; set; }
    public string UserName { get; set; }
    public string MachineName { get; set; }
    // etc
}

public class ExceptionReportBuilder
{
   public string ExceptionType { get; set; }
   public string StackTrace { get; set; }
   public string AssemblyName { get; set; }
   public string EntryPoint { get; set; }
   public string UserName { get; set; }
   public string MachineName { get; set; }
   // etc

// create the mapping when the static ctor is invoked
static ExceptionReportBuilder()
}
   Mapper.CreateMap<ExceptionReportBuilder, ExceptionReportDto>();
}

public void ExceptionReportDto()
{
    // set up defaults
    ExceptionType = "System.ArgumentException";
    StackTrace = "Oh no I am a stack trace";
    //etc.
}

 public ExceptionReportDto Build()
 {
     // go go automagic!
     return Mapper.Map<ExceptionReportBuilder, ExceptionReportDto>(this);
 }
}

I’ve had good results with this approach.  The only bit I’m remotely concerned about is creating the mapping in the static constructor.  Any AutoMapper gurus out there who can say whether there’s any reason I shouldn’t do that?

One of the things that really slashes the return on investment in testing is unreadable code.  “This is pretty obvious”, you say.  “What’s the point in a blog post about something so obvious?”  What’s not obvious is that the very people writing these tests are unaware of it.  Maybe you do it as well.  Given that this blog post is about things we don’t know we do, I think it’s a fair bet that I’ve also recently written test code that was convoluted without realising it, too.

It’s mostly down to testing in a vacuum.  The tests are often functionally fine.  However, as with nearly all code, maintenance is easily overlooked.  On the day that the test was written, it made sense to the author.  They understood the logic they wanted to test and how to implement it.  Code written, job done.

In the court of the test engineer, readability is king

In my opinion, readability in tests is the number one thing.  If your fellow programmers and your future self cannot decipher what a test is proving, the test becomes worthless.  If a monstrously bearded mathematics genius solved solved P versus NP but failed to write an understandable proof, it would all be for nothing.

It’s the same for testing.  You’ll know the shit has hit the fan when you’re refactoring something and break a load of tests.  “Balls.  I’ll fix the tests I guess”, you murmur, bleary eyed and idiot-faced.  However, when you examine the tests, you can’t figure out what they’re proving, why they’re proving it or how it’s actually proved!

You know in films when a bad guy is holding a hand grenade, then has a moment of realisation regarding an absent pin?  He slowly looks up, face furnished with a quizzical look, staring into oblivion.  That’s what a software engineer looks like when they encounter reams of broken tests that cannot be deciphered (and the person who wrote ‘em isn’t available to pick up the pieces).

Grab a friend and test your tests

To avoid these kinds of bomb-scares, do yourself a favour and have somebody else trawl through your test code without giving them any help.  If they cannot easily follow the test code’s intent, then the test needs re-written.  Everybody’s code can be improved through peer review.  Peer review is the litmus test — if somebody else cannot understand it, it’s useless.  As such, tests should be part of code reviews and buddy processes.

Think about how much time we spend reading, maintaining, refactoring and extending old code compared to writing new code, and then think about how self-defeating it is to write unreadable, un-reviewed test code.  Spending a little more time on reviewing new test code pays off in the long run.

There is a bewildering array of types of ‘mock’ object available to a tester.  The canonical list of test doubles was probably coined by the venerable Martin Fowler in his article “Mocks Aren’t Stubs” and, to me, this list is fairly complete and makes sense.  The reason it makes sense is that I’ve manually written classes that perform these roles.

• I needed to fill out a parameter list with non-null objects, so I created a dumb class with absolutely no implementation.
• I wanted to listen in on additions to a list, so I wrote a class that stored the objects, acting as a spy.
• I wanted to provide canned results to test another part of my system, so I wrote a stub.
• I needed to ensure a method was called, so I wrote a Mock.
• I needed a coherent, fast implementation of a class, so I wrote a fake implementation.

My problem with testing terminology is that it’s harmful to newcomers, especially when the semantics affect the result of the test!  Not only is there a high barrier to entry when it comes to writing accurate, robust and maintainable tests, but the terminology is another unwelcome complication.

For this reason, I personally advocate keeping new testers away from mocking frameworks until they’ve become comfortable with state-based testing and hand-rolled a variety of their own test doubles.

Information Overload

Even when no additional frameworks are involved (i.e. when using vanilla xUnit), writing good unit tests involves a steep learning curve.  It’s easy to take a wrong turn and the quality of the tests written will improve only with experience/guidance.  I was not surprised when I read Roy Osherove’s blog and discovered that the majority of organisations’ attempts to embrace unit testing resulted in failure.

Mocking frameworks like Rhino Mocks are absolutely excellent tools, but it’s yet another thing to learn.  Suddenly the type of object (mock, strict mock, stub etc.) affects the result of the test.  It took me a week to get my head around it, so it doesn’t surprise me when I see newcomers totally abusing these frameworks.  Not only does this create a maintenance nightmare, but it sours their first taste of testing.

The problem is doubly hard to tackle, as Mocking frameworks allow you to write the same tests with fewer lines of code.  Developers who are new to testing see this and immediately try to use the mocking framework.  After all, only a fool would eschew such benefits, right?  Well, some people just ‘get it’ from the start.  I’m not one of these people and, in my experience, nor are most others.  Starting with interaction based testing and mocking frameworks is akin to throwing someone out of the back of a van that’s moving at 70mph and expecting them to start running when they hit the tarmac.  Chances are they’re going to land on their face.

The unfortunate result is that some folk tie themselves in knots.  They don’t understand the responsibilities of each type of testing object and, as a result, create extremely brittle or utterly pointless tests. If you have no idea of what you’re trying to achieve with a test, there is no point in writing it.  I once saw a question on StackOverflow featuring a confused fellow asking why his test did not work.  The poster was using Rhino Mocks and created a Mock of the class under test.  I.e. it was not a collaborator, it was the class under test and he was mocking it!  I suspect he was simply overwhelmed by trying to learn multiple new things.

Other things I’ve witnessed include developers writing obscure lambda expressions using c#, then chaining together RhinoMocks methods to perform something which somehow works.  When I pointed out that I could barely infer its purpose by reading the code and that writing a hand rolled stub would probably be a better idea, I was met with “yes, you’re probably right but I want to use Rhino Mocks”.

Warning bells should also start ringing when you return a mock and assert that its method was called by another mock which returns a mock which… errrr!  It’s much easier to make a dog’s dinner of interaction-based testing; a good grounding in state-based testing is essential.

One step at a time

1. Learn to sit up before you crawl.  Write simple xUnit tests that involve state-based testing.  It doesn’t have to be great, isolated code.  Even writing tests that involve scores of classes is a good way to start.  Finer granularity is something that comes with experience.
2. Crawl before you walk.  Start to experiment and find better ways of testing pieces of functionality.  Ask yourself whether the test is useful, maintainable etc.  Will other parts of the system break it if they change?  Can you make the components and tests themselves finer grained?  This stage should be about developing your sense of what constitutes a good test.
3. Walk before you run.  Begin to experiment with different types of test doubles, but hand roll them.  Yes, it’s painful at times, but it will give you a better understanding of roles in tests and the different types of test doubles, even if you don’t have names for them yet.  Furthermore, constantly having to update your hand rolled stubs when disparate parts of your class changes will also give you an appreciation for the interface segregation principle.
4. Finally, install a mocking framework and start sprinting.

If you do sprint head-first into a wall, you will be better equipped to understand where you went wrong, as you understand the fundamentals.  You will also have a better grasp of the terminology, as it will be grounded in real, tangible code you’ve written.

One of the things I try to encourage is the careful selection of names.  Just as self-documenting code is easier to read, so is a self-documenting test.  As I have previously stated, unit testing is programming, too — you can apply the same good practices to tests.

On numerous occasions I’ve had to review some code and tests.  I open the classes and find well-named, self-documenting, loving crafted, carefully designed code.  Then I look at the test for that code and find that the same principles have not been applied to the tests.  In fact, it’s almost like the tests have been written by Evil Chuck, the programmer’s alter-ego.

It’s not uncommon to see something like the hugely descriptive and easy to read:

[Test]
public void TestMethodName()
{
....
}

.. the excellent

[Test]
public void TestMethodName4()
{
....
}

.. and the ubiquitous

[Test]
public void TestConstructor()
{
....
}

This is not a good state of affairs!

Problems with badly named tests

Numerous problems exist with a name like “TestSomeMethod4B”.  Here are a selection of good reasons not to name your tests like this:

Readability
It’s not descriptive.  In fact, it’s totally obscure.  It might as well not exist.  What can you say about “TestSomeMethod4B”? Nothing much.  Even if it is documented with a comment, it hinders readability in the IDE and the test runner.  It’s better to name something descriptively and without a comment than the other way around.  That’s not to say that comments and descriptions are redundant, but in most cases you don’t need them if the test is descriptively named.

Intent
It says absolutely nothing about the intent.  It might be checking an exception is thrown, it might be checking a value is clamped to an accepted range of values.  It might be doing nothing.  To be able to decipher its intent, you have to read the code.  Imagine if you couldn’t understand the intent of any part or method of a program.  How would you manage to break it up into understandable chunks?  Answer: with great difficulty.

Obfuscation
It defeats any kind of attempt to understand the state and thoroughness of the testing for that class as a whole.  You cannot obtain an overview.  You can’t determine whether you’ve tested a method with 10 different invalid parameters or whether you’ve tested 10 different simple cases.  Without good naming you are reduced to skimming the code while trying to remember too much.  Just as well named subroutines aid comprehension of a larger problem, well-named tests give a good overview without forcing you to examine the contents of those tests.

Redundant Prefix
In 99% of cases, if a public method is part of a test fixture and it has a [Test] attribute, it’s a test.  You don’t need the Test prefix.  It just hinders the skimming of the tests in alphabetic order.  If you really insist on putting “Test” somewhere, put it at the end of the name.  I personally wouldn’t bother, though.

What are you doing?
Finally and arguably most importantly, if you don’t name your test well, there is a greater chance that you don’t know what the test is trying to achieve. Would you start to write a production code method without any inkling as to what it did?  Even if you did, would you then leave it in existence with the name “DoSomeStuff”?

Badly named tests are a smell.  In my experience, well-named tests are nearly always attempting to prove something regardless of the quality of the test body.  I cannot say the same for badly named tests.

Good intent and bad execution is often better than a tidy, aimless test.  The former can be refactored into something useful, the latter requires decryption just to understand why it exists in the first place.  In many cases, the test proves nothing.

If you write a test method and can’t think of a name that describes the test before you write it, stop.  Think about what you’re trying to achieve, then choose a name that describes it adequately.

How do I choose a good name?

First and foremost, what are you trying to prove with the test?  Tests are meant to demonstrate something.  They are meant to assert that some meaningful state is set, or some sort of interaction has taken place.

You need at least three pieces of information to name a test.

1. The thing that is being tested, such as a function, method or property (or a sequence of them)
2. The arguments/data/circumstance involved
3. The expected outcome.  What is meant to happen?  Be as explicit as you can.

That’s it.  That’s all you need.  I prefer to write mine in the format 1_2_3, but I’m sure everyone has their own personal style.  As long as it’s readable and consistent, I don’t care.

Compare and contrast

Here’s a few examples of some good test names for a bounding box class.

• AddPoint_ValidPointOutsideExistingBounds_IncreasesBoundsToContainPoint()
• AddPoint_ValidPointInsideExistingBounds_DoesNothing()
• AddPoint_ValidPointOnEdgeOfBounds_DoesNothing()
• AddPoint_InvalidPointContainingNaN_DoesNothing()
• AddPoint_InvalidPointNull_ThrowsException()

Now compare those to equivalent, but badly named tests:

• TestAddPoint()
• TestAddPoint2()
• TestAddPoint3()
• TestAddPointBad()
• TestAddPointBad2()

One set is descriptive, easily graspable and allows you to skim the members list to get a good feel for the thoroughness of the tests.  The other set of names tells us very little.

To those who say “I don’t like long method names”, I say, “It’s a test.  You write it once and read it hundreds of times.  You never have to call into it from other code.  Your screen is plenty wide enough to accomodate it.”  There is no reason to use short or bad test names ‘just because’.  I’ve yet to hear any meaningful criticism against giving test methods long names.

As the famous nerd quote goes (paraphrasing):

“When I wrote this code, only God and I knew what I was doing.  Now only God knows”.

Just think about the poor sod who has to maintain your code in a couple of years. If you didn’t know what the hell you were doing, what are they going to make of it?

Other Advantages

If I have some good ideas about how to test something, or if I’m writing my tests before the production code, I will often use the 1_2_3 naming system to write out scores of empty test bodies.  You may be surprised to see how effective this is.

You can plough through a group of methods in no time at all, thinking of all of the horrible things that could go wrong and writing them down.  In no time at all, you can have a comprehensive test suite in waiting.  The names are there, the intent is clear and you can proceed.

This is also a great tactic for division of labour when you’re testing old code.  Prototype the test bodies, check in the skeleton fixture(s) and multiple people can get cracking on different areas.  I’ve done this quite successfully in the past.