Category Archives: testing - Page 2

The fundamentals of unit testing : Narrow & Focused

When it comes to writing unit tests, it pays to be specific.  

If you have a narrow & focused test, it will…

  • be easy to make a descriptive name for the test
  • perform the bare minimum of work required to achieve its goal. 
  • be easy to understand
  • be easy to maintain 
  • execute in a shorter period of time

An unfocused example

To give you an idea of what an unfocused test looks like, consider the following (and yes, I’ve seen plenty of these, both in professional [RTW] & open source code):

public void StackTestA()
        var stack = new Stack<int>();
        Assert.That(stack.Count, Is.EqualTo(0));

        Assert.That(stack.Count, Is.EqualTo(1)); 

        Assert.That(stack.Count, Is.EqualTo(2)); 

        var p = stack.Peek();
        Assert.That(p, Is.EqualTo(2);
        Assert.That(stack.Count, Is.EqualTo(3)); 
        Assert.That(stack.Count, Is.EqualTo(2)); 
        bool c0 = stack.Contains(1);
        Assert.That(c0, Is.True);
        bool c1 = stack.Contains(10000);
        Assert.That(c1, Is.False);
        Assert.That(stack.Count, Is.EqualTo(3)); 

        int popped = stack.Pop();
        Assert.That(popped, Is.EqualTo(3));
        Assert.That(stack.Count, Is.EqualTo(2));

Can you spot the problems? 

Read more »

The fundamentals of unit testing : It’s a skill

The first thing I’m writing about is probably the most important.  This is a bit of a meandering tale, but it is crucial to understanding the pitfalls of automated testing.

A Cautionary Tale

Back in the day at Realtime Worlds (the best part of 5 years ago), when it came to automated testing we were trying to pull ourselves up by our bootstraps.  Most people (including myself) didn’t have much of a clue about automated testing.  It was a pretty bold move for a games company to try it on a large scale, but it took a very long time for it to pay any benefits.  In fact, given the lackadaisical approach and lack of buy-in, it probably had an overall negative effect in the majority of areas.

In my first job fresh out of university, I had the nominal title of Junior Test Engineer.  At first, it was our job to try and write extra tests that developers missed, debug failing tests and generally take care of the build.  I soon learned to be very cynical.

As you can imagine, it wasn’t very effective (hint: the larger the distance between a person’s actions and the consequences of said actions, the less of an interest they will take).  I would characterise it as a baptism of fire.  Thanks to spending all day, every day, on automated testing, I acquired a large amount of knowledge by trial and error and learning with my colleagues.  That knowledge was eventually ploughed back into the team in a supporting role.

Read more »

The fundamentals of unit testing (series)

I’ve not written about testing for a while, but I still enjoy writing automated tests (I am boring) and trying to coach people on good testing practices (I am boring), so hopefully this series will be a useful refresher.

I plan to write roughly ~15 posts on the fundamentals of good unit testing.  By good, I mean useful, trustworthy and maintainable.  These tips are for new through to intermediate developers; they’re not designed to catch the attention of seasoned automated testers, but there may be the odd nugget that’s useful for older hands.

Here’s the list of things I’m going to cover (I’ll update the links as I add ‘em):


General principles to strive for:

General testing tips:

Data-Driven Testing with Python

Hello tests.  It’s been a while since I blogged about automated testing, so it’s nice to welcome an old friend back to the fold.  I’ve missed you.  I’ve recently started programming in python.  I say programming, but what I mean by that is ‘hopelessly jabbing at the keyboard and gawping as my lovingly written code explodes on being invoked’.  While I blub.

Python is a dynamic programming language.  It’s pretty dynamic in its ability to detonate in various ways, too.  There’s no compiler to sanity check my elementary mistakes like typos, or dotting onto a method that doesn’t exist.  Or accidentally comparing functions instead of values.  Or… well, you get the gist.

Anyzoom, the fundamentals of unit testing in python are very simple.  I can’t be bothered detailing how to use unittest or a test runner, as there’s nine zillion resources out there on it already. 

A Simple Setup

We’re using unittest as the framework, Nose as the runner and TeamCity Nose integration.  It worked very nicely out of the box, so I can’t complain.  Writing tests is simple, but then I wanted to make a data-driven test and…

Read more »

Multiple Mocks?

This post was in response to a post on the blog (the comments aren’t working for me, so I thought I’d post it here) which I read via Roy Osherove.

Basically, the author was querying Roy Osherove’s rule of thumb that stated, “more than one mock per test is harmful”.

He then goes on to provide an example where one action has two identifiable/observable results, and two mocks are used.

Figure 53b: Two Mocks (I’m so, so, sorry)


Read more »

Unit? Integration? Functional? Wha?

I answered a question titled, “What’s the difference between unit, functional, acceptance and integration tests?” on Stackoverflow and thought it’d be useful to re-post the answer here for future reference.

Read more »

Data-driven testing tricks

It’s a fairly common occurrence — somebody wants to use NUnit’s data driven testing, but they want to vary either the action under test, or the expectation.  I.e. they’re not parametrising simple data, they’re parametrising the actions.

You cannot encode these things via normal data-driven testing (short of doing really nasty things like passing string names of methods to be invoked or using enums and a dictionary of methods) and even if you use a hackish workaround, it’s unlikely to be flexible or terse.

Test readability is paramount, so if you have some tests written in an unfamiliar style, it is very important to express the intent clearly, too.

NUnit’s data-driven testing

NUnit uses a few mechanisms to parametrise tests.  Firstly, for simple test cases, it offers the [TestCase] attribute which takes a params object[] array in its constructor.  Each argument passed to the TestCaseAttribute’s constructor is stored, ready for retrieval by the framework.  NUnit does the heavy lifting for us and casts/converts each argument to the test method’s parameter types.  Here’s an example where three ints are passed, then correctly mapped to a test method:

The main limitation here is that we can only store intrinsic types.  Strings, ints, shorts, bools etc.  We can’t new up classes or structs because .NET doesn’t allow it.  How the devil do we do something more complicated?

Passing more complicated types

It would appear we’re screwed, but fortunately, we can use the [TestCaseSource] attribute.  There are numerous options for yielding the data, and one of them is to define an IEnumerable<TestCaseData> as a public method of your test class (it works if it’s private, but since it’s accessed via reflection it’s a good idea to keep it public so that ReSharper or other tools do not flag it as unused).  You can then fill up and yield individual TestCaseData instances in the same fashion as before.  Once again, NUnit does the mapping and the heavy lifting for us.

If you do not require any of the fancy SetDescription, ExpectedException etc. stuff associated with the TestCaseData type, you can skip one piece of ceremony by simply yielding your own arbitrary type instead (i.e. change the IEnumerable<TestCaseData> to IEnumerable<MyType> and then simply yield return new MyType()).

Passing a delegate as a parameter (simple)

The simplest case is that you want to vary which methods are called.  For example, if you have multiple types implementing the same interface or multiple static methods, encoding which method to call is very simple.

Here’s an example from Stackoverflow that I answered recently where the author wanted to call one of three different static methods, each with the same signature and asserts.  The solution was to examine the method signature of the call and then use the appropriate Func<> type (Funcs and Actions are convenience delegates provided by the .NET framework).  It was then easy to parametrise the test by passing in a delegates targeting the appropriate methods.

More advanced applications

Beyond calling simple, stateless methods via delegates or passing non-intrinsic types, you can do a lot of creative and cool stuff.  For example, you could new up an instance of a type T in the test body and pass in an Action<T> to call.  The test body would create an instance of type T, then apply the action to it.  You can even go as far as expressing Act/Assert pairs via a combination of Actions and mocking frameworks.  E.g. you could say “when I call method X on the controller, I expect method Y on the model to be called”, and so forth.

The caveat is that as you do use more and more ‘creative’ types of data-driven testing, it gets less and less readable for other programmers.  Always keep checking what you’re doing and determine whether there is a better way to implement the type of testing you’re doing.  It’s easy to get carried away when applying new techniques, but it’s often the case that a more verbose but familiar pattern is a better choice.

More readable data-driven tests

When the logic of a test method remains constant but the data varies, data-driven testing is a great tool.  It allows you, the test author, to write compact code and to add new test cases rapidly.  Unfortunately, data-driven tests have a disadvantage: The inputs are often less readable.

A simple example

Let’s take an example; testing Rob Conery’s PagedList implementation.  A page is basically a slice of the data returned by a linq query.  If more data exists beyond the ‘slice’ represented by the PagedList<T> instance, its “HasNextPage” property should return true to indicate that it is available.  Now, suppose we want to test whether a particular page has a next available page.  Three things spring to mind that can influence the result: The page size, the current page index and the number of items in the list.

Here’s a quick data-driven test for HasNextPage:

As you can see, the method itself is readable, but the parametrised values fed in via the [TestCase] attribute are not.  It’s really hard to keep everything in your head and remember what each number maps to in the function, especially when the method parameter types are all identical.  If you have a list of 20 [TestCase] attributes, you start wondering what’s for your tea and forget that the second value is the (checks image) page size.  Mince.  Mince for tea.

Hmm, if only we could those TestCases more readable; something like object initializers would be ideal.

A simple trick: Subclass TestCase

My friend Hughel helped me come up with this one and it works quite well.  Simply subclass the TestCaseAttribute class and add your own properties to represent the test parameters.  It gets a little bit hairy when you have to access the Arguments array directly (especially since Attributes can be weird), but in practice, it works fine.   In most of the tests, we’re only interested in parametrising three things, so it’s simple to add them as properties.

The end result

Finally, we apply these attributes to our data-driven test, significantly improving the readability!

I would hasten to add that I don’t recommend using this approach willy-nilly — only when you have a large amount of tests that are parametrised by the same data types, causing the test cases to become hard to follow.  I’ve used the [TestCaseSource] attribute and the [TestCase] attribute a lot in the past and most of the time it’s not a problem.

AutoMapper and Test Data Builders

I’ve recently been tinkering with WCF and, as many people already know, writing data transfer objects is a pain in the balls.  Nobody likes writing repetitive, duplicate and tedious code, so I was delighted when I read about AutoMapper.  It works really nicely;  with convention over configuration, you can bang out the entity => data transfer code in no time, the conversions are less error prone, the tests stay in sync and you’re left to concentrate on more important things.

Anyway, I immediately realised that I’ve used the same pattern in testing — with property initializers & test data builders.  I’ve posted before about Test Data Builders and I’d recommend you read that post first.

For small test data builder classes, it’s really not that big a deal.  For larger classes, using AutoMapper is quite useful.  For example, for testing purposes we’ve got an exception details class that is sent over to an exception logging service.

Every time the app dies, we create an exception data transfer object, fill it out and then send it over the wire.  When unit testing the service, I use a Test Data Builder to create the exception report so that I can vary its properties easily.  Guess what?  The test data builder’s properties map 1:1 with the exception report — hmm!

So, rather than create the same boilerplate code to map the 10+ properties on the exception builder => data transfer object, I just used AutoMapper to handle the mapping for me :)

I’ve had good results with this approach.  The only bit I’m remotely concerned about is creating the mapping in the static constructor.  Any AutoMapper gurus out there who can say whether there’s any reason I shouldn’t do that?

Avoiding the file system

Going from experience and, as illustrated by Misko’s recent presentation, the more dependencies you have on your environment, the less trustworthy and maintainable your tests become.  One of the foremost offenders in this area is touching the file system.

Any time someone says “hey, I’m trying to open a file in a unit test…”, my first reaction is to say “woah”, and not in the “I know Kung Fu” way!  If you introduce a dependency on the file system, bad things are more likely to happen.  You now depend on something that may not be there/accessible/consistent etc.  Ever written a test that tried to access a common file, or read a file that something else may write to?  It’s horrible.

It is for these reasons that many folk will say “it’s not a unit test if it hits the file system”.  In particular, if you have a TestFixtureSetUp/TearDown method that deletes a file, it’s a sure sign that the fixture is going to flake out at some point.

A real example

Recently at work, my colleagues have experienced the joy of a huge refactoring job pertaining to restructuring our projects/solutions to reduce build times and increase productivity.  This work included maintaining something close to ten thousand tests.

As the job progressed, they kept finding that some test fixtures did not live in isolation.  The tests depended on various things they shouldn’t have and, most saliently, file dependencies proved to be a total pain in the balls.  Everything built OK, but when run, the tests failed due to missing files.  Paths and file attributes had to be checked (Copy if newer, etc.), lost files had to be hunted down and so forth.  It’s hassle that people don’t need!  As I’ve stated before, when it comes to testing, maintenance is king.

Anyway, if you have a hard dependency on the file system, consider the alternatives.  This is never a hard rule as it is not suitable for all uses, but it always worth thinking about.

Alternative approaches

Firstly, abstract the file operations to some degree.  You can do numerous things here, from changing the internal loading strategy (via Dependency Injection) to — even better — separating the loading/use of the file so that the consumer of the file’s contents doesn’t even have to care about the loading strategy.

Once you’ve done this, you no longer need to use files in your unit tests, as you can use plain ‘ol strings, streams or even just directly construct instances to represent the contents of the file.

Say we had a simple class called “MyDocument”, and MyDocument could be loaded from a .doc file on disk.  The simplest approach would be to do something like this:

Approach One

Depending on your needs and the demands of the user, this may be OK.  However, to test MyDocument’s methods/properties properly, we need access to the file system to instantiate it.  If our tests are to be robust & fast, we need something better.  Here’s something that’s testable:

Approach Two

From a testability standpoint, this is slightly better, as we can now feed in an IDocumentLoader instance which the MyDocument constructor uses to get the data.

On the flipside, the MyDocument type now needs to know about IDocumentLoader.  To load a document, the user now needs to know about creating an IDocumentLoader and feeding it into the constructor — it’s more complicated.  I often see people do this as the default step for abstracting their file operations — they alter the code to make it testable, but fail to spot the problems it brings if done at the wrong ‘level’.  If you gnash your teeth every time you have to use your own code, it’s a warning sign that something is wrong.

When we think about it though, why should MyDocument need to know about loading strategies?  In many cases, we can parse a file and produce some output using a factory or builder instead.  E.g.:

Approach Three

To clarify how this works: The DocumentLoader would parse the .doc file and construct the object instances required to build up a real document, then pass them into the document’s constructor (or build up the document via some other means, such as iteratively calling methods on a blank document to fill it up as new items are found — whatever makes sense).  This totally decouples the loading and instantiation, meaning we can test each step in isolation.

I.e. the flow goes: Read Input => Parse Input => Create Document from Parsed Input

Life after the File System

Once you’re no longer dependent on the file system, you are free to use one of many strategies for loading/creating your type.  Depending on the abstraction, some options include:

  • Just declare your data inline in the test methods as a const string, or as a const/readonly field of the test fixture.  This works well for small amounts of text.
  • Add your test files as text file resources.  You can then access the file contents as a static string property.  This is handy, as you get a strongly typed resource name and don’t need to mess around with paths + copying files.  This works well for larger sets of data, or data you want to re-use in multiple tests.
  • Use embedded resources & GetManifestResourceStream.  This is slightly messier; it doesn’t require copying files, but it does require that the namespace + filenames used to reference the embedded resources are correct (runtime failures ahoy).  You also need to handle streams when using this method.

If my loading logic deals with strings, I tend to just build an ‘inner’ parser that works with the strings, then wrap it in another class that opens and reads files, then passes the (raw) string to the ‘inner’ parser class.  This allows me to thoroughly test the parsing logic independent of the file system, but also means I can re-use it for test classes or other cases.  I.e. I can exercise more of the production code without any of the file system pain :)

Depending on the thing being loaded, this isn’t always the best solution, but for relatively simple loading I tend to favour this method.