Mark’s Testblog

Coverage

I recently posted a stack overflow answer detailing why imposing coverage requirements can have a negative impact.  I think it’s worth recycling here.

What is coverage?

Playtest & Iterate.  Lots.

Make an effort to get a weekly playtest up and running.  Weekly is at the very least.  Even if you have 3 attendees and people stay for 15 minutes before they start cutting themselves, it’s significantly better than nothing.

Design and implementation are iterative processes.  Features usually consist of an initial idea, design and the implementation.  If any of those steps don’t pan out 100%, then improvements must be made.  Along the same lines, the feature just might be rubbish and should be removed.  You can only identity and improve features by playing the mod.  I know this sounds ridiculously simple, but you’d be amazed at how badly (or un-spectacularly) some of our ideas translated.  It often took us ages to either hammer the feature into shape or just recognise that it wasn’t going to work.

Similarly, if you encounter something that is unexpectedly fun, see if you can do anything with it.  Many distinctive features can come from unintended or side-effects of implementations.  E.g. Rocket Jumping is a natural result of the splash damage & kick-back of the rocket launcher.  I doubt someone sat down and said “hey let’s blast ourself skywards by shooting ourselves with a big gun”; it probably just happened.  If you don’t playtest a lot, then you won’t find these things or realise that players are using the mechanics in a new and interesting way.

Don’t be overly negative

Negative design is something of a quagmire.  By negative, I mean that someone suggests something which could be pretty cool, but then you immediately start picking holes in it.  You come up with 5 scenarios you think will make it suck, but you had to think about it to make criticisms.  In short, you may be thinking about problems that don’t exist. If it’s not a huge amount of work to implement and/or it’s somebody’s baby, just implement it.  I/We made this mistake a lot.  We talked things over to the point where all that came out was a sensible idea.  Nothing radical.

To further illustrate this point, consider the game Left4Dead.  So much of what makes L4D riotous fun could be picked apart at the design stage by over-thinking and watering down those choices.

• “Right, so let me guess this straight, the special infected just teleport in right next to the survivors with no fixed spawns? That’s ridiculous.  Let’s make some pre-defined spawns.”
• “The boomer is fat, slow, weak and can only vomit on people?  You’re seriously expecting me to waddle around for a while and occasionally throw up?  Boring.  Give him some other function and beef up his toughness.”
• “Once a hunter pounces someone or a smoker grabs a survivor, they simply stay on them until they die? Sounds stupid to only get to choose one target and then be stuck with it.  Let them release at will.”

Deconstructed and totally removed from the context of L4D’s constant waves of tension, these design ideas sound really bad.  In practice they all combine to make the game fantastic.  I’m betting that Turtle Rock iterated on their design constantly and tried a lot of new things, then sat some people down and watched their reaction.  They probably didn’t sit around in a room doom-mongering about why it won’t work!

I would talk more about design, but I’m not really qualified to do it.  AfterShock posted something interesting on design recently (and he is a designer), so I will perhaps reproduce it here (with his permission) at some point soon.

The mod leader must do something tangible

I should qualify this post by saying this is not a dig at those who led at FF towers.  I was part of the leadership and we made plenty of mistakes in the first year or two, but in the year before the first release we learned from the mistakes and made changes.  We didn’t change personnel, we just changed the way we worked.  Without those guys heading it up, we wouldn’t have released anything.

If you have a leader (or even the leads that so many people detest!), then those people have to be amongst the most active on the mod.  Who wants to work hard if their leader(s) aren’t doing the same?  This is also a fundamental problem in a lot of mods — leaders don’t do anything but try to manage (or meddle with) people.  In some situations this is OK, but I personally feel that on an unpaid mod you want to see your leaders getting their sleeves rolled up and doing something more tangible.

The danger is that people will look upon the leader as one of those folk who basically go “hay guyz I had an idea and you are going to do all the work”.  Regardless of leadership qualities, let’s face it — it’s not particularly inspiring stuff.  If a team grows to a size where it is not feasible to work on producing content for the mod as well as leadership duties then it’s understandable.  However, until that time comes, it’s heartening to see the leaders mucking in with you.

I had a few discussions with other modders about this, and quite a few of them claimed to be working under leaders who seemed to spend all of their time hyping their non-existent mod, bragging about their ‘talent’ and meddling.  They didn’t seem to do anything.  The modders openly bitched and disrespected their mod leaders on editing forums/chat channels and claimed that the people in charge were hindrances.  None of these mods have shipped, so I am guessing there was some truth in there.

Technical skills are valuable

If a leader has technical know-how, then they will also have a better idea of what is and is not feasible.  When mapping out the features to be implemented/cut or even just discussing progress with the team, it is helpful to have technical knowledge.  It doesn’t have to be a scary level of expertise — just a coarse understanding of some development areas will be beneficial (E.g. programming, scripting, art & modelling, pipelines to getting assets ingame etc).

This doesn’t just apply to leaders, it applies to all modders.  Microsoft and co. talk of so-called ‘T’ shaped developers being important and this is much the same thing.  Simply put: If you can understand another developer’s vocabulary, it becomes easier to talk about things and relate to their point of view.  For example, I know what someone is talking about if they’re speaking to me about shader constants, batches, unwrapping, texturing, normal mapping, bones, rigging, lua scripting, source control branches etc.  I may not be particularly great at some of them, but at least I understand what they’re talking about.  The only caveat with acquiring this knowledge is realising that a little knowledge is a dangerous thing.  I.e. Make sure you don’t go meddling too much.

As with everything else, there are exceptions to this rule.  Some people are exceptionally good at working with and motivating others regardless of technical knowledge.  Even so, you should try to pick up bits and pieces as you go.  You never know when it can come in handy.

Charts are nice.  Looking at the game is better

Make sure that everyone — from top to bottom — regularly jumps in and plays the mod so that everyone is aware of its state.  I’m sure that it’s great fun checking off ‘done’ features on forums/bug trackers and looking at screenshots, but in my experience this doesn’t really keep you abreast of the actual state of the mod.  You may have a spreadsheet full of green cells indicating that things are ‘done’, but if the mod looks horrible, crashes 30 times a minute and plays like a dog, then you’re kidding yourself.  Always play the mod.

Kick problem people in double quick time

Sometimes you ought to kick someone out. It’s tough to do, but you have to (or you should). I kicked a few people out of FF and God knows I should have done more of it. I wish I had been more ruthless when I had the capacity to kick.

My list of the usual suspects:

People who have constant personality clashes with others

If someone cannot get on with their team, they are actively hurting the mod. It doesn’t matter if it’s work or personal issues, just remove the problem and the others will thank you for the improved atmosphere.  If there are 10 major blow ups during the course of a couple of months and 7 of those have one person at the centre, it’s usually fairly easy to spot the common denominator.

People who offer a constant stream of negativity

Nobody wants to work on a project that has been poisoned by over the top negativity.  Constructive criticism is good to have, but some people just bitch, moan and whine about everything.  If a mod team contains one or more people who constantly snipe, troll and bash others, the impact can be serious.  This doesn’t have to be over the top criticism, little snide remarks here and there are enough to get folks’ backs up.  Have a word with the person and be honest about the problem.  If they do not change their ways, remove them.

Drama queens

If every little problem turns into a huge deal and a 40 page flame-fest which takes hours of people’s time to clean up, only for it to happen again and again then just cut your losses and remove them.  These sorts of shenanigans can offer some light relief at times (“Did you see what x did?  That guy is so highly strung it’s ridiculous!”), but clearing up the fallout can be time consuming.

Modders who are unable to work with others

By ‘work’, I’m talking about technical or artistic collaboration.  Whether the inability to collaborate is down to obtuseness, stubbornness or arrogance, it has the same effect.  If near enough any criticism of piece of design, art, sound, code implementation etc. leads to a slanging match with that person, then they are failures. It’s OK for people to get heated now and then — it’s natural that people feel strongly about their work, but it’s for the mod. If they don’t want to collaborate with others, then that person ought to go make their own personal project where they are free to do so.

Likewise, if someone cannot give constructive criticism, that person is a failure. Nobody likes working hard on something only for some jackass to trash their efforts.  “It’s shit” / “It’s not what I wanted” (no elaboration) / “I don’t like it” / “the last guy was much better” etc. are all totally pointless criticisms. If you have to criticise something, give the person some pointers. You can always point out some of the parts you liked, too.  I ran into quite a few people who just didn’t understand how to work with other people.  Try to mediate and change people’s attitudes, but if it doesn’t work, eject them.  You’re better off without them.

The bottom line

Clashes are almost inevitable, but you ought to know when certain individuals are frequently instigating and causing problems. Remember that modding is meant to be enjoyable. When it becomes a chore to do (in your spare time, no less!) I’m sure you can think of numerous more appealing things you’d like to do instead. For instance: Playing games or joining another mod with a better atmosphere.

You may read this and think “well, that’s obvious”.  However, sometimes you can be too nice and these problems, left unchecked, can snowball out of control.  Nip it in the bud.  Don’t just talk about it, do it.

Don’t expect too much, but DO remove slackers

Modding is unpaid, so you cannot and should not expect to be able to slave-drive people. Modding is also carried out during people’s free time — don’t expect the earth. Give people a bit of slack for getting their work done. If someone drops off for a few weeks or a month due to other commitments and they’re up front about it, that’s fine.  Be straight with people and expect the same in return.

However, if it becomes clear that someone is unreliable and/or lazy, just remove them.  Do not recruit someone to take their place while also keeping the original person on board.  It just bloats the mod team and complicates things when that person tries to return to being active, particularly if they are working on the same area of the mod.

In the three+ years of FF development, I can remember a single instance where an inactive person made a telling contribution after returning to the mod.  The rest of the time the person just did the exact same thing — worked a little for a few weeks and then flaked out.  You will obviously have to make exceptions here or there, but in general you ought to have an idea of what you’re willing to tolerate and stick to it. Once someone fails to hold up their end of the bargain more than a couple of times, just remove them.  It’s easier for all concerned.

It can be difficult if you have grown to know and like someone, but having a mod team with lots of passengers just doesn’t work.  You might think “well, maybe they will dip in and do something useful when they have time”, but believe me, most of them will not.  Acknowledge their contribution and move on.

Be honest

The major thing that pissed me off when working on FF was people who committed to reasonable goals but then stalled, lied and ultimately disappeared without having the good grace to simply admit that they haven’t got anything done. I much preferred it if someone came to me and said “I’m sorry but I just haven’t got anything done in the last few weeks”.  If someone is honest you can have a quick chat with them and try to sort out their situation (maybe they are unhappy with something)

Similarly, I was honestly never once angry if someone said to me “sorry, but I don’t have the time/energy/motivation/whatever to continue on the mod”.  Someone who is honest and tells you that they cannot continue is much better than someone who wastes months of your time by lying about non-existent progress.  If a mod isn’t for you, that’s fine.  Just tell someone and then leave!   It’s not hard.  The mod loses someone who was not producing much and can replace them.  The person can move on to something that is more their cup of tea.  Everyone is happier.

Keep your team as small as possible

There are numerous reasons for this. Firstly, keeping the team small keeps expectations low. Things remain friendly and it tends to foster an attitude of “we’re in this together”. If you’re working on something with a handful of friends or dedicated workers (who will likely become good friends!) then you feel inclined to pull for the team that little bit more. When FF was just starting, I felt guilty if I let the side down because there were maybe 5-10 people tops who were depending on me, nearly all of whom were working very hard themselves. I’d say we made much more progress (at least in terms of productivity per person) in the early days when our team was tight knit. There wasn’t much in the way of politics at that time, and things were much simpler to keep track of.

The more people you have working on a mod, the more people you have to manage. As more and more people join, you tend to have to add further levels of indirection. Most leaders cannot hope to keep tabs on 40 people or understand the ins and outs of every discipline. Furthermore, if someone joins and finds that they’re 1/40th of an unpaid mod team, they may be less inclined to work hard. This is especially the case if that person can look around and see that 20 people out of those 40 don’t seem to be doing anything. Unless a modder is a very self-driven and conscientious worker, they will feel demotivated by being a cog in a huge, messy machine. Creating something is meant to be gratifying, not galling. This is particularly important if a person’s work has dependencies. If Jim works 15 hours a week on the mod and finishes all of his work, but requires Iain and Bill to put in a combined 10 hours to complete the feature they’re all working on, Jim is going to be severely fucking demotivated if Iain and Bill consistently fail to hold up their end of the bargain.  If it’s a small team then this is less likely to happen. If it’s a big team, then Iain and Bill might just not give a shit. After all, Henry and Mike aren’t doing anything, either!

Avoid bad applies

Now, I know this is the INTERNET and it can be extremely difficult to gauge someone’s personality when your communication is handled through emails, IRC or voice chats but… try to do your best.  If you get a whiff of something being ‘not quite right’  (be it their personality or their dedication), then tread carefully.  For example, if someone claims to have been on multiple mod teams but has very little work to show for it, ask around about them.  Go to the mod teams’ websites and get in touch with some of their members.  Ask what the person did for them and whether they left on good terms etc.  This doesn’t have the invasive, just a cursory check. If someone is a bullshit-merchant, they are not going to be of use to you.

Similarly, be very aware of people who have some cool screenshots of very specific parts of their work but seemingly nothing else to show you.  Alarm bells should start ringing if all someone can show you is one room in a level or half of a gun that has been textured.  You should move to defcon1 when you’re linked to multiple unfinished images, no matter how good they look in isolation.  A lot of people like to do the first fun part of development, but are total flakes when it comes to finishing (or even thinking about finishing) because, in all walks of development, finishing is hard.  Doing the horrible little bits like balancing work, general tidying up, bug-fixing and optimisation is a pain.  People who cannot finish personal work under their own steam are nearly always useless to a mod.  If they cannot be bothered to finish stuff of their own choosing, are they going to muck in with everyone else during the parts that suck?  I doubt it.

Don’t recruit just to recruit

This one follows on from the previous point. If someone is keen on joining and they’re good but not outstanding, but you don’t need to recruit because everything is going fine…  don’t recruit! It can be hard to pass up a good modder, but you should be very selective if you have something good going on. If you’re just starting out then fine, grab anyone you can! Once you’re moving along nicely, don’t disrupt things unless you have to.  Also, before you bring on a new person, you should always know how to get them moving immediately. By this I mean they should feel included and know what they have to do from the first moment.  This doesn’t mean you pressure someone to produce immediately, but it means you should give them the means to get started.

This could simply mean that they join the mod and immediately have IRC/forum/vent/source code/bug tracker access and some suggestions on what they ought to be doing (“Hey, it’d be cool if the bug in feature y could be fixed” or “The balance of gun x isn’t quite right, you could try and sort that out when you have time”). If you leave a new modder dangling for days or weeks, it looks bad and makes you look like a bunch of clowns!  You don’t want that, do you?  No.

Background

Although these days I spent most of my time writing code and being concerned with the automation of testing, it was once not so.  Prior to joining Realtime Worlds as a Software Test Engineer (a programming discipline), I spent a lot of my time creating levels for the Half-Life (goldsrc) and Half-Life 2 (source) game engines.  In fact, at one point I fully intended to try and forge a career as a level designer rather than a programmer.

Anyway, the reason I mention I’m bringing this up is that I spent three+ years working on a mod called Fortress Forever (hereafter referred to as FF).  During the development of FF, we learned as we went along.  Almost none of us had ever made a mod before.  As the years rolled by, we made a lot of mistakes.  A lot.  Countless errors, big and small.  I was one of the modders who took it upon themselves to try and organise and direct the way we were working, so I had a good view of what was going on.  Oh, and I also had the chance to balls-up plenty of things myself.

What I’m going to cover

During this series of posts, I’m going to catalogue the problems we encountered, the mistakes we made and some suggestions on how to either solve or mitigate the problem.  Do not for one moment consider everything I say to be taken as ‘truth’.  Each point is related to the way *I* perceived a particular situation.  Also, depending on your team members’ experience levels, personalities and your relationship with them, my suggestions may be not so useful –or even counter-productive– to you!

Nevertheless, I still think many of these tips are good as a rule of thumb.

Intended Audience

As you might expect, this series of posts is intended for modders.  It’s mainly aimed at the people who have to try and organise things, but it should be useful for anyone who is thinking of working on a mod, starting up their own mod or changing the way their existing mod team works.  Furthermore, a lot of this mod stuff has parallels in the ‘real’ world — after all, a lot of the tips are to do with handling personality clashes, differences of opinion and that sort of thing.  You find these problems in all walks of life — this is part of the reason why I think modding is a hugely valuable thing to do for personal development, particularly if you’re trying to break into the games industry.  You learn things that can only be learned when collaboratively building something.  Compromise and teamwork do not exist when working on something in isolation.

It’s not uncommon to encounter developers that are wholly resistant to unit/functional/integration testing.  Some of them will simply dismiss the idea due to thinking it’s new-fangled rubbish (“I never unit test, and my code is fine”).  Other developers may recognise the positive impact tests can have, but simply deplore spending any time writing them.  Some folk will go to great lengths to either bypass the test writing stage, or write tests in a fashion that gives zero or even negative returns!

So, why should we write tests?  At a very high level, the answer is simple: If the tests are well-written, automated testing helps prove that your software works.  I’m the first person to admit that automated testing is not a silver bullet, but particularly with state-based testing, it can be extremely effective at verifying that your code does what you think it does.  What you think your code does and what it actually does are two entirely separate things.

To try to bring some of the test-sceptics on board, I will ask a simple question: What do you do when writing a fairly complicated algorithm that can be verified by looking at the values/output/state?

If you’re anything like me, you draw/write the desired result and then try to come up with an algorithm that satisfies all of the required conditions.  For example, for a terrain grid algorithm, I would draw a grid with the vertices, edges, indices and so on.  I’d then iteratively write an algorithm that generates terrain, testing its output values using pencil & paper.  When the algorithm finally satisfied all of the conditions I’d set, it’d be fairly robust.  To be sure it works for numerous inputs, I’d then draw a few more examples and double check that the algorithm can produce the result for multiple cases (e.g. 3×3 tile, 1×5 tile, 2×2 tile).   Can you see the parallels?  Each set of desired results/values is a test case.

I hope you can see what I’m getting at: Pencil & paper testing is a form of test-driven development.  The main difference is that the pencil and paper tests go back in the drawer never to be seen again, but the automated tests remain.  In addition to ensuring that the code does not regress, they provide a form of documentation and are a great aid when refactoring  & optimising.  Furthermore, at this stage, it is generally easier to write new code tests, meaning you can cover more edge cases without having to sharpen the ol’ pencil again.

If you’ve ever followed the pencil and paper process, you have been testing without knowing it.  Other than my own endeavours, I have personally witnessed software engineers transferring their pencil & paper tests into unit tests and then creating scores of test variants.  The result?  Numerous bugs were found and the time spent testing proved to be hugely valuable.