In (simplistic) archeological terms, an artifact is something that a previous culture left behind. In testing, we can use a similar definition and label everything we leave behind as artifacts. Test plans, test cases, test results, log files, screen shots are all examples of testing artifacts. But unlike archeology where they discover mundane pieces of everyday life, all our artifacts are things that have value. If something has value, then it gets stored in the version control system; if it doesn’t have value, we shouldn’t be doing it.

Test Strategy

A Test Strategy document is a multi-project, multi-year plan. It does not focus on a particular release or even project. These are in essence a manifesto or business plan for the test group as a whole. ‘We will implement test automation using Selenium for regression testing’ or ‘Retrospectives have value, we’ll do more’ both fall into the Strategy category.

Just like a business plan, a strategy’s dates are on a rolling basis and needs to be evaluated at least once a year to ensure that it reflects reality. The reality that this exists in is the greater company reality, so it is important that management buys into the content of this document more than any other document.

Test Plans

A Test Plan is, well, a plan for some testing activity. This can be a tricky document as it often needs to serve two distinct audiences.

For testers, the metaphor that I use is driving directions. You wouldn’t (likely) head out on a trip you have never done before and not look it up on a map so you know where you are going. These are not super detailed directions, but enough that different people could get to the destination without too much variation in route. And even if you have taken the trek a number of times, having directions is handy if something unexpected happens or some other distraction arrives.

The other audience that sometimes read test plans are product and project management. They are interested in seeing how the test group’s estimates can be justified. It is far easier to say ‘These are our test plans for these features, therefore we need X days’ than ‘Our gut tells us Y days’ and get a favorable result.

Another wrinkle to consider too is the scope a test plan addresses.

Major releases or testing efforts tend to have a ‘project’ scope test plan. ‘The Year 2000 Test Plan’ is a classic example of this. These documents gave a high-level view of all testing activities encompassed in the project’s sub-activities. Some, or most, of those individual sub-activities can have their own test plan which focuses in pointed detail about the details of that activity.

Notice however the word can in the previous sentence. Test plans are a completely optional document. If a feature does not warrant it, then writing a document ‘just because’ is a waste, and does not add value. So don’t do it.

Regardless of whether you are writing a test plan for a project or a feature, there are a number of things that a useful test plan has

  • Introduction – A couple paragraphs which gives an overview of the feature or project the plan is for and the desired outcome of your testing. This is not a restatement of the requirements document. The purpose of this section is to provide context for the rest of the document so it is sometimes useful to think of it as an executive summary for the plan.
  • Feature – This, and the next item are two halves of the same component. Like the introduction, the purpose of the Feature section is to explain the feature from a testing perspective. Things like, desired outcome, interesting implementation details (example: is it REST or WS*) and anything else that sets the context of testing goes here.
  • Feature Activities – With the context set in from above, in this section we describe the actual things that will be done to test this component. Don’t put in things that are obvious like ‘We will use a combination of black box and white box techniques’. Instead, put in the added value parts like ‘We will use this configuration because of X and we will intentionally not use this one because of Y’. These are not test cases, so don’t put that granularity of information in. To continue the driving directions metaphor, the information here is at ‘take the interstate through Pennsylvania’ level.
  • Logistics – Logistics is all about managing the flow of resources. In testing terms, we can think of logistics as having three components

    • Scheduling – This is ‘who is doing what’, but perhaps not how you might expect. The who is in terms of generic people (perhaps also in terms of experience). Do not put in ‘Bob will test this feature for 3 days’. Instead put ‘An intermediate tester will take approximately 3 days to test this feature’. You want to do this for a couple reasons. First, if Bob leaves the team, you don’t have to update your plan to reflect this. Also, even if Bob remains on the team, there is no guarantee that they will be testing that component next time which would result in an edit to the plan.

      Another think you don’t want to put in here is the GANTT chart as exported from MS Project for very similar reasons. As soon as the PM changes something there, your test plan needs the updated export. And project plans by definition have specific people associated with them.

    • Hardware – Is there specific hardware requirements for the execution of this plan? If so, list them here. Perhaps you need a specific architecture of machine, or specific number of machines. If you can recycle existing hardware, then just make mention of it so it is apparent that is the intention.
    • Software – Like hardware, if there is software that is needed to complete the testing activities then it should be included. For example, if one of your testing activities was to automate using a commercial product like QTP, then list it.
  • Risk – It can be argued that all testing activities are risk reduction ones. But what risk are there that affect this plan? Obviously there are the ‘hit by a bus’ scenarios we all tend to love, but they really don’t belong here. ‘This is the first time we have automated our product using this tool / technique’ is a risk that does belong though.

A logical topic of discussion after what goes in a test plan is how it should ‘look’ which blurs in with the discussion of templates. We’ll start there first.

Searching for “test plan template” in Google results in ‘about 1,900,000’ results. Care to guess how many match your context perfectly? Here’s a hint. It’s a nice round number. Round as in 0. And I include things like IEEE829 in this list of templates available to you. To try and fit your needs into a template that was designed to fit someone else’s needs is, at best, folly. That isn’t to say that all those results are useless. Take bits from one, and parts of another and find a format that works for you, and your needs.

Speaking of IEEE829, which is where the IEEE defines what it thinks should be in a test plan. But remember the warning about templates. And the requirement that anything being included actually adds value. That eliminates a lot of what 829 has. Another heuristic to keep in mind when trying to decide whether something should go in is ‘no fluff, just stuff’. Things like ‘table of contents’ or ‘version history’ are all fluff and should not be in a test plan.

One final comment on the format of a test plan: avoid binary formats. Tools like the major office suites can make your test plan look pretty, but they sacrifice the ease of being able to use text tools like ‘grep’ and ‘diff’ on them. I cannot stress how important it is to have these documents checked into your version control system, if for no other reason than you know where the Golden Copy is. But since you have them in the system, you might as well leverage the tools that come with the system, like comparing different versions. You can do that with text, but not binary. Also, text is cross platform, whereas some of the major office suite formats are not. This isn’t much of a headache if you are all one platform, but even in those environments you can run into version compatibility issues.

To wrap things up on test plans, let’s go back to the original directions metaphor.

Sometimes things change once you’ve left the driveway. An accident has closed an exit, or construction is causing a detour. The same things happen when we are testing which cause us to adjust our intended direction of travel. When these changes happen, you need to update your test plans. These are not written-once-in-stone documents, but are ever changing and evolving. An out of date test plan does more harm to a project than not having one.

Waterfall vs. Agile

One of the popular, if misguided, ideas people latch onto with Agile is we don’t need documentation and in some contexts, that might work, but it certainly won’t in others. I would be fairly surprised if an Agile team, working in small iterations and a definition of Done that included tested would have a Test Plan as outlined above. I would expect them to have a Test Strategy though. The big difference in that document would be that it is owned by the whole team, rather than just the test team.