Every person involved in testing professionally (and by this I mean someone who thinks about testing, and not just running WinRunner scripts) eventually evolves their own philosophy of what makes a testing process Good (or Mature). I’ve more of less distilled mine down to what I call The 3 Pillars of Testing. This number may change of course, as well as what they consist of, but at this point there are 3.

  1. Auditability and Accountability
  2. Repeatablility
  3. Traceability

These deal only with the Testing phase of development. Things like proper requirements management, definition of ‘done’, etc. are part my larger view on Quality and are not covered.

This post is about the first one; Accountability and Auditability.

The origins of this one can be traced directly back to my learning of the testing trade at one of the large Canadian banks. When you start there, there fear of Audit is drilled into your head. And it makes sense. If testing is not accurate, there is thousands, if not millions of dollars could be at risk. One one project I was involved in, we knew in advance that an audit was going to occur (mandatory audits in a project is a Good Thing). All testing work ceased for a couple days prior and we made darn sure our ducks were in a row. Where a duck was missing, the reason was identified and a plan put into place to ensure the duck would materialize at some further point in the project.

I’m not saying that each software development team employ an auditor, but I think each team should operate as if they had one. And how should they go about doing this? This for starters.

  • Maintain accurate and up-to-date test documentation
  • Be able to produce test results on a specific (certificate authentication with an embedded OCSP location that conflicts with one specified another manner) and aggregate level (all certificate authentication tests)
  • Testing decision log
  • Tester allocation – both past and future

One of the hardest things for me to wrap my head around when I went to a start-up was the lack of audit mentality and how it allows for some pretty slip-shod testing practices.

The slip-shod practices result because the removal of a formal audit process is the removal of a certain degree of accountability from the testing equation. One answer that doesn’t fly well with auditor is “I don’t know”, but in an environment without the possibility of an audit, that answer is much more entrenched/prevelant.

  • Was Feature X tested? I don’t know.
  • Who tested Feature Y? I don’t know.
  • Feature Z had some pretty big changes after you report having initial testing it. Did you re-test is? I don’t know.

So what do I mean by accountability?

  • From the individual Tester perspective, I expect to hold/be-held to a level of accountability where if I say a feature was tested and a large bug that should have been found (using the current test procedures should have been found) that I get asked what went wrong. Having that sort of a discussion too many times could have negative effects on my employment. In a process with little or no accountability, it would be easy for me to deflect questions along those lines with “I don’t know”.
  • From a Test Lead perspective, they should be held to the same standard as the individual Testers, but for a larger breadth of subject matter. If they say the release is of high enough Quality to ship, and the field easily shows otherwise, they should get the hard questions.

How far we take the notion of capturing accountability is something I am still grappling. At the bank, all project plans and release decision documents were signed — in pen. At HP, releases are okayed over a conference call (which is not recorded) and then typed into a document. This system could be fairly easily exploited. I don’t think that everything needs a physical signature in today’s email pervasive environment, but they certainly hold a place in key documents.

Testers, including Test Leads, should have nothing to fear from being accountable for their work. Well, the good ones at any rate. The poor ones will be made apparent and either brought up to the desired clue level or find other occupation. Unless of course there is politics involved, then all bets are off. 🙂