I spoke at the University of Toronto this week for an hour to about 40 of Greg Wilson’s third year students. It could have used more (any) rehearsal and as usual I had a major case of the nerves but overall I think it was a success and I got to the points I wanted. This is the mini-essay version of that talk (primarily so it can be sent to the students)

Being Picky about Terminology

Even though the introduction used the term QA a half dozen times, what I do and what I talked about is actually testing. Why? Well, let’s look at the term QA a bit.

  • Quality – The definition I currently use is ‘Value to some person that matters’. For more build-up on this concept see here and especially here.
  • Assurance – Most people who do QA cannot actually achieve or influence this part of the term. Can you actually assure the quality of the product? Do you make staffing decisions? How about schedule ones? Do you have final say on the release metrics and which bugs get fixed? Odds are you don’t and won’t. So you aren’t really assuring much.

Okay then. What is testing then? Well, the AST has defined it as

  • An empirical technical investigation conducted to provide stakeholders with quality related information
  • A cognitively complex activity that requires critical thinking, effective communication and rapid self-directed learning

(If memory servers, Cem Kaner heavily influenced that definition)

The key words in the first bullet are provide quality related information. That is primary motivator for all testing activities. Notice how neither point says anything remotely like ‘improve the quality’. The Quality is actually your (the programmer) responsibility. As Michael Bolton has been saying recently, testers are there to defend the Quality of the code.

Oracles

Whether you are a tester or a programmer, you need to know what your oracle is when you are testing. This is really an unfortunate term given the size of Larry Ellison’s company, but this is not Oracle as in the database, but Oracle as in the Oracle of Delphi (who could talk to the Gods but was really stoned on sulphur fumes). An oracle in testing terms is the principle or mechanism by which you recognize a problem (BBST). This could be a person, another application, your professor, the TA, or even internal.

Ask Questions

A large part of testing can be summarized as ‘asking questions of the application’. Ask a lot of different questions. If you ask the same questions all the time, you won’t learn anything new.

But don’t limit yourself to asking questions only of the product itself. Ask about the things circling the product. Such as the requirements. The requirements for the assignments you have received certainly have enough information to do something with (both create and test). But really, the requirements kinda suck. But that is okay. Requirements in the real world do generally suck. And they likely always will be. The world is a dynamic place which is undergoing constant change. As such, the requirements will change constantly too.

Your job though is to question every line of them though. Ask the obvious question, you might not get the obvious answer. Ask the crazy questions, that is often enlightening. Ask why we are bothering with this feature, that will help clarify your mission.

Mission

Knowing your mission is important to testing. What information do the people you are testing for asking you to find? Doing security testing might be fun, but if your employer is wanting performance information then you are off mission. It doesn’t mean you can’t do security checks, but make sure you have given them the performance data they were asking for first.

Complete Coverage

Complete coverage is technically possible if you are doing TDD in its purest form. However, that number is often abused. Yes, you have touched 100% of the lines of code, but you have not touched 100% of the code with all possible values and in all possible contexts. This is part of the ‘Impossibility of Complete Testing’ which is also covered in the BBST course linked to above. Anyone who tells you otherwise is lying or just confused.

This doesn’t mean there is not value in coverage tools, or even aiming for 100% coverage. They are a useful class of tools to tell you where you have under tested and also lets you determine when your coverage has change (for the better or worser) when then lets you question whether that can be lived with or not. (There is that questioning concept again…)

Code Review

Ask anyone who has written a book about the value of an editor. Just as writers fill in the blanks between what they meant to write and what they actually did write, programmers will do this with their code. While having every line of your code reviewed by a peer is a bit overkill, it is certainly valuable for anything of any sort of complexity. Learn how to review code.

Random Bits

  • Don’t fall into the trap of thinking ‘No one will ever do that…’. No one will ever do that who is acting in a sane manner who does not think they are clever or are trying to find holes and/or problems with the system and are are only ever using it in the manner it was envisioned. Well, the real world is messy. Someone is almost guaranteed to do that at some point. Hopefully you won’t look too dumb when they do.
  • You are nothing like the the person who will be using the code you produce. Well, most of the majority of the time at any rate. Think of the person who calls you to ‘fix their computer’. If they could use you application, then you are doing pretty good.
  • Bugs tend to cluster. When you find one in a piece of code, spend some time to poke around and see what else is lurking.

That was more-or-less the hour except some discussion are possible tests for a term assignment. I likely should have written this before the class shouldn’t I. 🙂