So I went to the monthly gathering of TASSQ this evening. I figured that I should probably go to a meeting or two before I’m slated to talk at the March one and now that I’m independent I should make myself known in various circles. What I discovered appeared to the a bastion of ‘old school’ qa (not testing).

Now, I know this post is going to sound like a bash, and it’s not really meant to be one. It is more rooted in incredulousness. I know it is hard to get up in front of people and expose your ideas and ways of doing things to others. But you have to remember, that when you do, you open yourself to this sort of thing.

Oh yes, and it is also important to realize (as most who read this blog likely do already) that I’m philosophically aligned with the Contextual, Agile and Craftsman schools of thought.

What was the name of the talk?

Factory School Testing Revisited / Redacted / Re-energized. aka Kickin’ it ‘Old School’

Gah! Right there I knew I was in trouble.

Anyhow, here are the notes / impressions / ideas I had during it.

  • One of the stated goals was to have a topic that resonated with folks in the room. I’ve only been to a couple TASSQ meetings before, but if this is the topic of interest then it kinda says that the group is not concerned with new ideas.
  • I haven’t worked in a regulated environment in a long time so I could be wrong, but don’t the regulatory bodies care about types of testing you are going to do before hand and the results after? I seemed to get the impression that one of the large arguments for creating a large body of test cases ahead of time was to show the ‘depth’ (or something) of your ‘to be done’ activities. Well, that doesn’t prove anything. What matters is what has been done. I could say that as part of the testing I’m going to build a rocket and go to the moon! Results are what matter.
  • Also provided on the same slide was the notion of resource variability. ‘We have high turnover of QA people so we need stacks of test cases to bring the replacements up to speed’ is my paraphrase. Well, if you have high turnover, then I would say that is a smell that you are doing something wrong. And what about turnover in test management? I’m guessing it has the exact opposite pattern — the entrenched stay entrenched and those that challenge or deviate from plan are removed.
  • He had a nice metaphor about Christmas and testing. Testing is like Christmas morning; when you wake up there are gifts lying under the tree. But they are wrapped and you can’t see what is under the paper. And sometimes you like what you get, but sometimes it doesn’t fit and you have to return it. This can sometimes hurt the feeling of the giver, but often they understand. And there is no guarantee the replacement you get is going not have a page missing or some other thing wrong with it that you don’t discover until the next day. So the cycle repeats itself.
  • I’m not the best speaker, but there is likely a business opportunity waiting to be exploited in teaching testers how to present to small-ish audiences of strangers. How to speak clearly, to the audience, without fidgeting and with a nice slide deck. I wonder how my Lessig-esque deck is going to go over in March. Care to bet the comments that come back are that I need more bullets?
  • Not once were the people involved referred to as ‘testers’. It was always ‘QA’. What you name something is critically important.
  • What I think was most scary was that I can remember a time when everything that was said would have rang true to me. I’m not saying that Rapid Software Testing threw me from my horse as I was likely travelling down that road anyways, but apparently the big Canadian banks need some heavy introductions to Context techniques and ideas.
  • I sat next to Fiona Charles who offered up to a discussion of metric a rule-of-thumb I hadn’t heard before. 3:3:1 – For every three test cases you write, you will find one bug. For every three bugs you find, one will be severe enough to warrant fixing. If you application is producing more bugs that the model, then you need to look at the quality of the application. If however you are finding less bugs than that, you need to look at the quality of the test cases.
  • If metrics and counting test cases executed is so important, isn’t it rather ironic to encourage testers to go ‘off-script’? You number are wrong(er!) then by your own admission.
  • I did like that it was stressed during the metric part that it is not the number exactly that matters on the graph. It is the pattern that it depicts that counts. Oh, and that it might take 20 minutes to generate the numbers daily; of which 15 minutes of that is writing the summary and story (context) around them.
  • My big issue with the Factory is around its reliance on test cases. So I asked a couple questions at the end about it (in as non-obnoxiously as I could). And through that I had to explain the Mine Field problem. Sigh. I know I associate with people who value constant learning as a core attribute, but c’mon! Learn the core theories of your craft! Someone in the audience added that in real-time/safety-critical systems they cover the system 100% thus removing all the mines. Except, that is impossible. You’ve covered every scenario and path you could conceive of at the time. But unless you test every scenario, every interaction, by every person, ever to use your product, ever, then you don’t have 100% coverage. Even if you have 100% code coverage, which I admit is likely what he meant, you can’t account for oracles like emotions at the time of use.