I can’t remember where, but I recently stumbled on a paper by James Bach and Patrick J. Schroeder called Pairwise Testing: A Best Practice That Isn’t. It has a really nice breakdown of the problems of testing combinatorial problems and the “best practices” of how to test them. The quotes are of course necessary because this is by James who is one of the leaders of the No Best Practices fight. I say that only slightly facetiously as I have not really bought into that notion completely. At least not yet. Anyways, back to the paper. James and Patrick nicely rip apart pairwise testing as a best practice and quite convincingly put it in the ‘one tool in a toolbox of combinatorial testing techniques’ category.

What I liked best about the paper however is neither the discussion on pairwise testing nor the best practices debunking. Instead it was how they ended the paper, which I reproduce here.

We recommend that you refuse to follow any technique that you don’t yet understand, except as an experiment to further your education. You cannot be a responsible tester and at the same timedo a technique just because some perceived authority says you should, unless that authority is personally taking responsibility for the quality of your work. Instead, relentlessly question the justification for using each and any test technique, and try to imagine how that technique can fail. This is what we have done with pairwise testing.

Don’t follow techniques, own them.

Achieving excellence in testing is therefore a long-term learning process. It’s learning for each of us, personally, and it’s also learning on the scale of the entire craft of testing. We the authors consider pairwise testing to be a valuable tool, but the important part of our story is how we came to be wiser in our understanding of the pairs technique, and came to see its popularity in a new light.

We believe testers have a professional responsibility to subject every technique they use to this sort of critical thinking. Only in that way can our craft take its place among other respectable engineering disciplines.

I especially like the first sentance. Maybe this is why I enjoy getting sucked down technical rabbit holes (much to the annoyance of my boss sometimes…).