I recently was invited to the second Toronto Workshop on Software Testing which is focused this time around on ‘Test Models’. Although I had to decline the invite due to scheduling conflicts I spent most of the weekend thinking about the test models I employ. I’ve come to the conclusion that I use two: The Onion Model and The Beauty is Skin Deep Model.

As usual, let’s get a definition out of the way. By ‘model’ I mean a series of practices or an approach to testing. Hopefully I am not munging the definition too badly, but like it or lump it, this is the definition this post revolves around.

I think it is best of I give some context to the models I am about to describe by describing an application and explaining how they would affect the testing approach. The application is JDBC proxy which does some pretty fancy SQL manipulation and other magic (this is a real product which has yet to be officially announced so I have to be vague) but is designed to operate transparently to the end user. It is distributed as a separate component to a larger application suite but is to seamlessly integrate with the suite. All configuration is done through a screen that resides in a Java Applet.

The Onion Model
This model is the one I utilize most frequently as I think it provides the highest level of overall Quality and applies to not just the Testing phase of development but overlays both coding and testing — remember, Quality is more than just Testing. It gets it’s name from how an onion looks if you cut it in half; lots of concentric rings each supported by an earlier ring. If there is another, more widely used label for this model, ping me and let me know. In deciding what to test, you make a list of the shared components and work your way out. The theory being that if module A is used by modules B and C, and module B is used by D and E, by testing module A you are also indirectly testing the functionality of B, E and F. Once you have got the kinks worked out of module A you move to module B, then C, etc.. There is also a bit of regression risk mitigation in this model as the bugs you find earlier in the process have a greater chance to be fixed earlier as well which means their effects on dependent modules will be already be in place when you get to them. If you had tested modules R, S and T only to find a bug in module A, you would have to test all the intermediary modules between A and R to make sure their was not a cascaded failure added.

The overlay with development comes with model because developer written unit tests make a fantastic center ring giving Test a decent level of confidence in the code when it arrives at their door.

Applying this model to our sample application we would order our testing as such

  1. ‘SELECT’ parsing – not only is SELECT the most common SQL command, but it is used by other commands and in the case of the sample it was architected smart enough to have all SELECTs handed by the same chunk of code.
  2. all other parsing activities, weighted based upon popularity
  3. ‘magic’ – as defined above
  4. configuration via the GUI
  5. integration within the suite

The Beauty is Skin Deep Model
This model is one I occasionally employ when thinking about testing applications which are either brand new (and therefore need to demo fantastically), are complex enough to be ensured to be run in a pre-production environment by customers or whose sales lead time is large/pipeline is small enough to all but guarantee a code rev between now and deployment. This one gets its name from underlying principle that how the feature/product looks is the thing that matters most. It would of course be nice if it did what it was supposed to as well, but what is truly important is the look and operation from the (potential) customer’s perspective.

Again applying this model to our sample application we would spend 2/3 of our testing effort with the configuration GUI and suite integration and the rest of the time making sure the parsing and magic portions were not completely broken. Partially broken in this case is okay.

Clearly both these models cannot be used in a vacuum. If Product Management and/or Sales needs to be able to demo the product/feature to customers in the near-term but deployment won’t be for awhile, the Beauty is Skin Deep model might be worth exploring, but make sure it is explicitly declared as the approach to testing in the product/feature test plan and there is functionality risk. You could then deploy the Onion model further down the road to capture the functionality problems.