Matt Heusser this last week has kicked off a number of discussions surrounding GUI automation tools. A phrase that came up during the course of it was ‘highly repeatable.’ It also happens to be one that causes some pretty strong emotions in the testers I swim with.

Hard Coded Test Values

Usually the term is met with scorn and disdain and implies hard-coded sets of test data that will remain unchanged forever and ever and might catch a bug the first time it is run with the probability decreasing with each run. See this university course’s test case examples for a set of highly reproducable test cases that does indeed get a pretty strong reaction from me. But maybe bug discovery and product exploration is not the intent of the test. Maybe you want to ensure that the bug that really, REALLY annoyed a large Dutch brokerage never, EVER comes back — or if it does you want to know about it right away.

  • Repeatable: Yes
  • Predicable: Yes
  • Useful: Not really. You can usually use a later strategy instead with a bit of thought and get better bang-for-buck
  • Summary: Pretty close to evil

Data Driven

Moving away from hard-coded data sets takes us into the hazy realm of data driven tests. The actual scripts tools follow when data driven are just husks that pull the critical test data from an external source. These external sources are often CSV files which are easy to maintain and have business people add things to but they still offer a constrained set of test data; if it is not in the file, it is not in the test. This lets you do more with less which is usually a pretty state to be in. It also means that your fix-the-broken-scripts cycles are shorter.

  • Repeatable: Yes
  • Predicable: Yes
  • Useful: More so than using Hard Coded Test Values, but still pretty limited
  • Summary: Lawful Evil – “May or may not help someone in need”

Data Driven (on drugs)

To use a random baseball comparison, what would happen if we injected our data driven tests in the butt with TGH (Test Growth Hormone)? We would still be data driving them, but instead of creating our test data, we would let the scripts themselves create the data. This is the strategy I use most often.

  • On screens which just display information from a data source, teach the script to request a random record
  • On screens which have to insert or manipulate the data, I try to teach the script the rules around the screen I am automating. I then add an element of randomness into the mix causing the script to think of its own data and do its own verification. (I’m not lazy, I’m efficient…)

For this strategy to be truly effective you turn on the scripts and let them run for a pretty decent period time. There is lots of math (that I don’t claim to know) which will determine / prove that a length of time is correct to find most of the issues through randomness.

Model Based Testing

The latest evolution in automation is model based testing. I don’t claim to know much about MBT, but as I understand it currently, this takes the notion of teaching the scripts where to find the data to the point where the whole thing becomes a giant state machine. Not only does it know about the data, but it also knows about the rules and relationships regarding the data. Repeatable? Again, yes, but in a non-predictable manner. Evil? Not even close. We’re approaching Nirvana actually. Useful for the long term? Absolutely (assuming you can handle changes to the model elegantly)

  • Repeatable: Given enough time, a test will be repeated.
  • Predicable: No
  • Useful: Absolutely
  • Summary: Is that the entry to Nirvana I see on the horizon?

So do ‘repeatable’ tests deserve (all) the bad press? Like most things the answer is ‘it depends on the context.’

(Before I hit ‘Publish’ it occurs to me that I did not properly capture the difference between DDoD and MBT? After seeing Harry Robinson at CAST last year I know there is a difference but have no idea how to illustrate it currently and since this is a blog and not a magazine I can post 2/3 baked ideas.)