As an external consultant or expert, I’m often dragged into the stakeholder budget debate about needing to justify the investment in automation. Most recent of these conversations was this week and my response is publishable without editing as side from salutations and other niceties. (Good job me!). So here it is.

*Automation and Testing in general is hard (impossible?) to pin a dollar amount to in terms of ROI. It is often better to talk about ROI in terms of time and allowance of better testing.

What I would try to do is identify a number of flows through the application that always need to be tested. These are often the ‘smoke’ or ‘sanity’ tests and have them running out of Hudson (after the unit testing job). You no longer need to do those by hand on each build which means you can can start doing manual exploratory testing faster. Which is very important. Automation succeeds long term when you think about it in these terms rather than as the finder of bugs.

A ‘green’ builds means you can start testing, not that testing is complete.

This of course doesn’t answer the money problem. Often people want to look at how much a bug escape costs the company and then you look at how many the automation finds. But even that is overly simplistic as a lot of the bugs the automation process finds is during the creation of the scripts. In fact, most automation fails to ever find a bug once it is running clean.

By going through the process though, the automation person now knows way more about the system than they did before. Which in turn lets them do better manual testing. See the pattern here?

Michael Bolton splits things between ‘checking’ and ‘testing’. Testing requires a human to determine if something passed or failed whereas checking is something a machine can decide the outcome. So the automation with Se is ‘checking’ not ‘testing’ under this definition. What is interesting is that when you are creating the script you are ‘testing’ but as soon as you call it done it is ‘checking’. This, to me, is an important distinction in terminology.

So I would talk to your boss and explain that Automation ROI is not in terms of measurable dollar amounts but in intangible benefits to the quality of the system.*

It seems kinda weasel-y and consultant-y, but I am a tester by origin. And it is a little easier to swallow than throwing out your scripts.

Oh, and I seem to recall hearing the green build comment from Michael as well, but he attributes it to J.B. Rainsberger as When all your unit tests pass, that doesn’t mean your product is done. It means that you’re ready to give it to a real tester to kick the snot out of it.. Again, not something you would put in an email that could float up a chain of command, but bang on.