Evolution of an Automated Test
For awhile now I have been advocating that building an automated test is a four phase process. The phases are:
- Record
- Add checks
- Data drive
- Make it smart
In this post I’ll illustrate these steps in automating our One Minute Calculators using Selenium. Because this is tutorial in nature, not all aspects are automated; just enough to illustrate.
Record
The first step in a new script is to record it’s basic actions. Yes, you could do this by hand, but then you have to know what all your object id’s are etc. which is often not easy to get right. Or if your tool doesn’t give you the ability to record, hopefully you are able to use an existing script as a template. The goal of this set is get the basic structure in place as fast as possible. This is the script that Selenium-IDE created (which a bit of formatting cleanup):
<pre lang="html4strict"><?xml version="1.0" encoding="UTF-8"??>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"></meta>
<link href="" rel="selenium.base"></link>
<title>New Test</title>
| New Test | ||
|---|---|---|
| open | /one_minute/earthhour | |
| click | link=more | |
| click | instance_answers_bcartype_8090754_radio_1 | |
| click | instance_answers_bcartype_8090754_radio_2 | |
| click | instance_answers_bcartype_8090754_radio_3 | |
| click | instance_answers_bcartype_8090754_radio_4 | |
| click | instance_answers_bcartype_8090754_radio_5 | |
| click | instance_answers_bcartype_8090754_radio_6 |
All the script does is goto a url, click a button then select each of the 6 radio buttons in order, top to bottom. Unfortunately because of how the radio button id’s are constructed, this script cannot be run more than once without failing to find buttons.
A second part of the Record step is making sure you can run the recorded script more than once, so we have to pull out some XPath wizardry to find the labels minus the dynamic part. I found the pattern I used here but I am sure there are multiple ways to solve the same problem. This is what the script looks like now that it can be run multiple times and have every step execute properly.
<pre lang="html4strict"><?xml version="1.0" encoding="UTF-8"??>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"></meta>
<link href="" rel="selenium.base"></link>
<title>New Test</title>
| New Test | ||
|---|---|---|
| open | /one_minute/earthhour | |
| click | link=more | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_1”)] | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_2”)] | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_3”)] | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_4”)] | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_5”)] | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_6”)] |
Add Checks
A script that runs end-to-end without falling provides value in that all the steps were not blocked. However, thats about that end of its usefulness from a testing perspective. What we really want our tests to do is test which means they have to check for things as they go along. Every script’s thing will be different.
In this case I want to make sure that the number at the bottom (tonnes of carbon) at the column changes when a different radio button is checked. To do this I am getting the existing value, then clicking a radio button and checking that the value is different. I’m still inside the Selenium-IDE for this as it is a convenient environment to constantly tweak in.
Also, note that I’m not checking the actual result or that the number is moving in the correct direction or any number of other possible tests. And yes, this could hang if the calculation the two values is the same, but thats a pretty small risk and would still provide information.
<pre lang="html4strict"><?xml version="1.0" encoding="UTF-8"??>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"></meta>
<link href="" rel="selenium.base"></link>
<title>New Test</title>
| New Test | ||
|---|---|---|
| open | /one_minute/earthhour | |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | link=more | |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_1”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_2”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_3”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_4”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_5”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
| store | //div[@class=”calculation_result”]/span[0] | branch_result |
| click | //input[contains(@id, “instance_answers_bcartype_”) and contains(@id, “_radio_6”)] | |
| waitForTextNotPresent | //div[@class=”calculation_result”]/span[0] | branch_result |
| verifyNotText | //div[@class=”calculation_result”]/span[starts-with(@id, “branch_”)] | branch_result |
We now have a script that not only interacts with a web application but does some behavior verification as well that was recorded, and modified through an IDE and required very little knowledge of scripting or programming aside from some XPath stuff. And you could stop here saying missing accomplished, but there is so much more you can do to this script to increase it’s value to the organization.
Data Drive
We are now beyond what the Selenium-IDE can handle. So the next thing to do is change the format it outputs into one of the Selenium-RC variants. In this case I chose Ruby as we’re a Ruby shop (primarily) and so I’m trying to use it whenever I can. Here is the Ruby version of the script looks like without any further modifications.
Actually, that is not quite accurate. The Selenium-IDE that you can install from the side and Selenium-RC Ruby implementations from subversion are widely different these days (essentially the Ruby client code is now a gem rather than a module you just ‘require’). I had to hack the script a bit to make it work with trunk, but mainly around session creation rather than the actual guts of the test.
<pre lang="ruby">require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
class NewTest
<p>
We now have our script in Ruby and can start leveraging everything a real scripting language gives you; such as file system access.<br></br>
<br></br>
The goal of this step is to extract out hard-coded test values and put those in a separate file. This lets use remove duplicate steps from the script and means the script gets modified less as the test file is what gets changed.<br></br>
<br></br>
In this particular test I am making two things data driven: the calculator and the kilometers travelled. This in effect doubles the testing I can do with this script as it will check 2 different calculators. I also am starting to take defensive action against the Pesticide Paradox by changing the value of the distance travelled.<br></br>
<br></br>
Since this is Ruby I am using a YAML file for this, but I have in the past successfully used CSV, Excel and XML for this as well.<br></br>
</p>
<pre lang="ruby">require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
require 'yaml'
class NewTest
<p>
This is the .yml file that is being loaded.</p>
<pre lang="text">calculators:
- earthday
- earthhour
kilometers:
- 10
- 4385
- 48597
- 43920
- 387
- 97874
<p>
<b>Make it smart</b><br></br>
If you have your scripts all data driven then you are in a pretty good place. If your input files are formatted correctly you might even get close to the holy grail of automation which is having your business analysts or even customers providing you your test data. What I find more powerful though is to go one (or many) step further and teach the script how to get it's own test data. You just let it run forever then. This means that your actual test execution harness has to have some extra smarts to handle stopping tests gracefully and dynamic discovery of new tests, etc. but if you are scripting up things at this phase you are likely starting to deal with those problems anyways.<br></br>
<br></br>
This first bit of smarts that I added to the script was to remove the data driving file and replaced it with Ruby module. The <i>OmcHelper.omc_finder</i> method will return a random url that uses the set of questions identified by 93. I also use a random value between 0 and 150000 as the amount driven annually. The selection of 150000 was pretty arbitrary but will likely be representative of most of the people using the calculator. I could likely figure out what the actual average is based upon historical data and do some clever math to come up with a better number, but in this case good enough really is good enough.<br></br>
</p>
<pre lang="ruby">require "test/unit"
require "rubygems"
gem "selenium-client", "=1.2.10"
require "selenium/client"
require 'omc_helper'
class NewTest
<p></p>
<pre lang="ruby">require 'mysql'
module OmcHelper
class e
puts "Error message: #{e.error}"
ensure
@dbh.close if @dbh
end
end
end
end
<p>
Now our script will automatically test any other calculators that have question set 93, though it might take a few runs since the calculator is randomly selected.<br></br>
<br></br>
But like tattoos or plastic surgery, why stop there?<br></br>
<br></br>
93 is a magic value which is not really scalable. So why not remove it and have the it select <i>any</i> calculator to test? Sure, that's easy<br></br>
</p>
<pre lang="ruby">require 'mysql'
module OmcHelper
class e
puts "Error message: #{e.error}"
ensure
@dbh.close if @dbh
end
end
end
end
<p>
Now you can get a calculator based on a question set id or a completely random one. I'm not making use of this now though because we are stepping firmly into 'can of worms' territory since I would then have to teach the script:<br></br>
</p>
- How many columns are on the screen?
- What are questions on the screen?
- Where are their locations?
- What type are they? Radio buttons? Text boxes? Sliders?
- And how to interact with all of the above
- Valid data in their contexts as some use miles and some use kilometers
And that is just off the top of my head. Which isn’t to say that it won’t happen, but not yet. One important lesson about automation to know is when things become fun rabbit holes instead of value producing. I know that some refactoring is due in the new year around these sorts of things which means my work would largely have to be redone so I should focus my energies elsewhere.
Hopefully people will find this useful. Like it or lump it, automated testing is here to stay and knowing how to create reusable, robust, intelligent scripts is and important tool to have in a tester’s bag of tricks.