One way to ensure that your product earns a reputation for poor quality with your customers is to fix a problem for them in a hot fix patch and then have it reoccur in the next release. One of the root causes of this is a testing process which not put a high enough value on repeatability of tests.

Let’s make sure that we’re working with the same definition of ‘repeatable’. In this context it means ‘the same result will happen by following certain steps unless a bug was introduced or removed at which point that becomes the new repeatable result’. The repeatable result can, and will, change throughout the lifecycle of a product. And that is okay; so long as it isn’t flipping every single run. Consistency is the desired element here.

Where this really has impact in the test cycle is during regression testing. Regression tests are by definition tests that have known repeatable outcomes. That is not to say that new feature testing cannot (or should not) be repeatable. By the very nature of the beast, new feature testing is ad hoc as you learn more about the feature and how it works. You should however be keeping track of what you are doing. This list then allows you to repeat your testing later when a new code drop happens or a developer asks you what you did to uncover a bug (if you bug description was found lacking). The feature I am working on at the moment is a JDBC proxy/SQL command parsing engine. I have a list of all the queries I have sent to the database as just a big flat file. If I need to reproduce a bug using a outer join with a join hint I just need to refer back to my list. This list also lets me hand off regression testing of this component at any point to someone without SQL knowledge. It could (will) also serve as the raw input into a bit of automation of this feature.

The notion of recording what commands I am running leads into the area of Test Documentation. Explicit Test Case definitions are something that rankles people in the Agile and agile (context-driven) communities. Explicit Test Cases are absolutely necessary for Repeatability. This does not mean they need to be of the level of detail as ‘click on file in the top left of the application window. wait for menu to appear. click on open. wait for browser to appear…’. Your testers are not morons (or shouldn’t be at any rate), so don’t treat them as such. It also doesn’t mean you go to the other side of the spectrum though with test cases of ‘open a file’. As usual, the middle-way is where we want to be, perhaps skewing a bit towards the less defined side of the equation. Ex: Open a text file using the ‘file’ menu and ensure the file has the following attributes: x, y, x. While not 100% repeatable in that you are not providing the exact file to open, it gives enough detail to repeat the purpose of the test and has the added benefit of not running the exact test case twice (a test case will only ever find a bug once, unless it is varied slightly). The exception of course is something like the SQL parser I mentioned. I want the exact commands sent to the parser every time. If someone thinks of a new one, it should be added to the list of things to run. Something like this only scales if automation is involved.

Being able to capture the execution environment is also important to repeatability. If a bug only happens when you install your product on an NT4 machine that has Office installed, do you have a way to getting that environment up quickly every release to test that it is hasn’t returned? It should be no surprise by now that I would suggest that virtualization could solve that issue. As it does for capturing the current state of a machine if Bad Things are happening and you want to be able to showcase them at will. Moving a little finer grain than the OS and the state of CPU/RAM, can you reproduce the state of your database or directory? Your test group should be very familiar with at least the extraction of information from the database for this purpose. Importing might be the responsibility of IT, but extraction should be able to be handled by Test. Since nothing is ever simple, as soon as you are pulling information from data sources there might be privacy ramifications and your test staff will need to be versed in the regulations surrounding the handling of such information in your jurisdiction. Even on a finer level, you should be able to get the internal state (where possible) of your server. If you have a 10 million row table and a query you are executing is taking hours to complete when the requirement is 20 minutes, can you extract the query plan from the database?

Repeatability of results is something development can help with by designing their code for testibility. Since I am not a programmer, I won’t try to explain this but will instead point you here. As a tester though, you should be raising this sort of thing with your architects and developers during the feature design phase. You are involved in all phases of the product, not just when it lands in test, right?

By having code which is designed to produce black and white results, proper procedures to reproduce testing environments, and proper guidance to testers in the form of well developed test cases, this pillar is quite within the realms of achievement.