Part of my consulting practice is to act as ‘Auditor’ and provide a third-party review of company’s automation. Before even trying to run them, I first just browse through the code looking for Smells. It is hard to itemize things that come from experience and ‘gut’ feel, but this is a partial list of what I realized I am looking for.

  • Excel – If automation code is reading test information or recording results in Microsoft Excel this is a smell that your automation is Windows specific. That is not always the case, but often is. It also is a smell that the people doing the automation code are not ‘trained’ programmers (which is OK, I’m not ‘trained’ either) as Excel is often the tool of Analysts and Classical Testers. There are three main problems with Excel. The first, and it is the killer, is that it is a binary format so any analysis between versions is impossible. If you need it to load in Excel then use a CSV as Excel will associate with it by default. No, you don’t need formatting in Test Data. This is the big reason I believe things like Robot Framework and Cucumber will succeed. The second problem is that it costs money. Yes, almost every desktop in your office has Office on it, but does your test farm as well? The last problem with Excel is that it is often used in place of something better. Why are you getting test data from Excel when it could come from the database directly? Why are you creating an extra layer of logging when the language of your automation has it already or the CI server has it?
  • Exception Handling – The goal of automation is to find problems and problems manifest themselves in xUnit style automation as assertion failures. Let the code fail. Do not catch the exceptions. The frameworks know how to behave on those types of failures. Often people catch the exceptions so they can record the result in their custom logging, but that begs the question of why you have written another layer of logging — often using Excel. Yes, there are times when you will have exception handling in your script, but that should be for things like “I tried to read a record from the database and it wasn’t there implying this environment is not configured correctly” not “catch AssertionException”.
  • Paths – The presences of a non-relative path on disk (windows or unix) is a smell that the automation is tied to a single platform at best, or a single specifically configured machine at worst. Automation should be self-contained with all necessary bits checked into version control. If I need to do more than ‘svn co’ and some minor editing of config files (my username to the database for instance) then we have a problem.
  • DRY – Don’t Repeat Yourself is becoming my favourite smell. Copy and pasting is not a technique for increasing LoC amounts. Look really hard at why you are pasting that method into that class and not abstracting it to a separate class. This, like most things is heuristic though. You don’t want to introduce extra coupling of code by accident which can lead to more brittle code not more robust code.
  • Configurability – How to I quickly and easily switch my automation from my local machine to something like Sauce Labs OnDemand? If the answer is anything other than ‘just change which config file is loaded’, then the automation framework needs some work. Ruby on Rails completely I think completely nailed this problem with their ‘environments’ and I have often implemented a ‘Selenium’ or ‘CI’ environment.
  • Versioning – This smell is one that is most seen in traditional testing teams that have somehow assumed the mantle of automation creators and maintainers. Many universities still don’t teach their undergrads version control, but developers soon learn that in The Real World. Not so much in Test which often is the realm of Shared Drives (Q for QA commonly) and document management through email. If it has value it is checked in, if it has no value is it not created should be the mantra of anyone doing anything with automation. That is the first half of the smell. The other half is how the version control system is used. Are the commits atomic? Are the comments associated with them relevant and provide the necessary information? Are updates updates and not delete plus add?
  • One Product – A final smell today is noticed through automation but is actually an organizational one and that is where the automation code is in relation to the production product code. If they are separate then the smell exists. I would guess that most organizations are not in the business of making a product and making automation. They are in the product business. The automation code needs to live with the product. This smell harkens back to the days when the Us vs. Them mentality of Dev vs. Test was seen as productive. There is one team, working towards on goal. Treat their artifacts the same.

That’s it for now. The next time I’m auditing I’ll try and keep track of my thoughts to see what other smells I can find.