So you want to build a framework…

My first foray into the ‘framework’ business was likely 1999 at one of the big Canadian banks. We were automating binders, literally 3″ ring binders, of manual test cases into WinRunner. There were 5 or 6 application silos with some shared things for login, etc. It was the ‘shared things’ that made it a framework. From there I have written one at pretty much every employer, the lessons learned having resulted in Py.Saunter and SaunterPHP. That was 14 years ago though. I’ve had a tonne of time to make mistakes [and hopefully learn from them] that a lot of people getting sucked into the automation whirlpool don’t have the advantage of having. They don’t know what they don’t know as it were.

This talk is about the various things a framework designer needs to be thinking about constantly from the perspective of someone who has lost sight of them at some point. The goal was not ‘here is how you write a framework’ since I could just point to my github… but to cause the attendees to go to their office later and start questioning the decisions they have made implicitly to see if those are the ones they are comfortable having as explicit ones.

One of the things you learn as a consultant is that things displace other things. This includes automation frameworks. It could be that it is replacing ‘nothing’, or another framework, or manual exploratory testing. But it is replacing something. Your job as a framework author is to be better than what you are replacing. And to keep improving it so that it doesn’t get displaced by something else.

Lightsabers is a favourite meme I use over and over. See Lightsabers, Time Machines, & Other Automation Heuristics

One of the things you learn writing frameworks is that the vast majority of stuff you write, your users won’t see if you do your job properly. Frameworks are all about hiding the abstractions and details behind the scenes. This can be a management problem though if you don’t keep them appraised of what you are working on. Trust me, it is possible to make a tonne of framework improvements and then catch trouble for not being productive on the automation…

Also a common rant of mine, and has a section in the Lightsabers article.

At the heart of your framework will be a runner. This is Py.Test, JUnit, PHPUnit, etc. Its job is to collect, execute and report on your scripts. Ideally the execution part will be done through some variation of the xUnit setup/run/teardown pattern. A well written framework tightly integrates into/around the runner. Look at the idioms, patterns and integration points of the runner. Once you choose one, it becomes really hard to replace it. Remember that the Audience is telling you which language you are using, but most languages have multiple runners you can choose from.

(Also, a hilariously meta photo.)

This is one of the biggest things you, the framework author, gets to control. Configs go here, and they look like this. Logs will go here, and they are in this format. Etc. The most successful frameworks all limit the decisions their users can make.

Don’t put your configuration details in your scripts. Don’t put them in your page objects. This is a pretty huge code smell. Put them somewhere that is completely separate and can have a different life within version control. The format of this is also dictated by the Audience. And is why, after 3.5 years of ‘selling’ Saunter to people I am switching the format to YAML since it has taken me that long to really understand who my ‘typical’ customer is.

Recall that one of the roles of the Runner is to discover the scripts that will be executed. Is it going to be by annotation/decoration/tag (my current favourite approach), method calls in a class, xml listing methods, etc. This seems like a small thing, but it actually has a pretty big impact since it also affects structure of scripts.

Logging is about diagnostics to the users. Presenting information to the user is crazy difficult. I pretty much avoid this problem and show stack traces from the the underlying runner. Heck, automation is programming … stack traces is how you diagnose crashes. Right? The key thing here is that I explicitly made that decision.

This is the dashboard-y stuff. The easiest thing here is to use the non-standard-yet-standard Ant JUnit XML format. I’m pretty sure every framework author has implemented this at some point. Please don’t come up with a new format unless you are also writing the consumer[s] of the reports.

Commercial frameworks live and die based on where / what they integrate with. OSS ones still somewhat do, but the expectation isn’t there so front-and-center. If your framework reports in the ant format you get most CI integrations ‘for free.’ Know what your framework is displacing. Unless it is part of a larger process displacement (waterfall to agile), it needs to integrate with at minimum what the existing thing does. And even within the context of a larger change it might need to.

Does the execution have to be on iron behind the firewall? Or can be in The Cloud? At this point I think all frameworks need to have a cloud execution story. What’s yours? Does it integrate with a specific cloud, all clouds? By configuration or by documentation?

Welcome to the house of cards. How important is backwards compatibility? How do you coax your users to upgrade? Is the upgrade process manual or automagic? You need to decide where you land on the spectrum with this. And then be consistent with it. When I release new versions of stuff, this is what worries the most. Especially for enterprise-y clients.

Lighting up browsers is slow. Its just a fact. Parallelizing the run is part of the solution to make making this tolerable. How does your framework handle this? Mine for instance doesn’t do parallelization, instead pushing it onto the CI servers it integrates with. (See how all these things tie together?)

Yes, your framework should run through a proxy. If yours doesn’t, this is your homework.

Sometimes you need to provide the user with the ability to hit things really, really hard in ways that they shouldn’t employ all the time. The framework should let ‘advanced’ users do this sort of thing. For instance, I think runtime parameterization of test methods is a really bad idea. But that doesn’t mean I have disabled the hooks in Py.Test that allows you to do that (though I could have..). This is analogous to the JS Executor in WebDriver.

One of the things that used to burn us with WinRunner was how it interacted with version control. Or more correctly how it didn’t. Everything in you framework should have a version control story around it. For instance, with my stuff, the actual config files that get used don’t get checked in, though the templates for them do. (Another thing I stole from Rails.)

Packaging is also one of the most horrific parts of most languages. But it is important that you work with the default packaging system of the language. Regardless of how horrific it is. [*cough* PEAR] Remember, ‘clone from github’ is not a distribution strategy.

It frustrates me that people are building frameworks for just ‘web’ or just ‘mobile’. You want to win? Be able to use the same framework for both. Figure it out.

Perhaps more important than what your framework does is what it doesn’t do. Well, doesn’t do on purpose. If it doesn’t do it because it is missing a feature, then you are at risk of having it displaced by something that has it. If you have a story / explanation about why you don’t support something, then that’s so much better. Of course, you could still get displaced if someone really, really wants that thing. But don’t compromise on your vision for the framework

At this point there isn’t much technical reason to not open source your framework. Of course, there are lots of business reasons not to, like ‘OMG! We don’t have a business model’ which is fine. But if the framework is a supporting application for your real business, open it up. Don’t underestimate the effect github has had on both distribution and instance community.

And finally, don’t be afraid to screw up. Often. And when you do, apologize and fix it. And then make a new mistake while trying to push things forward.