I tend to sit in #selenium on the freenet irc network during the day and almost all the questions that get asked in there seem to involve the IDE component of Selenium. Either they are trying to make it just through a hoop that it can’t jump through, or it could go though, but the hole is really small and its on fire. Inevitably, both of those problems would can be solved by using Selenium Remote Control (RC).

Source of pain is a useful heuristic in sniffing out problems, and the amount of pain people (appear to) go through with the IDE would seem to imply a problem.

So what to do about the Selenium-IDE?

This morning I quipped on Twitter that the Selenium gang should reduce the functionality of the IDE. I think that the IDE is an absolutely necessary component of the Selenium suite, but only as an initial skeleton producer that would be consumed by RC. The first thing that could get removed to this end is the notion of Test Suites. The only time you should be running a test inside Se-IDE is to debug the flow of things initially; then you export into language of choice. Next up would Custom JS Extensions. Use modules / libraries / whatever-your-RC-language-calls-them instead. Finally, I would take a hard look at all the Commands available to you when inside Se-IDE. addScript and its ilk are all suspect. I’m not suggesting they be removed from Se entirely, just the IDE.

Patrick Lightbody (one of the core Se folk) countered instead that the solution might be to increase the power of the IDE. For example, to convert it from a (pure) Firefox extension to a Firebug plug-in which would make its default language Javascript which would add powerful scripting features like conditions and loops. That is interesting, but only solves 60% of the problem. The other 40% I don’t think can be solved by having script execution live entirely in the browser.

The real value proposition of RC is not that you get conditionals and loops out of the box (as compared to custom JS extensions), but that you can access things that a browser cannot. The database for instance. It’s not enough to just validate a form for the string ‘Update Successful’ when testing. You have to go to the database to verify the update. There are any number of failure modes otherwise. And you can’t do that from the browser. You also cannot write a file out on disk to archive your results. Or data-drive your tests. Or wire them up to other domains (due to same origin). Or run them (from IDE) in non-Firefox browsers. Or. Or. Or.

The big argument for Se-IDE is that you don’t have to be technical (pronounced ‘know how to code’) in order to get value from Selenium. I think this might be a bit of a stretch. Yes, you can get a browser to do stuff, but is it high-value testing? I’d argue not.

Repeat after me: Automated testing is a programming project. You will write code.

And that scares a lot of people in test roles and is vendors rarely tell that in their slick pitches to customers.

I suppose I should amend the previous statement a bit.

Automated testing is a programming project. Someone will write code.

The path to success with automation systems like Selenium appears to be to write a DSL (Domain Specific Language) that abstracts the technical parts away from the non-technical users. Fitnesse is likely a classic example of this I think, but if you watch the various GTAC videos, this is a common theme. But to get to the point where you have a nice non-technical DSL takes a lot of technical coding (and thought). And is likely harder than just writing against the Se API directly in Python or Ruby.

So what to do about the Selenium-IDE? There are three options: nothing, add functionality, or remove functionality. I think doing nothing leaves us in the weird limbo we are in currently, adding functionality addresses some problems but not all of them, and removing functionality pushes people up the value chain earlier but also forces a higher degree of programming skill. There are trade-offs in each, but I think the value Se get from being on RC (and later Grid) means greater long-term success in using Selenium even if it means a bit of short-term pain as the user base adapts / evolves.