I’ve written a number of automation runners in my career so far. I’ve been writing another one recently as well. This time it is not tied to a specific employer or product.

Each incarnation has had some variation of this for test organization and discovery. It is very hierarchical and structured which means it maps well to tools like Quality Center and can generate nice graphs for completeness, etc..

The big flaw is when you have bits of different trees you want to run.

The solution I think might be to introduce ‘tags’ to a test. Run all test tagged as sanity could be one possible scenario. But how would you organize such things? By tag? Well, that doesn’t work if they have multiple tags, as any such system would require. Chances are a hierarchy of some sort would evolve taking us back to the first approach almost.

So I think the solution might be to offer the ability to run a section of a heirarchy and (or) a set of tagged tests.

The question is which would people get most value out of quickly.