Keith Stobie is a Test Architect at Microsoft who has spent most of his time there testing shrik-wrap applications but recently moved over to a web services group. In his keynote he highlighted both the similarities and differences between testing the two types of software.

As per usual, here are the notes I took.

  • WS should never fail. In order to do make this a reality you need to remove the number of possible code paths your application have. This make the happy path, the only path
  • Ad supported sites tend to have higher quality because the losses due to downtime can be easily calculated (number of average ad clicks per minute * number of minutes down)
  • Why WS testing is different
    • Must deploy into a live environment
    • Due to deployment into a live environment, you need a complete rollback strategy
    • Need to work with the Operations team to derive a capacity model for the WS. This capacity model will largely determine the architecture of your testing rigs
    • Need to test for all failure modes (including things like partial cluster loss due to things like ice storms taking out data centers
    • Test how you are going to monitor your service once it is production, not just how the task the WS is supposed to do
  • You can test your WS in production
    • Catch things you didn’t / could’t test in pre-production environments
    • Actual users could show you usage patterns you did not anticipate
    • You can inject test data into live systems, but you need to mark it somehow and have it not included in reports. MS uses the ‘zz’ country for this.
  • Service Monitoring
    • Be careful about the number of alerts that your application generates. The operations team will start to ignore all messages if your WS is too verbose missing the important one(s). In other words, don’t cry wolf
    • All alerts should be specific and actionable
    • Turning on / off alerts and thresholds at when alerts are triggered should be configurable and changes of those configurations should be automatic (no restart of the WS)
  • Map our your WS’s components. Any place where you have information coming into a system, or being passed from one component to another is a place were you can monitor and/or inject test data
  • One big advantage of WS testing is that once the initial release is rolled out, you know exactly what the user is doing (well, assuming you have decent logging). So,
    • Take the production data, apply a one-way hash to any personal identifiable information and feed it back into your application and see if things still work. This essentially turns the production system into your oracle
    • If your dataset is too large, scale it; 10% of the data into a system 10% of production
    • Cull out most of the ‘normal’ data and load your data set with the exceptional cases
  • You can do staged deployments of WS into production for A/B testing. MS does A/B testing based upon the IP/ISP to avoid confusion between friends / neighbours where one sees A and the other sees B. There seems to be lots of interest in the topic (which I take to mean that there is not a lot of information widely spread on this topic — at least in the CAST audience)
  • Some of what you should do to test WS? (according to Keith of course)
    • Automate deployments. Anytime your inject a human into the process, you inject a failure point. We used automated deployments quite successfully at Points
    • Emulate (stub) out dependent services to allow you to test without them or beyond their current thresholds
    • Production dog food – let certain classes of users use the new version of the app before you deploy it generally
    • Roll out new version in stages. This of course requires some preplanning because you have to make sure that your application can handle message to/from older versions of the product
  • Run your test lab’s instance just like the production one is run. This means that if a certain alert is raised, it is responded to exactly like how it would be by operations

While not a direct comment on Keith’s talk, I was sitting beside someone at lunch who commented that he couldn’t believe that at no point was how to test SOAP messages mentioned. I mentioned that SOAP was just a messaging formation and isn’t (necessarily) applicable to REST based WS. That stumped him. But it doesn’t matter, the talk was structured in a way that it was implementation neutral. Looking for examples of SOAP also shows that he was missing the point of the talk: to test a WS, you use all the same techniques (installation, monitoring, performance, usability, error handling, etc) as you would with any other type of application, you just do them in a different manner.