Hoffman has his own method of gauging controllers’ stress levels: Check the angles their spines make with the seats of their chairs. At 100-plus degrees—leaning back—the work is easy; straight up, things are getting interesting; once they cross the 90-degree threshold and begin to perch forward, the sky is roiling chaos. Most of the controllers at the simulation never crossed the 90-degree mark.

That is from an article in Wired called Air Repair is which about redesigning the flight paths around New York city and its cascading effect on the rest of North America’s air space. It’s an interesting article, but the bit I quoted was by far the most important to testers.

In doing simulations they had all sorts of checks to see whether the test was successful or not, but the one that involved observation of humans was likely right up there in terms of importance. Human beings don’t really work well under stressful conditions; its the whole fight-or-flight chemicals that start to do weird things to brain chemistry. It is far too easy to just rely on what your unit / system / integration / other test results are telling you and ignore what is in front of you. It’s not quite the same as the Narcotic Effect of the green bar, but more of a Spotlight Effect.

When someone is on stage and all the lights are out except for a lone spotlight on an individual person, it is really hard to make out the other details and events going on stage at the same time. But someone with a trained eye, and knows to be looking can still pick up important information.

Would you or I have noticed the angle of the chair? I wouldn’t have. I would have been concentrating on locations of planes, etc. during the actual test then likely asked the controllers afterwards how they felt during the test. But that introduces all sorts of biases and internal justifications and such which would likely taint the findings.

The Spotlight Effect is something to be wary of when determining your oracles. Especially when some of your success criteria are linked to internal states of people.