Yet another post regarding the scribbles I took during people’s presentations at TWST3. This time on Morven Gentleman’s talk about ‘How much testing is enough.’

  • A fundamental property of risk is that everything can be reduced to a comparable value. How do you compare the two risks of ‘being mentioned in parliament’ and a ‘specific dollar loss’? There is no common comparable value
  • Here’s a heuristic; father knows best
  • The reality of the world is that business risk trumps technical risk
  • Putting risks into bins doesn’t really help (due to the comparable value problem)
  • When doing any sort of risk assessment, you need to have access to all risk holders
  • Risk is dependent on whatever else is going on in the world at the time and by extension is constantly changing
  • The cost of at test based upon it’s risk can be substantially altered depending on where in the sequence of all tests it is conducted
  • Testing based on risk with test data that does not accurately reflect field data is a waste of a test
  • After the initial rush of bugs when a new version of software is released, organizations are lulled into a false sense of confidence because the number of bugs reported decreases (software reliability growth). That does not mean that new bugs are not being found, but that customers have stopped reporting them or are learning to work around those limitations.
  • The original purpose of requirements documents was to be able to pay for partially completed work
  • What is in the budget for a loss based upon a missed risk?
  • Something is better than nothing, but making sufficiently wrong decisions can be worse. Just do the best you can
  • There are two types of positive tests
    1. Tests that give you the same result every time
    2. Tests that give you a different result each time
  • Reliability is a perception of robustness
  • Robustness is how the software behaves when faces with bad things