The kids at Sauce Labs have put up a video from the last SF Selenium meet-up where the speaker was none other but Mr. Continuous Deployment himself, Timothy Fitz. You might remember the kerfuffle he raised in the testing world earlier in the year. If not, then you can see Michael Bolton’s thread on it here, here, here, here, here and here. I had hoped that Continuous Deployment (CD) would not get widespread adoption, but he is still flogging it so I decided to momentarily stop avoiding it and watch the video of the presentation.

Before I list my thoughts and notes I took while watching the video I think I’ll outline my mindset to frame them. CD to production is reckless and borderline irresponsible. ‘Automated tests do not a test strategy make’ to paraphrase Yoda. That said, there are some cool things he is doing that I can see be quite worthwhile exploring; especially if you are rolling out to dozens, or hundreds of machines. I suspect that some of them are almost required at that sort of scale. CD to a test environment for humans to test is an absolutely fantastic idea though. We should do more of that.

With that out of the way, the notes:

  • Don’t fix bugs on features no one uses. I would go further and question why something is in the product at all that is not used.
  • At it’s core, CD means ship more frequently
  • Regarding automated testing: now we’re playing for keeps
  • That is a huge problem for me though. That implies that we know what the automated tests should look like to replicate human behavior the first time. We, as humans, are very bad at looking into crystal balls regarding the future.
  • CD implies at a very deep level that tests will be executed in parallel. That can only be a good thing in my books.
  • As is the idea to run tests in random order even when there is no commit-triggered build in order to find race conditions.
  • Debug all failures. No excuses. Your automated tests are the only thing between your IDE and the customer.
  • The most powerful idea in Timothy’s talk was about creating an Immune System for your application. I completely want to steal this idea. Essentially you instrument the application out the wazoo and it will automatically back out changes that cause a rush of white blood cells. Things like ‘number of registrations per minute’ or ‘conversion rate in 30 minutes’ are monitored; real business drivers not low-level server stuff. Well, likely in addition to, not as complete replacement.
  • The crux of CD is that you are putting code in front of real users as soon as humanly possible so they can derive the most value from it. This is standard Agile theory, pumped up on steroids to clown-like proportions as you could be delivering code 30+ times a day easily.
  • But customers are also the primary human means of problem discovery as well. Which is the principle problem with CD from a ‘tester’ perspective. I don’t want my customers to experience annoyance, frustration or confusion from my application and take their money to a competitor and tell their friends why they couldn’t do X as advertised on the system. CD seems to have accepted that ‘customers will find bugs’ and are ok with it. I accept that they will find bugs (perfect software is a myth, etc.), but I’m not ok with letting it happen with just a shrug.
  • Lock the repo when something fails since you are now in an unstable state.
  • Integrate the operations team into the development team. Again, nothing new here. We want to develop a culture of ‘the product team’ not silos.
  • We don’t have QA people writing tests, we have engineers writing tests. Ummm, ya. I’m having deja-vu here. This sounds a lot like early Agile/XP where testers were not needed because ‘the bar is green dammit!’. What is state-of-the-art now? Oh ya, having ‘agile testers’. Guess that didn’t work so well, did it.
  • Wow. What a shock. CD is also related to Lean thinking. That metaphor has taken a beating recently in the testing world.
  • You have to make sure your application/platform can handle every edge case. Which of course means you can predict them all.
  • You can’t just start doing CD; your application needs to be built around it.
  • At 58:30 he says that another option is to deploy not to production, but to a production-like environment. That’s the winning idea
  • Have ‘dials’ for all features. The volume of the dial determines how much of the cluster gets the new feature.
  • Gradual deployment is also a component of CD. That is the great backstop that lets your pull things out of production if they don’t work.
  • In response to a question about whether to use it in a ‘mission critical’ (which I’m going to take as ‘someone could die’) he admits that CD is likely not right in that situation. This reinforces the notion I’ve had from seeing other stuff on CD that it is fine for ‘toy’ applications (where things like even large social media things like Twitter and Facebook could possibly be lumped as ‘toy’), but if you are in the ‘enterprise’ or higher then you need to really, really think twice about doing ‘pure’ CD
  • Once a failure is rolled back from production you need to write unit and acceptance tests for it so you don’t find that problem in that way again. Fine. But what about all the other ways?
  • Engineering problems quickly become people problems. Ain’t that the truth.