Continuous deployment is often described as an Agile or a Lean technique where all the code written for the application is immediately deployed into production. There are numerous perceived benefits of this technique including reduced cycle time and reduced time to market for bug fixes and new features. However, is it as easy as it sounds?
Jim Bird suggested that most changes that people talk about when they mention continuous deployment are trivial changes like minor tweaks, cosmetic fiddling or small bug fixes. Anything larger than that requires a relatively detailed and careful approach.
According to Jim,
Schema changes can’t be made continuously. Bigger functional changes can’t and shouldn’t be made continuously, even with dark launching. Etsy for example (one of the companies used as a poster child for Continuous Deployment), doesn’t continuously deploy bigger public-facing features. They take their time and design them and prototype them and test them and review them and plan them out with operations and customer support and product management like any sane organization.
Jo Liss mentioned that real challenge with continuous deployment is the non-zero cost of reverting a change. According to Jo, for continuous integration, the limit to how often you integrate is mostly technical but this is not the case for continuous deployment where the cost of reverting a change could be huge.
But once you have deployed to a production site with users and valuable data, reverting is expensive, because you may have to:
- Migrate the database back to the previous schema and conventions.
- Think about what happens to users who are using your site right now and having the application changed under their feet (potentially causing links to break, and Ajax requests to fail).
- If it’s a bug (rather than just a decision you’d like to reverse), you might even have to email users who were affected by the problem, or deal with support requests.
Likewise, Eric Ries suggested that one of the biggest challenge with continuous deployment is getting ready to release all the time.
On the one hand, this is the ultimate in customer responsiveness. On the other hand, it is scary as hell. With staged releases, time provides a (somewhat illusory) safety net. There is also comfort in sharing test responsibility with someone else (the QA team).
So how can a team ensure that they can realize the benefits of continuous deployment?
- Dont push features, build in response to signal from customer
- Code in small batches
- Prefer functional tests over unit tests whenever possible
- Implement alerts and monitoring both at system and application level
- Tolerate unexpected errors exactly once and fix them immediately
Jo suggested that one should push fewer commits to the server. According to her, the reasonable deployment delay is in the order of five hours to two days.
So if you can sit on your hands for a while instead of succumbing to the temptation to be hyper-aggressive and deploy immediately, then you might be able avoid most of those nasty 5% of changes you really wish you hadn’t pushed to the production server, and at the same time only lose unsubstantial amounts of early user feedback.
This is not to say that continuous deployment is not possible. Many organizations like Etsy, Heyo, IMVU & Atlassian all do continuous deployment and supposedly, do it effectively as well.
As Jim summed it up,
Yes there’s a lot to learn from Continuous Deployment, about streamlining and simplifying release and deployment, and reducing risk by breaking work down into smaller and smaller pieces and tying all of this together with ops monitoring and metrics. But it’s not the “Holy Grail of Devops”, or at least it shouldn’t be.