BT

Lessons on Building Continuous Delivery for Infrastructure

| by João Miranda Follow 2 Followers on May 31, 2015. Estimated reading time: 2 minutes |

Lindsay Holmwood recently recounted his learning experiences on building Continuous Delivery (CD) for infrastructure services. Holmwood offers advice to enable fast feedback loops, notably of Service Level Agreements (SLA) compliance, and to support small, discrete changes. Holmwood is the creator of Flapjack, a monitoring notification routing and event processing system.

Holmwood defines infrastructure services as "a configuration of software and data that is consumed by other software". This includes, for instance, DNS, databases or monitoring services.

Holmwood asserts that to get quality, not just fast, feedback there are five main issues to think about. Start with the CAP (Consistency, Availability, Partition Tolerance) theorem. Decide if the services you're building should go for consistency or availability. Examples of the former may include databases, of the latter, load-balanced servers. Once you've made that choice, it's time to agree with the infrastructure users the SLAs to commit to. For instance, "95 percentile response time for monitoring metric queries in a one hour window is < 1 second". Third, use your CD pipeline to check your SLA's, whenever possible. You must also have clear, well-defined, interfaces between infrastructure services. Those clear interfaces help to verify interactions between parts, both during testing and normal operation. Finally, Holmwood argues for pushing data to one place to enable immutable infrastructure.

On fast feedback, Holmwood and his team found that placing the continuous integration (CI) system near the IaaS provider (be it on-premises or in the cloud) is crucial to reduce latency times. Holmwood cites a project where the build time reduced from twenty to five minutes, just by moving the CI close to the IaaS API.

If your team mostly does its job through the command-line, context switching to and from the CI web interface can be an user experience blocker. To ensure that everyone complies with the rule of pushing every change through the CD pipeline, Holmwood suggests tools such as jenkins-cli. The build process can then be followed from the command-line:

git push && jenkins start $job && jenkins tail $job

You can even graph some data in the terminal with the help of gnuplot.

Fast feedback requires small, discrete changes. This means the tests must execute fast. Holmwood advises to make each test run in under ten seconds. Given we're in an infrastructure context, making tests run fast also means compromises. So Holmwood automates tests that look for unmistakable problems, such as checking for running services and executing simple queries. Monitoring checks complement the CD pipeline tests to verify end-to-end behaviour.

It should be noted that Continuous Delivery for infrastructure is getting more vendor support over time. Hashicorp's Atlas and the forthcoming Chef Delivery are examples of these developments.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT