BT

Ready for InfoQ 3.0? Try the new design and let us know what you think!

The Software Defined Delivery Manifesto: Collaborative, Model-Based, Event-Driven Automation

| by Daniel Bryant Follow 823 Followers on Nov 22, 2018. Estimated reading time: 6 minutes |

At GOTO Copenhagen, Rod Johnson announced "The Software Defined Delivery Manifesto", written by multiple experts within the field of software delivery. The authors of the manifesto argue that the delivery of software "is not a detail, it is our job", and accordingly, "now is the time to engineer our delivery". Key takeaways include that software defined delivery should be core (and strategic to every organisation), well-engineered, collaborative, accelerated (through automation and reuse), and observable.

InfoQ sat down with Johnson, CEO of Atomist and creator of the Spring Framework, and discussed the motivations and creation of this public declaration. He introduced the manifesto as being written by several industry luminaries, including contributions from many others across the software delivery community. The primary goal is to encourage software developers and engineers to think about the state, benefits, and challenges of the current software delivery tooling and practices. The goal is not to "reinvent the software delivery wheel", but rather to "stand on the shoulder of giants within the automation, infrastructure as code, and continuous delivery space", in order to provide best-practices, consistency, and structure.

The manifesto begins with an acknowledgement of the impact and pervasive nature of software within the modern world, and also outlines that value is only derived from code when this is actually running within production:

We recognize that delivering useful software shapes our world. We recognize that code is the best way to specify precise action. We recognize that code is only useful when we deliver it.

The manifesto authors argue that the delivery process of software creation is not simply a detail, and accordingly, "now is the time to engineer our delivery". Acknowledging the impact that situational context has on the way that each organisation delivers software, the authors continue by stating that they "recognize that while continuous delivery is essential to meeting business needs, automating all repeated tasks is important". As delivery infrastructure is now programmable, this means that it should be defined and programmed using well-accepted practices from software development.

We accelerate our automation the same way we accelerate application development: with modern architecture and programming languages plus frameworks, libraries, and services for generic capabilities.

The remainder of the manifesto is divided into five primary sections that characterise software defined delivery:

  • Core: Delivery of software is a fundamental and strategic capability for every software team and organization. Delivery code should be treated as "first class" and strategic: "Decide policy at the team and organization level; implement it with precision, without toil, in code."
  • Engineered: Delivery code must be engineered to be testable and robust; not written using scripts that don't scale: "70s-era scripting languages are insufficient". Code should also be backed by a model of the software delivery domain. It should follow current good-practices within software development, such as being event-driven and easily extensible.
  • Collaborative: Delivery of software requires collaboration among people and software, and between people and software. "Each person can express their expertise in code for everyone's benefit", and they should combine "best-of-breed" tools to provide collaborative automation.
  • Accelerated: Repeated tasks should be automated to speed up work and avoid errors, and common functionality should be shared between developers, teams and organisations.
  • Observable: There should be a common means to observe and troubleshoot what happens in the delivery process, as there would be within any a production system.

Many organisations have provided inspiration for the manifesto, and Johnson name-checked Google for their work within the Site Reliability Engineering (SRE) movement. Their approach of applying software engineering thinking and practices to the operational space, combined with the use of automation and the reduction of toil has prompted many engineering teams to attempt to recreate this within their organisation. As laudable as this is, the resulting tooling is often bespoke and takes more of a tactical approach to addressing requirements. The Software Defined Delivery Manifesto attempts to drive a more strategic approach to the entire process of software delivery.

Johnson was keen to keep separate the ideas presented within the manifesto from the implementation details, but offered his personal thoughts in relation to the work he has recently undertaken. Although Infrastructure as Code (IaC) is now a well established approach, there tends to be an over-reliance on YAML, which does not offer the power of a full programming language. As an example of tooling within in space that does follow good practice, Pulumi, with its tag line of delivering "Infrastructure as Code on any cloud with real programming languages and a consistent programming model" was mentioned.

The majority of delivery code "does not use a model", Johnson mused, which means that it is hard to reason about the process and also to re-use functionality. Citing Tasktop connect as an example of an implementation of a delivery process implementation done well, he discussed that this offering has a clear model of software delivery, and this allows the easy interchange of information between different implementations of components like issue trackers, delivery pipelines, and build artifact metadata stores.

Storing software delivery data as rich events allows an accurate history to be recorded, and may also enable additional data to be mined and added to an enhanced model at a later date. Here Johnson commented on the work of the GitHub team, and discussed the rich metadata associated within actions undertaken, such as commits, pull requests, and releases. The GitHub Developer GraphQL API v4 was provided of an example of a well-implemented API within this space, and here he liked the functionality offered by the GraphQL implementation -- in particular, the ability for a developer to query and explore the model using tools like GraphiQL, a graphical interactive in-browser GraphQL IDE.

As the GitHub model of their domain is event-based, this has easily and effectively supported the recently announced GitHub Actions, which allow developers to react to key lifecycle stages and trigger container-based actions that can be composed as part of a delivery workflow.

Finally, Johnson mentioned the benefits of ChatOps and automation, and "Why You Need a Software Delivery Machine". Discussing the recent work of the Atomist team, he has seen many engineering teams benefit from using developer-focused instant messaging tooling like Slack and Microsoft Teams to trigger automated delivery practices, such as the creation of a well-formed application template, or the deployment and execution of tests within an environment.

With a reference to the famous quote from Mark Andresson, Johnson concluded the conversation by stating that as much as "software is eating the world", he believes that "software developers are eating operations". Accordingly, now is the time to take a step back and make sure the industry is taking a strategic approach to how operational tasks can be codified and automated. He noted that "operations isn't going anywhere", and that "ops-focused skill sets will be in demand for many years to come", but much of the higher-level delivery-focused aspects of this discipline are now becoming the domain of the software engineer.

The reaction to the announcement has been generally positive online, with over 170 signatories of the manifesto at the time of publication of this news piece, although Stefan Tilkov, CEO at INNOQ, did note on Twitter that the "'70s scripting language' hate seems a little odd".

Authors of the manifesto include: Kenny Bastani, Marc Holmes, Rod Johnson, Jessica Kerr, Mik Kersten, Russ Miles, Erin Schnabel, and Matt Stine. The manifesto also acknowledges the help and refinement of many members within the software delivery community. The Software Defined Delivery Manifesto can be read in full on the supporting website.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

GitOps by alexis richardson

The ideas described here are very similar to GitOps, in which we deliver software by automating convergence of an entire stack from observed state to desired state. We need to combine IaC, immutable infra, Git-based workflows, model-driven deployment, etc to make this work. The criticism of scripts is valid: you can't do GitOps by chucking a load of scripts into CI jobs and hoping that deployment will always converge.

www.weave.works/blog/what-is-gitops-really

Autonomic Computing by Greg Liebowitz

Apache Brooklyn is one of the only orchestration engines I've seen that is actually based on an implementation-independent declarative model (OASIS TOSCA blueprints). But developers would much rather stick to their beloved shell scripts and Terraform files over a monolithic control plane.

Rod is a visionary but I think the items outlined have largely been addressed in current tooling. What still remains elusive are the "self-X" autonomic systems that IBM had envisioned in 2001, i.e. self-configuring, self-optimizing, self-healing, and self-protecting.

Re: GitOps by Daniel Bryant

Agreed in regard to GitOps, Alexis, and Rod did mention this approach in passing when I was chatting to him.

Re: Autonomic Computing by Mark N

"We recognize prior art. This is not a work of invention but of articulation, of a timely and much needed approach."

Re: Autonomic Computing by Greg Liebowitz

Got it. More sponsored vendor-speak from Atomist, Weave, and Pivotal.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

5 Discuss
BT