BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles DevOps and Cloud InfoQ Trends Report - February 2019

DevOps and Cloud InfoQ Trends Report - February 2019

This item in japanese

Bookmarks

Key Takeaways

  • Kubernetes has effectively cornered the market for container orchestration, and is arguably becoming the cloud-agnostic compute abstraction. The next "hot topics" in this space appear to be "service meshes" and developer experience/workflow tooling.
  • Chaos Engineering has moved into the early adoption phase. We have separated out the topic of "resilience engineering", as this was previously being conflated with chaos engineering, and we have placed this as within the innovator adoption phase.
  • We are following the innovative developments with edge computing, ML inference at the edge, and "originless" computing with keen interest. 
  • The book "Accelerate" by Nicole Forsgren, Jez Humble and Gene Kim, has helped to define several early adopter topics within the domains of digital transformation and high-performing teams.
  • Although a potentially overloaded buzzword, we are tracking "AIOps" within the innovator space, and more specifically the potential value offered by ML for operational insight and alerting.
  • We see that there is a danger of conflating continuous integration (CI) and continuous delivery (CD), and also conceptually placing practice and tooling together in the same category. We see applications/services that are being marketed as CI tooling as within the late majority adoption phase, but best practices here as still within the early majority phase. We also believe CD practices are in the early majority phase.
  • We are following the recent work around the software-defined delivery manifesto and "code vs config" approaches for delivery and pipeline automation with interest.
     

Both InfoQ and QCon focus on topics that we believe fall into the “innovator, early adopter, and early majority stages”. What we try to do is identify ideas that fit into what Geoffrey Moore referred to as the early market, where “the customer base is made up of technology enthusiasts and visionaries who are looking to get ahead of either an opportunity or a looming problem”. We are also on the lookout for ideas that are likely to “cross the chasm” to broader adoption.  It is perhaps worth saying, in this context, that a technology’s exact position on the adoption curve can vary. As an example, microservices is widely adopted at this point amongst Bay Area companies, but may be less widely adopted, and perhaps less appropriate, elsewhere.

This article provides a summary of how we currently see the “cloud computing and DevOps” space, which focuses on fundamental architectural patterns, realisation of patterns in technology frameworks, and the design processes and skills that a software engineer must cultivate.

In this edition of the cloud computing and DevOps trend report we believe that Kubernetes has well and truly cornered the domain of container orchestration, and is arguably becoming the default cloud-agnostic compute abstraction. However, Kubernetes is not a complete Platform-as-a-Service (PaaS), which is arguably what most organisations require for effective deployment and operation of software, and so the next "hot topics" in this space appear to be "service meshes" for managing interservice communication and release control, and developer experience and workflow tooling to allow engineers to implement effective dev-test-deploy-observe cycles.

We believe that the topic of chaos engineering has moved into early adopter, largely due to the increased promotion by the Netflix team and the O'Reilly Chaos Engineering book authors, and tooling such as the Chaos Toolkit and Gremlin's as-a-service offerings. Based on discussions with John AllspawCasey RosenthalNora Jones and other members of the community, we have separated out the topic of "resilience engineering", which we have previously conflated with chaos engineering, and placed this within the innovator category.

We are following the developments with edge computing, ML inference at the edge, and "originless" computing with keen interest. Work here is mostly within the innovator space, but the public cloud vendors and innovators like Cloudflare are clearly investing large amounts of resources here.

The book "Accelerate" by Nicole Forsgren, Jez Humble and Gene Kim, has helped to define several topics within the domains of digital transformation and high-performing teams. Based on discussions with external contributors we are now tracking with interest topics such as "measuring [for high] performance", evidence-based transformations, and transformational leadership. We see these as early adopter, and are also keen to publish more articles within this space on InfoQ.

Although a potentially overloaded buzzword, we are tracking "AIOps" within the innovator space, and more specifically the potential value offered by ML for operational insight and alerting. As the software systems being built become more complex and distributed, this type of technology could complement distributed (fullstack) tracing for problem detection and help reduce the search space during fault isolation.

We see that there is a danger with conflating continuous integration (CI) and continuous delivery (CD), and also lumping practice and tooling together in the same category -- indeed we acknowledge that our own published articles and associated article topic tagging is far from perfect within these domains. We see applications/services that are being marketed as CI tooling as within the late majority adoption phase, but CI best practices are still within the early majority phase. We also believe that CD is within the early majority category.

We are keenly following recent work around the software-defined delivery manifesto and "code vs config" for delivery and pipeline automation closely. The Atomist and Pivotal teams are currently driving a lot of the conversation (and technology) within this space.

For context, here is what the topic graph looked like for the first half of 2018. The 2019 version is at the top of the article.

The following is a lightly-edited copy of the corresponding internal chat-log between several of the InfoQ cloud computing and DevOps topic editors, which provides for more context for our recommended positioning on the adoption graph.

Daniel Bryant, Independent Tech Consultant, Product Architect at Datawire, and InfoQ News Manager:

As a starter-for-ten, I'm thinking that Chaos Engineering is moving to Early Adopter (EA), what with the Gremlin and AWS teams' talking a lot about this as a re-imagining/extension to traditional DR/BC; I also think that the minimalist container images move to Early Majority (EM) -- what with build-your-own kits like LinuxKit entering in Innovator -- distributed tracing potentially to EM, general DevOps to Late Majority (LM), and potentially containers, IaC and continuous delivery to LM? 

We need to put service meshes in here somewhere, as they are a super-hot topic, and I'm thinking EA?

I'm interested in hearing what your thoughts are please, and also to ask if we need to move things around, or add or remove topics?

Helen Beal, DevOps consultant, coach, trainer, speaker and writer.

My thoughts are:

  • CI/CD to late majority as you say
  • Add AIOps to Innovator along with ML for Ops?
  • Should ‘DevOps Toolchain’ be there? In early majority?
  • Add Electric Cloud and Xebia Labs to DevOps Dashboard
  • add AI to dashboards?
  • ChatOps to early majority

Steffen Opel, is managing partner at Utoolity

Agreed on moving DevOps, containers, IaC, and CD to LM; everyone not adopting this will be left behind quickly. Adding service meshes to EA sounds good as well.

I'd also join the proposal to add AIOps as a topic to innovator, though I see it more as the generalized superset of and thus replacement for the current “ML for Ops insight and alerting topic” (which also covers other areas like predictive scaling for example, which has been recently released for AWS EC2).

However, while I see further increasing interest in both ChatOps and fullstack tracing, I don't think their adoption qualifies for EM already (at least in terms of “established way of handling most/all of your operations”), and tooling maturity around ChatOps leaves a lot to be desired still. On the other hand, “Log aggregation/analytics” seems to have surpassed that threshold (complementing “Centralized log aggregation”, maybe these topics can be merged now).

Finally, I'm still missing 'edge computing' as an innovator/early adopter topic (e.g. services like Lambda@Edge, AWS Snowball Edge, AWS Greengrass, Cloudflare Workers) - these capabilities are quickly becoming accessible once provided by already widely adopted cloud services like CloudFront, Lambda, or just recently Cloudflare.

Chris Swan, Fellow, VP, CTO for the Global Delivery at DXC Technology

I'd go along with what Daniel, Helen and Steffen have already said.

What I'm not seeing here is the swing to code vs config for CD per Atomist etc. and the Software Defined Delivery (SDD) manifesto.

On the edge topic that Steffen highlighted I also wonder if 'originless' will turn into a thing? It's at that awkward stage right now where it's hard to tell whether the early demos are pointless parlour tricks or the beginning of a new trend (but when combined with IoT and the quest for lower UI latency I'm inclined to give it some credence).

One nitpick - I know people lump together CI/CD because they're often done with some of the same tools, but as we all know too well they're not remotely the same thing. CI is definitely in the late majority now, because it's (relatively) easy to do in a traditional organisation. CD is I think struggling to get to late majority because it doesn't happen without that Dev/Ops to DevOps reorganization, and many companies simply haven't got there yet. The innovation gap that we're used to early on in the adoption curve is actually a giant canyon now later along that curve because adopting CD isn't just a technology pick.

Helen Beal 

I think that’s a really important clarification Chris et al -

Manuel Pais, independent DevOps and Continuous Delivery Consultant

On the CI/CD discussion, my concern is that we're conflating tooling and practices which are at different stages. CI/CD tooling I agree is moving to late majority, but only some of the practices (as defined in the CI and CD books) are commonly applied. Many organizations are not practicing CI as such (a recent Jenkins survey clearly shows a lot less builds are performed than you'd expected when practicing CI), they are simply using a CI server. That's illustrated also by Dave Farley's rant earlier this year on (not) branching

(note: I'm guilty of conflating them as well, but it wasn't such a clear issue when the topic was in early majority)

If we split between tools and practices, I'd put tools in late majority and practices (still) in early majority. I also suggest mapping some of the practices in the "Accelerate" book on the topics page, that makes for a very good reference/framework on important practices for high performing teams.

As for other changes here are my thoughts:

  • DevEx -> move to EA - being pushed by Atomist as a common concern 
  • GitOps -> move to EA - being pushed forward by Weave
  • Chaos Eng -> move to EA - being pushed forward by Gremlin
  • AIOps (what we used to refer as "ML for Ops insight and alerting") -> still in Innovator
  • Immutable infrastructure in application pipelines -> move to EA
  • Log aggregation/analytics -> move to EM
  • SRE/CRE -> this needs to be split up, Google is still figuring out CRE (should be moved to Innovator) while SRE should move to EM
  • K8s -> move to EM
  • Self-service platforms -> definitely move to EM
  • Software-defined networks -> I'm not an expert here but it seems to me to be in EM
  • Containers/General DevOps/infrastructure as code -> move to LM
  • centralized log aggregation -> duplicate of "Log aggregation" (see above, should be single entry in EM)

I would also add this new topic in Innovator:

  • software defined delivery -> being pushed forward by Atomist and the SDD manifesto

Daniel Bryant

Thanks all for the excellent discussion -- I’ll be sure to summarise this for the topic report.  The split between the maturity of CI and CD (and the associated next level split between practices and tooling) is especially interesting to me, as is something we should probably focus on in regard to our own tagging of topics.

I also like Manuel’s suggestion of incorporating key ideas that are gaining adoption from the great work of the “Accelerate” book from Forsgren, Humble and Kim into this edition of the topic graph. This book is rapidly becoming a go-to reference within the industry, as are several other books by the same authors that Manurel recently recommended in the InfoQ editors list of recommended reading article series.

The InfoQ editorial team is built by recruiting and training expert practitioners to write news items and articles and to analyse current and future trends. Apply to become an editor via the editor page, and get involved with the conversation.

About the Authors

Daniel Bryant is leading change within organisations and technology. His current work includes enabling agility within organisations by introducing better requirement gathering and planning techniques, focusing on the relevance of architecture within agile development, and facilitating continuous integration/delivery. Daniel’s current technical expertise focuses on ‘DevOps’ tooling, cloud/container platforms and microservice implementations. He is also a leader within the London Java Community (LJC), contributes to several open source projects, writes for well-known technical websites such as InfoQ, DZone and Voxxed, and regularly presents at international conferences such as QCon, JavaOne and Devoxx.

Chris Swan is CTO for the Global Delivery Organisation at DXC.technology, where he leads the shift towards design for operations across the offerings families, and the use of data to drive optimisation of customer transformation and service fulfilment. He was previously CTO for Global Infrastructure Services and General Manager for x86 and Distributed Compute at CSC. Before that he held CTO and Director of R&D roles at Cohesive Networks, UBS, Capital SCF and Credit Suisse, where he worked on app servers, compute grids, security, mobile, cloud, networking and containers.

Steffen Opel is managing partner at Utoolity, a provider of tools for cloud computing operations and software development processes. With a formal education in C++, and an early focus on rich client technologies, he joined the paradigm shift to RESTful web service architectures early on. The major industry move towards cloud computing refueled his interests in thorough automation of development processes and his focus shifted to DevOps scenarios, where he enjoys API driven development in agile teams.

Helen Beal is a DevOps consultant, coach, trainer, speaker and writer. Her focus is on helping organisations optimise the flow from idea to value realisation through behavioural, interaction based and technology improvements. Fond of llamas. Once she saw a flamingo lay an egg. You can reach her at helen.beal@infoq.com - especially if there's something you want me to write about.

Manuel Pais is a DevOps and Delivery Consultant, focused on teams and flow. Manuel helps organizations adopt test automation and continuous delivery, as well as understand DevOps from both technical and human perspectives. Co-curator of DevOpsTopologies.com. DevOps lead editor for InfoQ. Co-founder of DevOps Lisbon meetup. Co-author of the upcoming book "Team Guide to Software Releasability". Tweets @manupaisable

Rate this Article

Adoption
Style

BT