Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Q&A with Gene Kim on the State of DevOps Survey

Q&A with Gene Kim on the State of DevOps Survey

The 4th edition of the State of DevOps survey is out. InfoQ talked to the survey's co-author Gene Kim to understand what the goals are for this edition, how the data is analysed and what have we learned so far from past surveys.  

InfoQ: The 2015 State Of DevOps Survey is now open. What is the purpose of this research?

Gene: Studying high-performing technology organizations and how DevOps impacts performance has been one of the most educational and gratifying projects I’ve ever had the privilege of working on. Dr. Tom Longstaff, a mentor of mine, once told me, “The goal of science is to explain the most amount of observable phenomena with the fewest number of principles, confirm deeply held intuitions, and reveal surprising insights.”

In my opinion, our study of high-performing organizations fulfills all of these scientific goals. Our research goal has always been to measure the performance difference between the high and low performers, in both IT performance and organizational performance, and better understand what cultural norms and work practices best enable high performance.

Last year at the Velocity Conference, we presented our findings from the over 9,200 respondents who took the survey. I would summarize our findings as follows:

  • High performers are more agile
    • 8x more frequent production deployments
    • 8000x faster deployments (high performers complete deployments in minutes or hours, compared with low performers who required weeks or months)
  • High performers are more reliable
    • 2x higher success rates during deployments
    • 12x faster fixes when things went wrong

And even more surprisingly, we found that high performers had much higher organizational performance. They were 2.5x more likely to exceed profitability, market share, and productivity goals than low performers, and employees reported far higher levels of job satisfaction. Furthermore, our analysis of the stock market performance of high-performing IT organizations suggests that they had 50% higher market capitalization growth over low IT performers over a three-year period.

For me, it shows that DevOps is not only great for Dev and Ops but also creates competitive advantage for the organizations we serve.

We found that the top predictors of both organizational performance and IT performance were:

  • Version control of all production artifacts
  • Peer review of production changes (vs. external change approval)
  • Proactive monitoring of the production environment
  • High-trust culture
  • Win-win relationship between Dev and Ops
  • High job satisfaction

This is exciting because it’s findings like this that enable all of us to replicate the outcomes that we observe in the high performers. By doing this, we demystify high performance, and break it down to specific practices so that the “rest of us” can replicate high performing outcomes.

InfoQ: What is the goal of the 2015 State Of DevOps Survey?

Gene: The 2015 State Of DevOps Survey team comprises Puppet Labs and IT Revolution. This year, we continue our quest to better understand the top predictors of organizational performance and IT performance.

As a community, we have many hypotheses about what enables high performance, or at the very least, what contributes to high performance. Some of the things we're trying to test include:

  • Business value of DevOps:
    • How DevOps creates business value and impacts the bottom line
    • How DevOps increases developer productivity and how that impacts organizational performance.
  • Planning and organizing the DevOps initiative:
    • How high performers organize their DevOps (e.g., shared services, dedicated teams, level of sponsorship, etc.)
    • How high performers choose where to deploy DevOps (e.g., system of record vs. engagement, nature of the business process, architecture of the service, etc.)
  • Leadership and cultural norms:
    • High vs. low trust
    • Seeing problems, swarming problems and enabling global improvements
    • Perception of management and leadership
    • Gender composition of the workforce
  • Technical and work practices:
    • Better understanding of “where to start”
    • Better understanding of test and deployment practices

I’m particularly excited by this because I think the result of this research will provide better guidance to both leaders and practitioners, as well as help create more effective business cases.

InfoQ: What have been the reactions to the 2014 State of DevOps report since it was published last May?

Gene: I’m so delighted at the feedback we’ve gotten. I think we all desire a more scientifically rigorous approach about making claims about how we should do our work. And it’s studies like this that make that possible.

I love the diagram below, because it shows where we believe research like this fits. Like last year, this year’s DevOps survey is a cross-population study, designed to explore the link between organizational performance and organizational practices and cultural norms.

What is a cross-population study? It’s a statistical research technique designed to uncover what factors (e.g., practices, cultural norms, etc.) correlate with outcomes (e.g., IT performance). Cross-population studies are often used in medical research to answer questions like, “Is cigarette smoking a significant factor in early mortality?”

Properly designed cross-population studies are considered a much more rigorous approach of testing efficacy of what practices work than say, interviewing people about what they think worked, ROI stories from vendors, or collecting “known best practices.”

InfoQ: Let’s start with one of the main conclusions from the 2014 report: “high-performing IT organizations were twice as likely to exceed their profitability goals”.

Gene: I love this finding, because it validates something that anyone in technology already knows but is often completely overlooked (or not believed) by business management.

Clearly, IT matters, and apparently, it matters far more than we think. Our technology systems are not only the central nervous system of the organization, running the back-office of our organizations; it is now the majority of the muscle-mass that we rely upon to achieve our organizational goals.

In most organizations, 50% of all capital spending is technology-related, and 95% of capital projects contain an IT component. IT is increasingly the means of how we create value and acquire customers.

To put this into perspective, in the 1980s, researchers observed that Toyota Motor Corporation had productivity four times higher than other manufacturers. That is, they were able to produce twice the output (e.g., assembled cars ready to be sold) with only one-half the inputs (e.g., floor space, work in process, etc.). Over the next decade, the adoption of Lean philosophies and methods led to an unprecedented surge in worker productivity, better delivery performance, reduced inventory levels and higher customer satisfaction and employee happiness.

Our research is showing that in the technology space, we’re not seeing a four-fold difference in productivity. Instead, we’re seeing orders of magnitude higher productivity, and that organizational performance seems to be correlated with high IT performance.

Clearly, this gap will enable those high performers adopting DevOps to win in the marketplace, at the expense of those who are not.

InfoQ: What are potential links between IT and business performance? Could high IT performance be simply a consequence of profitable (high business performance) companies investing strongly in IT?

Gene: One of the lessons I cherish the most came from the work I did with the Software Engineering Institute at Carnegie Mellon University, where I learned this phrase: “high performers in every profession accelerate away from the herd—the best always get better.”

Put a different way, high performers win in the marketplace at the expense of the lower performers—potentially depriving them from the capital they need to improve their IT processes and systems. But this is a part of the human condition, applying far more broadly than just technology.

Regardless of what industry our organization is operating in, we are increasingly dependent upon technology to acquire customers and deliver value to them.

It’s not just about doing the right things—it’s about doing the right things right. To paraphrase Dr. W. Edwards Deming said, “DevOps is not compulsory... but then again, neither is survival.”

InfoQ: Another conclusion highlighted in that report is that “organizational culture matters”, in the sense that high-trust organizations encourage core DevOps practices. Do you believe (from the survey data and/or personal experience) that kind of culture can be instilled in an organization? In other words is it possible to change a broken culture by introducing DevOps?

Gene: I love this quote from Dr. Steven Spear, credited for decoding the Toyota Production System as part of his doctoral thesis at Harvard Business School and establishing the causal mechanisms behind a continuously self-improving system:

“Culture isn’t just touchy-feely kumbaya. Instead, it is the consistent response by a group of people to conditions. When we change culture, we fundamentally shift how people respond to a situation. The most effective way is for senior leaders to change the conversation from ‘did you carry your orders out?’ to ‘what did you learn today?’”

Dr. Spear went on to establish that designing perfectly safe systems is likely beyond our abilities, “safe systems are close to achievable when (a) complex work is managed so that problems in design are revealed, (b) problems that are seen are solved so that new knowledge is built quickly, and (c) the new knowledge, although discovered locally, is shared throughout the organization.”

We can see this in so many practices that high-performing DevOps organizations adopt: continuous testing and pervasive production telemetry, the value we place on safe systems of work, high-trust cultures, blameless postmortems, using shared source code repositories so that best known configurations can be globally adopted, and so many more…

All my personal experiences have shown that we can change cultures, and that creating the conditions for these practices to exist is the top responsibility of technical leaders.

About the Interviewee

Gene Kim has been studying high performing technology organizations since 1999. He is a multiple award winning CTO, researcher and author. He was founder and CTO of Tripwire for 13 years. He has written three books, including co-authoring “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” and co-authoring the upcoming “DevOps Cookbook.” In 2007, ComputerWorld added Gene to the “40 Innovative IT People Under The Age Of 40” list, and was given the Outstanding Alumnus Award by the Department of Computer Sciences at Purdue University for achievement and leadership in the profession.

Rate this Article