BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles DevEx, a New Metrics Framework from the Authors of SPACE

DevEx, a New Metrics Framework from the Authors of SPACE

Bookmarks

Key Takeaways

  • Researchers Abi Noda, Dr. Nicole Forsgren, Dr. Margaret-Anne Storey, and Dr. Michaela Greiler have published a paper that provides a practical path for improving productivity, which is focused on Developer Experience (DevEx).
  • Developer experience focuses on the lived experience of developers and the points of friction they encounter in their everyday work. The authors assert that focusing on developer experience is the key to maximizing engineering effectiveness, and introduce a framework for measuring and improving DevEx.
  • Organizations can improve the developer experience by identifying the top points of friction that developers encounter, and then investing in improving areas that will increase the capacity or satisfaction of developers.
  • The DevEx framework distills the factors affecting developer experience into three dimensions: feedback loops, cognitive load, and flow state. Leaders can select metrics within these three dimensions in order to measure and identify areas to focus on that would ultimately drive productivity improvements.
  • Surveys provide a practical starting point for capturing a holistic set of measures to fully understand the developer experience. Effective survey programs require attention to survey design, as well as the ability to break down results by persona and team and compare results against internal and external benchmarks.

A recently published research paper outlines a new framework for measuring and improving developer productivity.

This framework, referred to as the DevEx framework, was authored by Abi Noda, Dr. Margaret-Anne Storey, Dr. Nicole Forsgren, and Dr. Michaela Greiler.

Leaders have long sought to improve the productivity of their engineering organizations in order to help their businesses move faster, build new products, and tap into new and emerging trends.

However, knowing what to focus on in order to achieve this goal has remained elusive, despite recent methods such as DORA and SPACE. This new framework aims to address this gap.

Drawing on their extensive research and experience, the authors assert that focusing on developer experience is the key to maximizing engineering effectiveness.

Their paper presents frameworks which distill developer experience into its three core dimensions and provide a methodology for measuring it.

This article includes a summary of the paper’s key points along with commentary from the lead author, Abi Noda. Here’s also a link to the full paper.

Measuring productivity is hard

In a recent article, Google researchers Ciera Jaspan and Collin Green suggest two reasons why measuring developer productivity is so challenging: software engineering is not repeatable, and a developer’s productivity is highly affected by outside forces.

As for the latter, an outside force could be the complexity of the work (and whether it is necessarily that complex), the interactions with others to get the job done, or the organizational design. There are also factors that specifically affect developers, including flaky tests, build speeds, and technical debt.

The other reason why measuring productivity is difficult is that software development is a creative endeavor: it is not about the production of uniform, interchangeable outputs. "Attempts to quantify work productivity by borrowing methods from operating machinery are not suited to software engineering."

"We have to remember that we’re working with a fundamentally human system. To understand how the system can be improved, we need to find out from human beings what they are experiencing."

Developer experience offers a new lens

Developer experience provides a new way of understanding developer productivity: from the perspective of developers themselves. Developer experience encompasses how developers "feel about, think about, and value their work," and focuses on the everyday realities and friction that developers face while performing their work.

Prior research has identified numerous factors that affect developer experience: for example, interruptions, unrealistic deadlines, and friction in development tools negatively impact how developers feel about their work. Having clear tasks, well-organized code, and pain-free releases improve developer experience.

Organizations can improve developer experience by identifying the top points of friction that developers encounter, and then investing in improving areas that will increase the capacity or satisfaction of developers. For example, an organization can focus on reducing friction in development tools in order to allow developers to complete tasks more seamlessly. Even a small reduction in wasted time, when multiplied across an engineering organization, can have a greater impact on productivity than hiring additional engineers.

More companies are focusing on developer experience

A recent study from Gartner revealed that 78% of surveyed organizations have a formal developer experience initiative either established or planned, while a similar study from Forrester showed that 75% of enterprise leaders regarded developer experience as crucial to executing business strategy. These findings indicate a growing recognition of the substantial benefits that can be derived from investing in developer experience programs.

Related studies from McKinsey and Stripe have further validated the business impact of optimizing work environments for developers, and consequently there is an increasing number of organizations establishing C-level initiatives around developer experience.

The three dimensions of developer experience

In their paper, the authors distill the previously identified factors affecting developer experience into three core dimensions: feedback loops, cognitive load, and flow state.

"This framework was informed by our prior research and experience, and seeing the gaps in how organizations approach developer productivity and experience. Our goal was to create a practical framework that would be easy for people to understand and apply, and capture the most important aspects of developer experience."

To summarize each of the dimensions:

Feedback loops refer to the speed and quality of responses relative to actions performed. Fast feedback loops are a critical component of efficient development processes, as they enable developers to complete their work expeditiously and with minimal friction. On the other hand, slow feedback loops can lead to disruptions in the development cycle, causing developers to become frustrated and delays to occur. Hence, organizations must endeavor to shorten feedback loops by identifying areas where development tools can be accelerated and human hand-off processes can be optimized, such as build and test processes or development environment setup.

Cognitive load encompasses the amount of mental processing required for a developer to perform a task. High cognitive load can result from challenges like poorly documented code or systems, forcing developers to devote extra time and effort to completing their work and avoiding mistakes. To improve the developer experience, teams and organizations should aim to alleviate cognitive load by removing any unnecessary hurdles in the development process.

Flow state refers to the mental state of being fully absorbed and energized while engaged in an activity, characterized by intense focus and enjoyment. This is often referred to as "being in the zone." Experiencing flow state frequently at work can lead to greater productivity, innovation, and employee development. Similarly, research has shown that developers who derive satisfaction from their work tend to produce higher quality products. Therefore, teams and organizations should focus on creating optimal conditions that promote the flow state to foster employee well-being and performance.

Taken together, these three dimensions encapsulate the full range of friction types encountered by developers. Although developer experience is complex and nuanced, teams and organizations can take steps toward improvement by focusing on these three key areas.

Taken together, these three dimensions encapsulate the full range of friction types encountered by developers. Leaders can surface opportunities to improve productivity by selecting metrics within the three dimensions.

"The DevEx framework provides a way to improve developer productivity in a systematic and developer-centric way. We encourage readers to capture metrics within each of the three dimensions in order to illuminate areas of friction as well as effectively prioritize the areas that will have the biggest impact on the organization’s intended outcomes."

What to measure

The first task for organizations looking to improve their developer experience is to measure where friction exists across the three previously described dimensions. The authors recommend selecting topics within each dimension to measure, capturing both perceptual and workflow metrics for each topic, and also capturing KPIs to stay aligned with the intended higher-level outcomes.

Measure topics across the three dimensions. For example, an organization may choose to measure test efficiency (Feedback loops), codebase complexity (Cognitive load), the balance of tech debt (Cognitive load), and time for deep work (Flow state). Some topics may map onto more than one dimension.

"We advocate that leaders opt for metrics that span each of the three dimensions to acquire a comprehensive understanding of the developer experience. For instance, a topic that could be evaluated within the Feedback Loops dimension is Test Efficiency, while Codebase Complexity could be measured within the Cognitive Load dimension."

Capture both perceptual and workflow measures for each topic. Measuring developer experience requires capturing developers’ perceptions - their attitudes, feelings, and opinions - in addition to objective data about engineering systems and processes. This is because neither perceptual nor workflow measures alone can tell the full story.

For instance, a seemingly fast build process may feel disruptive to developers if it regularly interrupts their work progress. Conversely, even if developers feel content with their build processes, using objective measures such as build time may reveal that feedback loops are slower than they could be and development workflows less streamlined than they might be. Hence, analyzing both perceptual and workflow measures is necessary to gain a complete understanding of the points of friction that developers encounter in their everyday work.  

Measure KPIs to stay focused on driving business outcomes that matter. KPIs serve as north-star metrics for DevEx initiatives. Well-designed KPIs should measure the outcomes that the business seeks to drive, including improvements to productivity, satisfaction, engagement, and retention.

How to measure

The authors recommend starting with surveys to capture the above metrics. Surveys provide the advantage of being able to capture all aspects of the developer experience, including KPIs, perceptual measures, and workflow measures.

"Companies like Google, Microsoft, and Spotify have relied on survey-based developer productivity metrics for years. However, designing and administering surveys can be difficult, so we hope our framework provides a good starting point for leaders to follow."

Given the importance of surveys, the authors outline several important considerations for survey programs to be successful:

  • Design surveys carefully. Poorly designed survey questions lead to inaccurate and unreliable results. At minimum, the authors say that survey questions should be based on well-defined constructs, and rigorously tested in interviews for consistent interpretation.  
  • Break down results by team and persona. A common mistake made by organizational leaders is to focus on company-wide results instead of data broken-down by team and persona (e.g. role, tenure, seniority). Focusing only on aggregate results can lead to overlooking problems that affect small but important populations within the company.
  • Compare results against benchmarks. Comparative analysis can help contextualize data and help drive action. For example, developer sentiment toward tech debt is commonly negative, making it difficult to identify problems or gauge their scale. Benchmarks allow leaders to see when teams have lower sentiment scores than their peers, and when organizations have lower scores than their industry competitors. These signals flag notable opportunities for improvement.
  • Mix in transactional surveys. In addition to periodic surveys, organizations can use transactional surveys to collect feedback based on specific touchpoints. For example, Platform teams can use transactional surveys to prompt developers for feedback when a specific error occurs during the installation of a CLI tool. Transactional surveys provide a continuous stream of feedback and can generate higher quality responses due to the timeliness of their posed questions.
  • Watch out for survey fatigue. Many organizations struggle to sustain high participation rates in surveys over time. A lack of follow-up action commonly causes developers to feel that repeatedly responding to surveys is not a worthwhile exercise. It is therefore critical that leaders and teams follow up on surveys.

Conclusion

The DevEx framework provides a practical framework for understanding developer experience, while the accompanying measurement approaches systematically help guide improvement. Organizations should begin measuring developer experience now, even if they have not yet established or planned a formal DevEx investment.

About the Author

Rate this Article

Adoption
Style

BT