BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News SPACE, a New Framework to Understand and Measure Developer Productivity

SPACE, a New Framework to Understand and Measure Developer Productivity

This item in japanese

Bookmarks

A recent paper by researchers at GitHub, University of Victoria, and Microsoft delves into developer productivity to propose a new approach to defining, measuring and predicting it. InfoQ has taken the chance to speak with the paper lead author, GitHub vice-president of research & strategy Nicole Forsgren.

According to the authors, SPACE was developed to capture different dimensions of developer productivity and dispel persisting myths about it, such as the idea that productivity is about activity volume, tools, or individual performance.

The framework provides a way to think rationally about productivity in a much bigger space and to choose metrics carefully in a way that reveals not only what those metrics mean, but also what their limitations are if used alone or in the wrong context.

SPACE is the acronym for satisfaction, performance, activity, communication, and efficiency. Each of these dimensions is key to understanding and measuring productivity, according to the researchers. For each of them, they suggest a number of distinct metrics that apply to different levels, including individual-, team- or group-, and system-level. Interestingly, SPACE does not advocate for using all of the metrics at once, rather to carefully select a reduced set of metrics that span across all three levels and capture different productivity dimesions.

InfoQ has taken the chance to speak with the paper lead author, GitHub vice-president of research & strategy Nicole Forsgren, to learn more about SPACE.

InfoQ: Many studies focus on developer productivity, myths and stereotypes about it abound, as you explain in your paper, why has productivity proven to be such an elusive topic?

Nicole Forsgren: Productivity remains pretty elusive because it cannot be reduced to a single dimension or metric.

Over a decade of research has found that productivity is complex and nuanced; this has important implications for software development teams. Far too often teams or managers attempt to measure developer productivity with "one metric that matters" like lines of code. But we know that productivity is much more than that, because when we ask people about their productivity, they will talk about completing work, working with others, getting into a “flow,” and improving engineering outcomes.

InfoQ: What is the worst misconception in thinking about productivity, in your opinion?

Forsgren: One of the most common myths — and potentially most threatening to developer happiness — is the notion that productivity is all about developer activity, things like lines of code or number of commits. More activity can appear for various reasons: working longer hours may signal developers having to "brute-force" work to overcome bad systems or poor planning to meet a predefined release schedule.

In fact, our 2020 Octoverse Report included a deep dive on developer productivity, and we saw increased development work—both time spent and amount of work—across all time zones we investigated. Developers may be taking advantage of flexible schedules to manage their time and energy, which contributed to this sustained activity. If work happens at the expense of personal time and breaks to maintain a healthy work life balance, this pace may not be sustainable in the long run.

On the other hand, increased activity may reflect better engineering systems, providing developers with the tools they need to do their jobs effectively, or better collaboration and communication with team members in unblocking their changes and code reviews.

As you can see, activity metrics alone do not reveal which of these is the case, so they should never be used in isolation either to reward or to penalize developers. Even relatively straightforward metrics such as number of pull requests, commits, or code reviews are prone to errors because of gaps in data and measurement errors. Systems that report these metrics will miss the benefits of collaboration seen in peer programming or brainstorming. Finally, developers often flex their hours to meet deadlines, making certain activity measures difficult to rely on in assessing productivity.

InfoQ: In your paper, you present a multidimensional framework, SPACE, as an attempt to do away with myths and misconceptions about developer productivity. Could you explain how you came to conceive it?

Forsgren: This work was informed by prior experience with users and customers, and seeing the gaps in how they measure developer productivity.

In a common example, we’re seeing that many teams currently measure developer productivity with just one dimension of the SPACE framework, such as activity. Activity is a count of actions or outputs completed in the course of performing work, including producing lines of code, number of commits, or pull request turnaround time. This single focus is narrow and at the exclusion of other important factors.

In another example, we worked with a team that focused on PR turnaround time and committed themselves to an SLA of one hour. This resulted in interruptions to coding time for team members, so developers started ignoring PRs for a few hours so they could focus on development. In response, team leads and managers turned on alerts for all team members, which contributed to alert fatigue. Metrics influence decisions and behavior, so any time we focus a single metric or even a few metrics from just one dimension, we risk causing changes in behavior that cause unintended consequences – in this example, interrupting coding time. We also miss other important signals of productivity like collaboration, satisfaction, and performance.

InfoQ: How should organizations change the way they measure developer productivity if they want to use SPACE?

Forsgren: The SPACE framework provides a way to logically and systematically think about productivity in a much bigger space and to carefully choose balanced metrics linked to goals—and how they may be limited if used alone or in the wrong context. We encourage readers to take a look at the five categories in relation to their team or organization, which will help illuminate some of the tradeoffs that may not be immediately obvious.

InfoQ: Assuming SPACE is not the last word on this subject, how do you envision the further evolution of research about developer productivity? What factors will shape it in the near future?

Forsgren: I’m excited to think about the way new technologies are accelerating and evolving developer work, and what this could mean for productivity. For example, low-code and no-code tooling reduces the marginal cost of prototyping and extends the developer audience. This has implications for experimentation and innovation loops for the developers themselves, then extends and amplifies out to their teams and organizations. As ML and AI continue to advance, there are interesting opportunities to see how we can help developers build their prototypes and tooling faster, and even help discover novel solutions.

Another interesting evolution is the shift in the global workplace. The pandemic has shown many companies that we can work from home and be productive. So as more companies continue to embrace the long-term value of incorporating a remote or hybrid workplace model, and even expand to increasingly distributed teams, I think we’ll see a shift in developer collaboration patterns and flexibility, with increases in satisfaction and well-being.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Money

    by Javier Paniza,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Satisfaction, performance, activity, communication, and efficiency. You left out money. Obviously when you talk about satisfaction is not the satisfaction of the customer, the one willing to pay for your software.

    What matter if you finished your work on time and with high quality if nobody is willing to pay for it. Having it into account make you develop the code with a different perspective and making decisions in a different way.

  • Re: Money

    by Sergio De Simone,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi there,

    thanks for your comment. As far as I understand SPACE, you are right in thinking satisfaction is mostly meant as developer satisfaction.

    On the other hand, it is my understanding that customer satisfaction is included under "performance". From the original paper:

    "A developer who produces a large amount of code may not be producing high-quality code. High-quality code may not deliver customer value. [...] For these reasons, performance is often best evaluated as outcomes instead of output. [...] Impact: Customer satisfaction, customer adoption and retention, feature usage, cost reduction."

    Maybe this is what you were thinking of, maybe not...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT