Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Measure Outcomes, Not Outputs: Software Development in Today’s Remote Work World

Measure Outcomes, Not Outputs: Software Development in Today’s Remote Work World

Key Takeaways

  • Software startups need to optimize developer productivity to get both the speed and quality of code necessary to move features into production quickly.
  • There is no standard metric for measuring developer productivity and widely used methods have significant flaws.
  • Code can be measured objectively by analyzing 3 factors: the amount of code put into production, rate of code churn and overall code quality.
  • By focusing on outcomes (code in production, code churn, and code quality) rather than outputs (lines of code and story points) it is possible to measure software developer productivity objectively.
  • Effectively managing developer productivity accelerates product time-to-market, which in turn can create a competitive advantage, increase revenue and improve profitability.

The expression ‘time is money’ is particularly suitable for software startups who are under immense pressure to develop innovative products faster than their competitors. In order to build products more quickly, one of the metrics that startups need to track is the elapsed time from the point a feature is defined to when it is rolled out to customers. This duration is known as Cycle Time. In many cases, company revenues are often directly impacted by the cycle time for product feature delivery. Cycle time depends on how fast developers can write code that can be moved to production. But speed is not the only consideration. The quality of code is equally important. Code quality decides whether product features would work as envisioned and also influences the ease or difficulty of testing and maintaining the product.

Therefore, it is paramount that software startups optimize developer productivity to get both the speed and quality of code necessary to move features into production quickly. This is easier said than done. Even before the COVID-19 pandemic forced most developers to leave the office and write code from home, there was not a standard, agreed upon metric for measuring developer productivity. Now that developers are working remotely in separate locations from product owners, the need for an objective way to measure the productivity and quality of code has become glaringly apparent.

Why Current Measurements Are Flawed

Although there is no standard metric for measuring developer productivity, there are two approaches commonly used to measure software development efficiency. One method is to count the lines of code (LOC) or source lines of code (SLOC) a developer writes. This measurement is based on the premise that the size of the software project and amount of source code for a particular system directly relates to the technical complexity of the system. SLOC is typically used as a way to predict the amount of time and number of people required to develop a software program. Once the project begins, the LOC metric is also often used to estimate programming productivity.

But the issue with using LOC as a metric for developer productivity, is that this measurement prioritizes quantity over quality and can indirectly encourage developers to write code that is inefficient and can create a huge technical debt over time. The LOC metric only captures the code written by the developers, but not what really went to production.  It is possible that a developer writes a bunch of inefficient code that gets refactored or deleted during code review and feature testing.  In fact, with the LOC metric, the more code the developer writes and deletes, the higher the productivity number. So it’s important to look at what part of the code written was actually useful.

The second approach is to measure the number of ‘story points’ completed. A story point is "a metric used in agile project management and development to determine (or estimate) the difficulty of implementing a given story. Elements considered in assigning a story point include the complexity of the story, the number of unknown factors and the potential effort required to implement it."

The issue with story points is that they are arbitrary and measuring them is subjective. What one person considers as one story point, another may consider two. For example, a development team may assign 1 story point to the feature based on the complexity, but another team might consider the same feature to be 2 story points based on their understanding. There is no defined formula or measurement that people can use to determine story points – it is all subjective. Therefore, this form of measurement could be inexact and can lead a developer or a development team to think they are doing a good job, when in reality, they are highly inefficient.

I have worked with many engineering teams in my role as CTO at a software development company that has built technology products for more than 150 startups across the globe. Over the years, my experience has taught me that the code released to production (outcome) is more important than the amount of code each developer writes/ commits (output) every day. In an ideal situation, both should be the same, but multiple factors affect the outcomes.

The most objective way to measure developer efficiency is to analyze the code developers write based on three factors:

  1. How much code is put into production?
  2. What is the rate of code churn?
  3. What is the overall code quality?

These metrics not only reveal how skilled the developers are, but also how well they understand requirements - which is much more challenging these days with the majority of developers working remotely and working on the right priorities.

Measure Productivity through Useful Code

Measuring the useful lines of code is an important aspect of understanding a software developer’s productivity. Usually, developers write code to contribute to a feature.

A typical development process would involve:

  1. Developer committing code to a development branch - which is reworked by the developer based on the review comments. Code may be discarded because it does not adhere to guidelines or due to poor design, which is an indicator of developer skills or lack of them
  2. As a next step, the code is moved to an integration branch where the feature is tested. The developer might have to rework the code to fix issues reported by the test team. These issues could be caused by a poor understanding of the feature or bad code written by the developer.
  3. The product owner sees the feature and if it doesn’t fulfill the requirement, then more code is lost while the developers iterate on this process to get it right. This happens when the requirement is not well understood by the developer or the feature is not specified correctly.
  4. Finally, the code may move to production or get discarded based on the product priorities.

By using lines of code committed as a metric, a developer who builds features with efficient code, minimal iterations and good design will seem less productive as compared to someone who copy pastes code or commits generated files. Therefore, it’s important to measure the ‘Useful or productive lines of code’ (PLOC). PLOC is defined as the lines of code released to production after code review, unit testing and integration testing.  

One could argue that even after code is moved into production, the code may need to be cleanly refactored to remove bloat, technical debt and/or to remove features that are no longer useful to reduce the size of the product, which in turn means lower PLOC. Refactoring is good and it does lead to reduction of code. However, if we consider the productivity trend of 6 months and see that the developer has spent most of the time in refactoring rather than building new features, then there is likely an issue with the quality of code delivered and it might make sense to rewrite the feature from scratch rather than spend time fixing it.

PLOC can be calculated by looking at all the commits to the release branch after identifying and removing anomalies such as large one time commits (typically open source code), duplicate code, generated files etc. and applying a churn factor. Anomalies can be automatically detected by using machine learning algorithms. Churn factor can be calculated by looking at the additions / deletions to the file or repository.

PLOC ensures that developers do not game the system by committing extra unnecessary code. The measure also helps identify highly productive developers who not only write well designed, good quality code but also do so by understanding the requirements, so things get done right the first time.

Measure Code Churn

Lower productivity does not always mean that the developer lacks skills and is therefore inefficient. Comparing how much code was written to how much was moved into production provides some key insights.

The first insight is whether or not the developer was working on features that are important to the business. Suppose the development team wrote a lot of code, but only a small amount made it to production. In such a scenario, it could mean they weren’t working on the right features because someone misunderstood the business priorities or spent a lot of time on prototyping.

Secondly, it is possible that the product owner did not fully define the requirement and kept on changing it, resulting in code churn.

Code churn measures the amount of code that was re-written for a feature to be done right. Code churn can happen because of a) inexperienced developers writing bad code, b) the developer’s poor understanding of the product requirements, or c) the product owner not defining the feature well leading to scope changes, or d) the prioritization of features not done right by the product owner.

Churn is not necessarily bad, especially for early-stage startups that want to validate features through quick iterations. Measuring churn gives you a mirror to see which of these four causes is the underlying problem, so you can work on them to become more efficient. It lets teams know whether they need to focus on prioritization of features, defining features and explaining them better to developers, improving the design, or spending time helping inexperienced developers.

Measure Code Quality

Understanding code quality includes examining the code design and code structure. The design is an indicator of how good the underlying code is. Ensuring the code is well designed and modular makes it easier to test as well as maintain.  

Tools like SonarQube do static code analysis to reveal code design and structure for all the code in a particular module or feature. However, these tools do not tell you the code quality for an individual developer, which is required to improve developer efficiency.

SonarQube identifies a list of issues that need to be looked into and fixed.  It’s important to figure out which ones should be fixed first to optimize developer time, impact and resources. Metrics such as coupling, age, fragmentation and churn can help in identifying the impact a particular file or a component would have on the upcoming releases.

Once the issues are prioritized, it is important to identify the right developer who can fix these issues by looking at who contributed most code to the file/component – Code Owner, her productivity and quality.

Assessing the value of that code (whether it is good or bad) will indicate which developer should have ownership of that module for things like changes or iterations. To assess the value, one needs to look at the contribution, churn, and quality. For example, if a particular developer contributed a lot of code for a feature and the quality is good, you know the developer is good. If the code quality is bad in that same case, it is necessary to make another developer the owner to improve the quality.

The Bottom Line:  Focus On Outcomes, Not Outputs

Focusing on outcomes (code in production, code churn, and code quality) rather than outputs (lines of code) makes it possible to measure software developer productivity objectively. Software development teams that take an outcome-focused approach can accelerate product development, enhance the quality of products and improve the team overall.

In addition to measuring productivity and quality, analyzing code in production can provide a myriad of other useful insights such as understanding a developer’s skill matrix, early signs of developer burnout, top and bottom performers, and quality of product owners. Analyzing a developer’s code quality and commits provides a knowledge base of the skill matrix for a developer. One can detect signs of burnout by looking at the productivity trend of the developer and evaluating whether their productivity has started to drop steadily.

The overall business is also positively impacted by measuring developer outcomes. Managers can reduce delivery risks by estimating features more precisely.  If there is a production issue, this approach helps identify the right developer who can fix it without issues getting regressed. Customer satisfaction improves as product managers are focused on features that are required in production, rather than on proof-of-concepts.

Most importantly, by embracing an outcome-focused approach to development, the cycle time for product delivery can be reduced, which in turn accelerates product time-to-market, increasing revenue and can establish a competitive advantage, improve profitability or help secure financing for the business.

About the Author

Manjusha Madabushi is the Co-Founder and CTO of Talentica Software, the global leader in outsourced product development services for early and growth-stage technology businesses. She has mentored over 70 start-ups in building successful products across domains like FinTech, Media & Ads, EdTech, and Healthcare in a career spanning more than 3 decades. Manjusha is an avid hiker and an IIT Mumbai & Northwestern University alumnus. Follow Manjusha on LinkedIn.



Rate this Article