BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Critical Metrics to Keep Delivering Software Effectively in the "New Normal" World

Critical Metrics to Keep Delivering Software Effectively in the "New Normal" World

Key Takeaways

  • It is vital that technology leadership understand the health of their delivery capability in the ‘new normal’ world of remote working, uncertainty and cost pressure  
  • This requires the ability to track a set of critical metrics.  For organisations delivering software in an Agile way, a sensible place to start is a hierarchy of metrics that tie back to the core Agile principle of “the early and continuous delivery of valuable software”.
  • Our five overall delivery health metrics for the ‘new normal’ world which are meaningful when tracked over time at an aggregate level and give your whole organisation a simple set of metrics around which to align are: Time to Value; Deployment Frequency; Throughput; Defect Density; Team Engagement 
  • Our top five cascaded delivery metrics for managers and teams, which drive the five over-riding metrics at the top of the metrics hierarchy are: Deployment Frequency; Flow Efficiency; Cycle Time and Lead Time; Completion Rate; Engineer Morale Score
  • We hope that many organisations will take the metrics suggested as a good place to start.  However, you may prefer to build your own bespoke metrics.  Whichever metrics you choose, In our view it is the discipline of tracking and managing to metrics (that reflect core Agile principles) that is critical in the ‘new normal’ world.

The world has changed dramatically and a "new normal" has appeared almost overnight - a time of remote working, great uncertainty, changing priorities and dramatic cost pressures.  

Software delivery teams sit at the heart of this challenging new environment as organisations look to them to deliver more, for less in strategically critical areas. 

Metrics, visibility and risk management were already an increasing priority in Agile software delivery – particularly in large scale organisations. But recent events have seen these catapulted from important to essential, as the ‘new normal’ world presents a whole new set of challenges.

This article describes how one company (which itself is involved in data driven software delivery) - is practicing what it preaches, by not losing sight of a few key metrics during this period of intense change and uncertainty.

Using metrics to understand the health of your delivery capability in the 'new normal' world

The ‘new normal’ world creates three sets of immediate challenges for delivery teams:

  • it requires an immediate change to well-established ways of working - most notably due to forced remote working – so that daily routines and face-to-face communication are changed in both obvious and more subtle ways;
  • demands on delivery teams are remain the same, and are in many of our client’s cases increasing – with more to do, changing priorities and less available resources of all types; and 
  • the very people upon which success depends will never have felt more unsettled and unsure about the future.

It is vital that technology leadership fully understand the health of their delivery capability.  How are teams really feeling?   How is the delivery process being affected by remote working and rapidly changing priorities?  And is Time to Value slowing dramatically and if so how do we mitigate?

And this, in turn, means better visibility of processes and teams and the ability to track a set of critical metrics that objectively answer questions such as these.

For organisations delivering software in an Agile way, a sensible place to start is a set of metrics that tie back to core Agile principles – so that everyone is focused on the ultimate Agile goal of increasing customer satisfaction through “the early and continuous delivery of valuable software” – despite the challenges thrown up by the ‘new normal’ world.

As Reuben Sutton, Plandek’s VP Engineering notes, "We have had to move to a fully remote working environment overnight, during one of the most intense software delivery periods our company has ever known.  The Agile delivery metrics that our teams track and understand have been our ‘North star’.  We know that we are still going in the right direction, as we can see it objectively in the metrics."

Creating a hierarchy of simple metrics that everyone understands

If Agile principles are the ‘north star’ around which you set your goals in the ‘new normal’ world, then you will need an effective framework for adopting them. 

In our experience, this framework needs to provide a simple hierarchy of metrics, so that they are understood and adopted by everyone.  This is especially important when team members are working remotely and are under increased pressure to deliver.  

As shown conceptually below, the metrics are self-reinforcing, with team-level metrics rolling-up to your overall delivery health metrics – which in turn ensure that you are still heading to the ‘north star’ of effective Agile delivery.  

Figure 1 – the principle of a metrics hierarchy supporting the core objective of Agile software delivery

Critically also, the key is to adopt leading metrics that are deterministic of improving the process (and predicting likely outcomes) – rather than lagging metrics that simply “look in the rear-view mirror”.  So, what is a sensible set of metrics to adopt?

The top of the metrics hierarchy - five overall delivery health metrics for the ‘new normal’ world

We have chosen five simple metrics that capture the overall health of your delivery capability as it navigates through the tricky road ahead.  These metrics have three characteristics:

  • they are well known “Agile metrics” that ensure you are still meeting your core Agile commitment of delivering value as quickly and efficiently as possible;
  • they are meaningful when aggregated; and
  • they are simple.

The metrics that we have selected are shown below in Figure 2.

They need to be sponsored and viewed at aggregate level by technology leadership.  And they are supported by a set of sub-metrics that are cascaded down the organisation to the team level, so that teams work to continuously improve them and hence deliver to the technology leadership’s overall delivery goals.  

Figure 2. The five core metrics that underpin delivery health in the ‘new world’

Together these five metrics summarise your “Agile health” – your ability to deliver software effectively, despite the constraints.  They are meaningful when tracked over time at an aggregate level – and give your whole organisation a simple set of metrics around which to align.

  1. Time to Value – the core Agile metric tracking how “early” you are delivering value for customers.  Measured from the beginning of the development process through to deployment.  
  2. Deployment Frequency – key metric of how “continuously” you are delivering 
  3. Throughput – how much value you are delivering.  There are many ways Agile teams will measure this, but Story Points, Value Points or Tickets Completed are a common place to start
  4. Defect Density – a key measure of your ability to continue to deliver high quality software, commonly measured by the ratio of stories delivered to escaped defects (or production defects)
  5. Team Engagement – our favourite metric in the ‘new normal’ world of remote working.  Best collected via polling on collaboration hubs like Slack and in our view the key leading indicator of delivery health, as software delivery is absolutely dependent on your team struggling with the pressures of the current environment. 

As Reuben Sutton comments: “These are the metrics that I have held close as the level of uncertainty has increased in recent weeks.  We all understand them, it’s pointless to try and gamify them – and assuming they remain in relatively good shape, I can stay confident that we will continue to deliver effectively going forward”.

Cascaded metrics - our top 5 delivery metrics for managers and teams in the ‘new normal’ world

The power of metrics is realised if they are vocally sponsored by leadership and are then cascaded across the organisation to the key functions, programs and Agile teams (squads) responsible for software delivery.

As shown in Figure 3, our preferred cascade of metrics fall into easily understood areas, all of which drive the five over-riding Agile metrics shown in Figure 2.

The metrics fall into seven logical groups, which are measurable (and actionable) at program and team level and which drive delivery team health and effectiveness.  These are:

  1. DevOps process effectiveness metrics – includes metrics such as Deployment Frequency and Lead Time for Changes and remain key measures of the team’s ability to deploy effectively despite the new pressures 
  2. Backlog analysis – backlog size and health metrics are always an important leading indicator of programme health and are even more salient at a time of remote working and constantly changing priorities 
  3. Development process efficiency – Flow Efficiency is a little measured but critical metric, especially at a time like this. It looks at the proportion of time that a team is actively working on a ticket (i.e. active vs inactive states of delivery). Ensuring that collaboration remains high and members are effectively working together during this period is a critical measure of your delivery health. 
  4. Throughput and Time to Value – the core Agile metrics of Velocity, Cycle Time and Lead Time have never been more important as is the measure of throughput 
  5. Delivery Accuracy - evidence shows that if teams (working in a Scrum Agile environment) are unable to deliver their Sprint goals accurately, then the predictability of output over longer time periods and broader team sizes, is seriously compromised.  Predictable delivery will be a highly valued capability in the ‘new normal’ world, hence the leading indicator of Completion Rate is so important. Completion Rate shows the percent of planned story points completed over a time period
  6. Quality – this is always a ‘north star’ Agile metric and can be measured in many different ways. We like Defect Density as an expression of defects created per unit of code, but also feel strongly that First Time Pass Rate is an important measure at the individual engineer level (but only when viewed in context within the team)
  7. Talent - we can’t emphasise enough how important we believe it is to really understand how your team is feeling in the ‘new normal’ world.   We think you should track a measure of morale and also a measure of how well people view their team to be functioning in the new remote working environment.  

Figure 3 – suggested programme and team level metrics in the ‘new normal’ software delivery world

There are a lot of metrics here to consider (and too many to realistically collect without an Agile metric BI platform).  So, if we had to choose five simple metrics to start with in the ‘new normal’ world, the top five that we would choose are:

 

Deployment Frequency

Critical DevOps metrics such as Deployment Frequency which confirm our ability to maintain the continuity of our delivery 

Flow Efficiency

Development efficiency metrics such as Flow Efficiency (the % of time tickets spend in an active versus inactive status) which are likely to be negatively impacted as teams move to unfamiliar remote working

Cycle Time and Lead Time

Cycle Time and Lead Time - the critical measures of time to value, which are also likely to suffer in ties of change and stress

Completion Rate

Completion rate which measures our ability to deliver our sprint goals over time.   This is highly likely to be negatively impacted in the new normal world and will then adversely affect the predictability of delivery timing

Engineer Morale Score

Engineer morale score – (measured in Plandek via polling using collaboration tools like Slack).  This is a critical measure in the new normal world as people adjust to the often unfamiliar set of circumstances that they find themselves in.

As Alice Dunn, Senior Developer at Plandek notes: “It’s been pretty crazy and the move to home working took some getting used to.  So at team level we have really focused on our Cycle Time and our Completion Rate.  We know that if we can keep hitting our sprint goals consistently and our Cycle Time remains tight, we won’t miss a beat in our delivery...”

Choosing metrics that reflect your particular objectives 

We hope that many organisations will take the metrics suggested in this article as a good place to start.  And we would agree, as they would ensure that you continue to deliver against the most central Agile principles at a time of rapid change and stress.   

However, you may prefer to build your own bespoke metrics set that more closely mirror your own specific objectives.  

With this in mind, there are a variety of commonly used metrics setting approaches - such as OKRs (Objectives and Key Results) or GQMs (Goal, Metric, Question) as popularised by Victor Basili which can be applied to define a bespoke metric set.  

In our view, whichever route you take is very much up to you, but it is the discipline of tracking and managing to metrics (that reflect core Agile principles) that is critical in the ‘new normal’ world - so that when the chips are down, everyone across your organisation is focused on the things that really matter.

As Reuben Sutton, Plandek’s VP Engineering concludes “Aligning around shared goals and metrics is so critical in difficult times.  We have a simple set of Agile metrics that we track very closely and so far our delivery teams seem able to more than meet the challenges being thrown at us.”

About the Authors

Charlie Ponsonby started his career as an economist in the developing world, before moving to Andersen Consulting. He was Marketing Director at Sky until 2007, before leaving to found and run Simplifydigital in 2007. Simplifydigital was three times in the UK Sunday Times Tech Track 100 and grew to become the UK’s largest broadband comparison service. It was acquired by Dixons Carphone plc in April 2016. Ponsonby co-founded Plandek with Dan Lee in October 2017. Plandek as an end-to-end delivery metrics analytics and BI platform, to help technology teams better manage delivery risk and improve their delivery effectiveness.

 

Rate this Article

Adoption
Style

BT