What Color is your Backlog?

Posted by Shane Hastie on May 02, 2010

Sections

Process & Practices,

Architecture & Design

Topics

Performance & Scalability ,

Agile ,

Architecture ,

Design

Tags

Quality

 At the recent SDC conference in Sydney & Wellington Prof Philippe Kruchten delivered a talk titled “What Color is Your Backlog”.  The thrust of his talk is about bringing a focus on architecturally significant aspects of software into Agile projects, along with delivering the functional components of the system.

Kruchten maintains that the Agile focus on YAGNI (You Ain’t Gonna Need It), refactoring and leaving decisions until the “last responsible moment” brings a risk of delaying important decisions for too long.  On any project there are a set of key decisions that need to be made early that will set the scene for ongoing work – for these decisions the “last” responsible moment is in fact very close to the beginning of the project. 

These key decisions are the ones that drive the architectural structure of the system being developed.  Making poor choices (or not making conscious choices at all) can result in a system that can’t adapt to the evolving business needs. They are choices about the architectural structure of the system that need to be considered early in the development process. 

He states that many Agile projects focus on the functional needs (expressed in user stories) and neglect the architectural aspects with a irresponsible “we can refactor for that later” attitude, all too frequently leaving the “later” until far too late. 

He gives an example of a financial trading system that used Agile methods. 

 

User stories were identified, releases planned and development started.  The first few iterations were resounding successes – features were delivered, customer engagement was great and the functionality was exactly what the business asked for.  Once there were sufficient features to constitute a “minimum viable product” a release was put into the production environment and the project hit the wall – the features which had been built worked, but the product couldn’t cope with the volume of transactions that were needed in the live environment.  The customers had to revert to the old system (which had not been turned off) and the development team went back to refactor the product to cope with the live environment demands – unfortunately refactoring meant that most of the work that had been done had to be completely rebuilt, the only components that could be salvaged were the screen designs.   The project was eventually cancelled, and the customer organisation was very reluctant to try Agile again. 

 

 He is not recommending reverting to big design up front, but encourages giving careful consideration to the key quality aspects of the system, and ensuring that architectural features get considered along with the functional stories. Architectural features are driven by the “non-functional” or quality requirements of the product – aspects such as performance, reliability, capacity, maintainability, security, usability and testability.  These are the aspects that have an architectural impact on the product – they must be designed for from the beginning rather than refactored in afterwards.

He acknowledges the vital need to show progress – we don’t want the development team spending the first two or three iterations beavering away on “plumbing stuff” that has no visible value to the users.  He suggests that it is possible to weave architectural requirements into the overall release plan of the product, ensuring that as the product is built both the architectural underpinnings and functional features are built and we can show progress both in terms of increased functionality and the ability to meet the quality goals.  So, for example, the first iteration may only result in a single entry screen but load testing will show that that component of the system will cope with the likely expected demand in the production environment.

He offers a technique for making work visible and ensuring that architectural needs get addressed and prioritised along with visible feature-based work -  applying the concept of color coding to the items in the product backlog.

 

He states that there are likely to be four colors to the work in the backlog:

• Green – Features to be delivered, the functional user stories
• Yellow – Architectural infrastructure that support the quality requirements
• Red – Defects that are identified and need to be addressed
• Black – Technical debt that builds up as the product is built and key decisions are deferred or poor work done

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure: Colors in the Backlog

Note the two dimensions – positive vs negative value and visible vs invisible features.

The green components are the Visible Features most commonly found on product backlog lists – the User Stories that define the functionality to be delivered in the product.  These will be visible to the users of the final product.

The yellow pieces of work are invisible to the users but provide significant value to the overall product.  These include work done for:

• Architecture
• Infrastructure
• Common elements
• Libraries
• Frameworks
• Making components reusable

Yellow (valuable but invisible work) needs to be made explicit in the product backlog and release plan, so project stakeholders know to prioritise it along with visible features.

The red parts (Visible Defects) are inserted as the product is developed – in an ideal world there will be very few red components but it is naive to assume there will be none.

The black (Technical Debt) is what happens when the yellow is ignored; technical debt may also have accumulated from previous work where the project is working on a pre-existing code base.

When building an initial release plan it is important to plan for both green and yellow components – a single-minded focus on visible features could result in a product that looks wonderful, but doesn’t scale to the production environment, or doesn’t cope with real-world security threats.  Likewise purely focusing on the yellow leads us back to BDUF (big up-front design) and delays delivery of any visible value until so late in the project that the customer engagement is lost. 

 

He suggest building a release plan that addresses both “like a zipper” – weaving the functional and architectural components together, as in this image:

 

 

 

 

 

 

 

 

 

 

Figure: Plan your releases to merge visible and invisible value components like a zipper 

The red and black (Visible Defects and Technical Debt) will not appear on the initial release plan, but they do have a significant impact on the velocity (rate of work done and value delivered) available on the project. 

Defects need to be addressed as soon as they are identified as is practical and feasible; left unresolved they fester and result in significant impact when they are addressed later.  After a few iterations the team will have an understanding of their likely rate of defect insertion, and can adjust the planned velocity to ensure defects are removed as close to the point of insertion as possible.

If the project starts with known Technical Debt then this must be taken into account when planning the work to be done – an older code-base is likely to have hidden defects already there, and the code may not be easily refectored so again the rate of value delivery will be reduced by the quality of the underlying product.

When building a new product there is not initial technical debt, but there is a risk of debt being incurred.  Deferring defect fixes, ignoring the yellow components, deferring key decisions until beyond the “last responsible moment”  all result in an accumulation of Technical Debt and a reduction in the rate of value delivery.

Where defects and technical debt are identified in the course of the project, it is important to make them visible to the project team – hence the concept of color coding the backlog.  New stories need to be added to the product backlog to show the defects and work to be done to pay down technical debt.

He strongly recommends color coding the master story list to clearly show the type of work to be done – is the story a new user feature, an architectural component, a defect that must be fixed or some technical debt that needs to be paid?

By highlighting the colors in your backlog, and by making all four aspects of the product development clearly visible more realistic planning and tracking of the project will be achievable.

---

 

What are the colors in your backlog – what techniques do you use to make all work visible?

 

 

 

 

 

Shane Hastie is an agile coach, trainer and consultant working for Software Education in Australia & New Zealand

1. Back to top

   Invisible? In Agile? Huh?

   by Morgan Creighton

   So, "black" represents "debt," contradicting the most common meaning of the color. What a very unfortunate choice.
   
   But more to the point, how can anything be "invisible" on an agile project? Agile is all about surfacing issues transparently and early. Suggesting that it's about sloughing off difficult decisions is a mischaracterization.
   
   If the project of which Mr. Krutchen speaks were really following agile, then it failed because the product owner deliberately chose not to make scalability an important story. Handling the necessary throughput was not a priority, and the team merely spent the customer's money as they were asked.
   
   This does not necessarily mean that the product owner was inept, although that is the simplest explanation.
   
   However, just as it takes two bad drivers to make a traffic accident, the development team can be faulted too. They probably failed to impress upon the product owner that performance and other "ilities" can't be bolted on at the end. But it's exactly the short iteration / tight communication loops fostered by agile that should have helped them hammer this point.
   
   More agility, not less, might have saved the project.

   Reply

2. Back to top

   Agiles fault?

   by Antonio Gurisatti

   I completelly agree with Mr Morgan- Leaving out nonfunctional stories has nothing to do with the method used to build the project. And the whole team is to be blamed. The fact that writing user stories is the product owner responsability, in any way means that the rest of the team stays with they brain idle and behave like a recording machine. The must read and analyze the user stories asking and helping the product owner in getting a sound set of user stories.
   
   I think that using colors is a good idea. I use a "Home made" Agile Development software that uses icons for that purpose and colors to show the status of the user stories (Registered, in release backlog, in sprint backlog, done, deferred to future versions dropped, etc).

   Reply

3. Back to top

   Budget by color

   by D M

   I like the idea of these categorisations for some projects. I think it's quite common for the 'product owner' to be a proxy for multiple stakeholders, many non-technical, but also accountable to them (e.g. the CIO may be responsible to COO, CFO, Head of Sales/Marketing, etc.). If hidden-architectural features and technical debt are always getting pushed down the backlog, e.g. because each department 'must have' their latest feature, then budgets for sprint item types can be used. This could be stated like, 'we need a minimum 30% each sprint allocated to architectural/technical debt'. It gets things out in the clear and if this causes delay problems on visible features and defects, then at least capacity constraints become visible.

   Reply

4. Back to top

   Re: Agiles fault?

   by Elderclei Reami

   In my opinion, really mature agile teams do more than the Product Owner asks. Keeping it simple, doesn´t mean you must only do what you are asked for. It means you are doing no more than necessary.
   
   Mature agile teams knows that quality requirements (scalability, maintainability, testability, usability, and so on) must be DONE by the end of the iteration to some agreed upon level. Doing this requires brain, previous experience and some learning through short iterations and feedback process.

   Reply

5. Back to top

   Delayed decisions

   by Miguel Ping

   If you design your components in a way that you cannot easily refactor it or making it scale, you already made some decision up front. You made the decision that "scale doesn't matter, so we go with whatever works".
   
   I interpret the "delay decisions to the end" motto as a way of designing your system with no compromises, this is, you can always change some components easily. This is a good engineering practice and good architecture.
   
   This colour concepts makes it clearer for person to SEE what's going on with the project from a developer's perspective.

   Reply

• Older >