Discussion: Measuring Success of an Agile Project from the Customer’s Perspective

Posted by Vikas Hazrati on Feb 05, 2008

Sections

Process & Practices,

Architecture & Design

Topics

Agile ,

Delivering Value ,

Customers & Requirements

Tags

Value & Metrics ,

Planning

Recently there was an interesting discussion on the Scrum Development user group trying to answer “How does a customer measure the success of an Agile project?” The important piece here is “measure”.The discussion seemed to agree that measurement of success from a client’s perspective is important, and there are various ways to achieve it. The best way to measure would depend on the situation and the customer.

Traditional development methodologies had certain metrics: quality metrics such as defects per release; schedule metrics, such as actual effort/days versus estimates and delivery milestone achievements; and cost metrics, such as budgeted versus actual costs.

However, most Agile development organizations are notoriously known for being weak on collecting metrics and reporting them back for the project. While they do have technical development metrics like checkstyle errors, test code coverage, NCSS, etc, these metrics are of little significance to the customer.

Measurement of success from a customer’s perspective becomes important because, though the team might be following the right process, the end result might not be what the customer is looking for. In fact, it might be very different.

Mike Dwyer said

You are asking me the same question that a surgeon asks. Am I doing a good job if I follow all the steps, and if the patient gets well or if they die is not my concern? My response is what is the goal here? A well patient or a successful procedure?

In this case, if the end result is what we logically expect, i.e. the patient gets well, then we would call it a successful project with a successful process.

Vikrama Dhiman provided another example:

My project could really be to make this 1000 story building which I hope to sell to United Nations - the team organizes itself beautifully and they help me realize my vision in time - but United Nations does not buy the same and I don't want to sell to anyone else. Is the [process followed for the] project a success? I would say yes. Is the end product successful? I would say no. Who is at fault here? Product Owner, most certainly.

Here the team might have got all the process right, however they probably built the wrong product. Was the product owner to blame? Yes. Was the team to be blamed? Yes. Is the customer satisfied? Definitely not.

So, is there a way in which a customer can measure the success of an Agile project? The answer does not lie in showcasing a great process as is evident from the above user statements.

A few people suggested that the customer would consider the Agile project a success if he is satisfied. Now, “satisfaction” could be hard to quantify and measure. A similar discussion on Agile India user group suggested a few subjective metrics for measuring customer satisfaction. The customer would be satisfied if:

• He got the business value that he envisioned from the project.

• He was able to change course according to business demands and get functionality delivered which makes more sense today.

• He was able to give early feedback

• He was able to prioritize the development of features/stories, thus getting the right functionality at the right time

• He was able to stop development before all the functionality was implemented, because he realized that the remaining functionality might never be used.

• He developed a successful two-way trust relationship with the development team

• He got no surprises during the project.

• He got value for money

A few others suggested that, before the team starts the project, a definition of “done” is important. The customer and the team should be able to come to an agreement on when a story is “done”. Once the team has decided on what “done” is, then customer satisfaction and the rest of the metrics flow from that.

Mike Dwyer added:

Definition of DONE simply states that when the customer accepts work from the team at the end of a sprint then that work is DONE. At this point all the effort and associated costs move from waste to value. This answers the questions, “how far along are we, against plan” (product backlog – cumulative accepted work) and how are we doing, as to budget (total planned budget – actual costs). It also answers questions like: how efficient are we (total actual costs / planned costs for DONE tasks) and what is our burn rate (average cost per accepted or DONE element).

Scott Ambler suggests a combination of objective and subjective metrics. He suggests that teams start by collecting some basic data, such as:

1. Actual cost (hours, $, ...)
2. Some sort of functionality delivered (use case points, user story points, ...)
3. Defect trends
4. Stakeholder satisfaction

In this way, the team can communicate to the business in the terms it uses (basically, money and systems capability) and still continue to track metrics that are core to development success, such as quality. A measure of stakeholder satisfaction is one of the most important metrics that a team (Agile or not) should introduce. Delivering defect-free code doesn't necessarily mean delivering valuable code.

There seem to be both subjective criteria, such as customer satisfaction, customer viewpoint and objective criteria, such as stories done, cost and defect trends, which would help a customer gauge the success of an Agile project.

The discussion seemed to agree that just following the right process is not important. In addition to the process there need to be both objective and subjective checks which would ensure that the project is going to be successful and that the customer would be satisfied. A development team could choose one of them or the combination which makes the most sense in a given scenario.

1. Back to top

   Provided metrics should be 'well-known'

   by Cesario Ramos

   Nice post.
   
   When introducing Agile at customers providing the ‘correct’ metrics is very important. In order to keep the project stakeholders satisfied and increase their confidence in the new approach you have to provide them with the right feedback. You must be able to show the advantage this new approach gives them compared to other or past projects in their organization.
   I think that the metrics to use for this should be based on the metrics already used in the organization so that a useful comparison is possible. I think assessing the already used metrics and incorporating and/or extending them is the way to go.

   Reply

2. Back to top

   Re: Provided metrics should be 'well-known'

   by Amr Elssamadisy

   But usually past metrics don't have anything to do with customer value but internal measures. Depending on your metrics, for example, you may have 'productivity' as kloc/developer - which may even go down as we make solutions more simple. Or, it may be an internal measure of done-ness when nothing has really been completed for sure, and again, an agile process that builds things from end-to-end, and takes integration and deployment costs early, will seem to have inferior performance.

   Reply

3. Back to top

   Re: Provided metrics should be 'well-known'

   by Henrik Mårtensson

   I am sorry, but for most organizations that will not work very well. They just do not have metrics that will provide the right answers.
   
   Many companies spend a lot of effort on metrics. For example, they spend a lot of money developing balanced scorecards. After awhile, they find they have no idea whether a metric really is connected to the goal of the organization. (There is a revision of the scorecard method that attempts to fix this, but it is not widely used.)
   
   The best approach I know is to use a systems approach, i.e. figure out the goal of the organization first, then figure out Critical Success Factors, and Necessary Conditions for acheving the goal. After that, it is usually easy to work out how to measure whether an activity brings the company closer to achieving the goal or not.
   
   I have just published an

   Reply

4. Back to top

   Re: Provided metrics should be 'well-known'

   by Henrik Mårtensson

   Of course, I managed to botch the link:
   kallokain.blogspot.com/2008/03/business-value.html

   Reply

• Older >