BT

Presentation: When Working Software Is Not Enough: A Story of Project Failure

| by Abel Avram on Sep 22, 2008. Estimated reading time: 1 minute |

In this presentation filmed during Agile 2008, Mitch Lacey talks about a real life project that was on the verge of being successful, but was deemed as unsuccessful by the customer. Considering that "the true measure of project progress is working software", Mitch and his team delivered the software, but the client was not satisfied.

Watch: When Working Software Is Not Enough: A Story of Project Failure (1h 27min.)

Mitch tells the story of a project which started very well, used Scrum and XP, user stories and delivered working software every 2 weeks. The customer saw the software and asked for changes, which were implemented accordingly. After 8 months in development, the customer said that the team failed the project. The reasons were:

  1. Unclear stories on the product backlog
  2. Not providing project deliverable status
  3. Not providing reasons for stories and functionality being prioritized lower

The team made a retrospective, trying to understand what went wrong. They came up with these conclusions:

  1. Sharing risk in the project. The customer was used to fixed bid contracts, where the vendor takes on all the risk. The team, on the other hand, thought the customer was sharing the risk in the project, and that was wrong.
  2. Providing timely and valuable feedback. Multiple customer user representatives sat in each demo meeting, telling the team that what was being built was correct. It was not until there were six weeks remaining in the project when the customer “woke up” and started actually using the application that had been delivered, resulting in changes to components that had been complete for months.
  3. How decisions for requested functionality impacted other functionality. The customer, used to a fixed bid scope, did not understand that when functionality was de-prioritized that it meant it was at risk of not being complete. The backlog waterline was not understood.
  4. Accountability. The team identified that some customer representatives were not comfortable with accountability. This became obvious when, at the end of the project, several people began blaming the team project manager for work not being accomplished, regardless of what documentation and decisions were available. The project manager for the team was to blame.

The final conclusion was: the team and the customer had different approaches to project development, and different expectations, leading to different evaluations of project progress and success.

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Great presentation - Hits the mark. by Krishnan Ramani

Thanks Mitch,
Wonderful presentation. I had the exact experience in the very first Scrum project I implemented and was fired from my job.
I have been waiting for this kind of insight for alomost 3 years. Thanks again, I will look forward to reading your blogs.

Sounds like... by Dave Rooney

...the story Josh Kerievsky told about an XP transition back in 2003, I believe, that eventually went bad. That's where Industrial XP and its Readiness Assessments came from.



This also sounds like a case where the whole world looks like a nail when all you have is a hammer! The amount of 'complete' up-front work is a warning sign to me, indicating that the organization is still in a serial, phased mind-set. The overall education would need to be applied to much more than just the immediate team. This is also something from Industrial XP - the Project Community - which includes everyone even peripherally affected by the project. Taking that into account may have helped, although it's still easy to overlook some people or groups.



Finally, if the Customer is not an integral part of the team on a daily basis, they aren't a Customer. That may sound harsh and even dogmatic, but they are the most critical aspect of an Agile process. They need to be there and be willing to play by the rules.



It's a good cautionary tale, though, and something all Agile practitioners should watch and read.



Dave Rooney

Mayford Technologies

Risk and Reward are like Love and Marriage... by Douglas Stein

...which go together like a horse and carriage :-). Seriously, if a customer can't be bothered to stay in the loop, then it's clear that Agile isn't right for them.

Also, if a customer's goal is to offload all risk, then they won't get much reward. There are lots of products that are bid, specified to the hilt, designed, coded, tested - and even finish on time and budget. Unfortunately, they land with a resounding "thud" because they don't really solve a problem anyone gives a hoot about.

The comment about not providing "project deliverable status" is a real hoot - because an Agile method gives you measurable value rather than the illusion of progress.

Sorry about the experience, but some customers need to be fired first. Make sure you write your contracts so that you can cancel just as easily as the customer.

You lost me 9 minutes in, unfortunately by Chris Morris

6 months to create a SOW? Cut costs by deferring testing to the Customer? A project with a pre-ordained deadline a year in advance? I guess I had the wrong expectations of this presentation - that it was a success within the definitions of agile but still failed - but I'm getting lots of process smell during the introduction.

where was the EA during all this? by Richard Kucera

No doubt PM is a stakeholder in the EA program...

This is the flip-side of what the EA folks complain about -- PM delivers on-time and under-budget, customer happy -- EA gets no credit, the PM being unaware of EA.

In this case PM fails due to failure of EA program which PM is equally unaware of. PM blames self.

Re: where was the EA during all this? by David Marsh

Sounds like severe process smell to me. Agile means testing is part of development, data migration part of development, customer acceptance part of development, integration part of development, training part of development. I can't believe you are an agile coach, it sounds like you ran a waterfall project, man thats not even iterative.

You traded of things that aren't allowed to be traded off in agile. You have to hold customers accountable, business people are often lazy, you have to force them to perform their role, if you don't have stakeholders to assist in this you're finished.

Documenting failure is not agile, if you're resorting to documents to cover your ass you need waterfall. Agile is normally a new form of time and materials, explain that, its not fixed price / fixed deadline, don't oversell it to the customer. You need key stakeholders and buy in in any process, you need to educate the customer on the process, if they can't understand it, don't do it.

It sounds like you now understand agile, but you burnt the project and the dev team, sounds like you should not have been the agile coach on the project.

Fundamentally Agile has nearly all the same problems as other processes because these are people problems, to fix these you have to fix the way people act, agile trys harder than other processes to educate and assign roles but people are strange beasts. They are not always motivated like you are. Agiles no longer new and basically success rates seem equivalent to iterative or waterfall, good people = good process, bad people = bad process.

Agile needs to mature by Jose Luis Rodriguez

Excelent presentation and very enlightening.



I work in a large SW dev. outsourcing company and these problems are exactly the reasons why we are not adopting agile for medium to large projects. A convincing engagement model for fixed price and fixed calendar projects simply can't be signed with agile since the changing functionality can't be predicted or planned. This hurts both the client and the contractor since both are commited to an invisible and unpreditcable target.



The industry and our clients still need to mature regarding agile. In my view, there will always exist projects in two flavors: Unified Process based (for fixed price/calendar projects) and Agile based (For internal time unbound projects)



I see more future in the UP-like process with PSP/TSP and Lean than the established Agile Processes for fixed price projects.

Re: Agile needs to mature by David Marsh

Agile is like time and materials, its the customers money, and they are buying dev hours, the responsibility is on them to endure those dev hours are well spent.

Agile says if something is wrong you do whatever it takes to fix it, run round the office with your hands in the air, scream at your boss, resign, anything, don't 'collect the data' and send emails in bold marked priority.

Agile says you assume you cannot mitigate much risk, it therefore reccomends minimal upfront planning and analysis, you wasted six months and instead of removing risk you front loaded your project with risk. It says you cannot trade off Quality, you decoped major elements making it impossible to ensure quality. The idea of 'done is done' is related to dev complete, Agile says the customer can screw you around with requirements as much as they like, customer is king, but its their time and money, they have to realise this, thats why their must be an accountable customer onsite 24/7 !

Business types are used to dealing with other business types, any price / estimate you give them they will assume is full of fat, they don't understand engineers and will insist on 50% cuts as a matter of course. They will also create artificial deadlines which must not move, countering all this is in the partnership idea of Agile, remove the uneccessary stuff and just trust each other, unfortunately with multiple companies involved this is politically naive.

Sounds like you drank too much 'koool-aid', nobody in general cares about your openess or honesty, and other Agile mantra. First and formerly you are a project manager, your job is to organise and protect the team, ensure work is delivered on time and budget, ensure relevant stakeholders are onboard.

You failed in two of those objectives within the first few months, you failed to protect the team by increasing risk by descoping crucial elements, you increased risk with wasted time, you failed to get the correct stakeholders onboard and fully involved.

All through this time you yourself seemed to be operating a hybrid process, probably because as an outsource partner you were on a fixed cost contract, this was another massive warning sign.

It was a good presentation and I appreciate the openess and honesty in the presentation, but really theres no excuse for an experienced Project Manager to be caught with his pants round his ankles just because he bought into the latest methodology, you get paid because of your experience and your brain, its your job to pick the right process for the job.

I wish you the best with your book, at least it will give your dev team a break ! Lets hope it doen't cause another generation of managers to blindly follow a methodology and not use their brains.

Odd... by Dick Alstein

Two things immediately struck me as odd.


The first is what was communicated when the budget was only half of what was estimated. The standard reaction should be: "then you're basically going to get half of what you asked". Instead, project tasks were offloaded to the customer (as if they don't cost anything when the customer does it). The customer could keep on thinking he could have his cake and eat it too.


The second is the lack of trust. The openness and intensive communication in Agile projects ought to create a high level of trust between team and customer (or at least, with the customer representatives involved in the project). That apparently didn't happen. Acceptance was given without grounds, and decisions reversed. To me, this points to a pathological culture in the organization.

Re: Odd... by David Marsh

Business schools are full of people with mediocure intelligence, look at George Bush ! Where do you think these people end up ? What do you think happens when they get involved in a project ? Its part of a project managers job to protect the project from such people. Every generation seems to think they are smarter than the last, maybe people wrote all those documents and contracts for a reason ?

Re: where was the EA during all this? by Richard Kucera

You're being too hard on the PM... the failure is EA's fault. EA as currently envisioned is supposed to touch on all the areas that caused the failure. The story he told was that the "architects" came in one day and talked about some policy minutiae having to do with sql or ddl style, bored them to death. Agile envisioned as a PM also doing EA's job without understanding what EA is in the first place is, well how can that work anywhere?

Re: Agile needs to mature by Richard Kucera

> ensure relevant stakeholders are onboard

That's EA's job, not PM. PM should be able to manage in whatever style they want. Stakeholder messaging is something you *get* from EA, and you just have to repeat the message if you run into somebody important on the elevator.

Agile is biting off more than it can chew if it takes on these environmental responsibilities.

Re: Agile needs to mature by David Marsh

Well in this area there is very little concensus, many Agile shops don't pay any lipservice to EA, they prefer project focused or creating self sufficient mini companies with product owners. The role however still exists, so in the absence of a deadicated EA I'd expect PM/Scrumaster/Account Manager/Programme Manager/Product Owner or someone else to ensure theres some joined up thinking. I've seen ineffectual EA sometimes even when it does exist.

The stakeholders are supposed to be agreed at the start of the project, someone should have said something to the effect of "Anyone that isn't in this meeting or hasn't signed this document isn't allowed to impact on the project directly in any way, other than by consulting an agreed stakeholder", someone from IT should have been added to the list at that point or lose their rights to have direct input.

Database standards are often a nightmare in exactly this manner, I've seen many a trival DB schema turned into a one year exercise by over zealous DBA's. Let them re-write it later if they are so fussed, normally the changes are pretty minimal.

Everyone thinks they are a stakeholder when in reality only few people really are, Agiles about doing what matters most and avoiding such naval gazing.

People, Process, Environment and Technology are issues that have to be addressed on every IT project regardless of whether theres a formal EA function or not.

Re: Sounds like... by Mitch Lacey

...the story Josh Kerievsky told about an XP transition back in 2003, I believe, that eventually went bad. That's where Industrial XP and its Readiness Assessments came from.

This also sounds like a case where the whole world looks like a nail when all you have is a hammer! The amount of 'complete' up-front work is a warning sign to me, indicating that the organization is still in a serial, phased mind-set. The overall education would need to be applied to much more than just the immediate team. This is also something from Industrial XP - the Project Community - which includes everyone even peripherally affected by the project. Taking that into account may have helped, although it's still easy to overlook some people or groups.

Finally, if the Customer is not an integral part of the team on a daily basis, they aren't a Customer. That may sound harsh and even dogmatic, but they are the most critical aspect of an Agile process. They need to be there and be willing to play by the rules.


Hi Dave,

Thank you for the comments. This project was full of warning signs. I had just left Microsoft to work at a consultancy and this was my first "real" project. The customer was well aware of their own dysfucntion and it was this acknowledgement that drove me to suggest that we use Scrum and XP (aka agile mix). The problem was that they agreed on paper, but did not understand what they agreed to - and the result was not what they expected (meaning fixed everything), so we "failed". The people we worked with were there almost daily. We spoke with our "customers" daily, but it was the surfacing of the others that messed us up. Lots of lessons and yes, a cautionary tale where I hope others will learn.
Mitch

Re: Risk and Reward are like Love and Marriage... by Mitch Lacey

Seriously, if a customer can't be bothered to stay in the loop, then it's clear that Agile isn't right for them. Also, if a customer's goal is to offload all risk, then they won't get much reward.

Agreed! They were in the loop in the beginning and in the end, just the wrong customers. We all thought they accepted some risk, but they did not.


Unfortunately, they land with a resounding "thud" because they don't really solve a problem anyone gives a hoot about.


Yup, which is what the customer wanted to prevent from happening. They showed us the legacy application and told us of how it was built but didn't solve the problems it set out to solve - it met the specification, not the needs of the users.


The comment about not providing "project deliverable status" is a real hoot - because an Agile method gives you measurable value rather than the illusion of progress.


Yes, this gave us a good laugh too. :)

Thanks for the post
Mitch

Re: where was the EA during all this? by Mitch Lacey

Hi David,


Thank you for taking the time to watch and comment. I am having a hard time what you are trying to say this this post - is it an assertion, a statement or is there a question that you are trying to ask?


Mitch

Re: Odd... by Mitch Lacey

Hi Dick


The first is what was communicated when the budget was only half of what was estimated. The standard reaction should be: "then you're basically going to get half of what you asked".

Absolutely! In fact, when I said that, I was told by our engagement and sales managers to "make it work" - this customer accounted for nearly 15% of our total business and they were in the tens of millions of dollars in projects for us. This project was small compared to some that we had done in the past with good (not great) success. My common sense answer was "cut the budget, cut the work" but that was not going to fly. I event went as far as to suggest that we just turn down the project flat - but we had a pending $6m statement of work with the customer and doing that would have put that at great risk. There were a lot of variables that I did not have time to discuss in the presentation unfortunately, this was one of them.



The second is the lack of trust.


Yup. We had what we thought was high trust in the beginning but that quickly eroded. Surprisingly, I still keep in touch on a personal level with three of the people from the customer side who were very apologetic. They are good people with good hearts.


Thanks for your post!

Re: Agile needs to mature by Mitch Lacey

We had an EA on our side who is very good and delivered a great solution. It was when the customer DBA, EA's and the rest of the 10 people swarmed on us and started to tell us to adhere to undocumented standards on design from 10 year old systems. Having a user control a GUID or a primary key versus a system? You can see where we had our arguments.

Do the customer test or do they not by Pavel Veller

Not exactly the same situation but similar. 11 months in a 13 months project to replace a big old system with the new fresh one (same hot 1.0 vs. old 4.0). At this time the customer team was supposed to facilitate user acceptance testing and we were told they were doing it. Two weeks into the UAT process and I just started questioning number of defects/issues they had reported. 50 issues for a "hot 1.0 vs. old 4.0" was exactly the indicator that real business users were not in fact testing! "how do I do what I used to do in the old system" should have been all over the place and it was not...

We escalated to the sponsor on the customer side and explained all implications. Basically we explained that if this is the way their users "accept" the new system, their business would well just break right after we roll it out... Be careful what you wish for. after the real UAT started to happen we were flooded. over one thousand issues were reported during the next couple weeks. This is another story though.

What I'm trying to say is: it should not be hard to figure if the customer team is in fact doing what they committed to do. The recipe that worked for us - measure it.

Thank You by Phil Chandler

Mitch,

Thank you for your insightful presentation.
I am suffering in exactly the same way.
The client is at the 'escalating the incompetency of IT to a board with no IT representation on it' stage and we're at the 'uncomfortable truths' and reverting to traditional methods stage.
I hate to say it, but in truth, the traditional method highlights the lack of accountability,ownership and specification vaguaries inherently within the process every time and stops it dead in its tracks - whereas agile may not.
Just hoping my team survive in time to prove to a 'traditional' board
where the problem lies using the standard waterfall process.
Agile gives you lots of data to prove what you've been doing, but it's subjective enough to leave you exposed by a threatened senior user base.

Wish I'd seen your presentation 6 months ago, but at least I know we're not alone and I've stopped doubting my competency and knowledge.

Thank you.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

20 Discuss
BT