BT

IT Projects: 400% Over-Budget and only 25% of Benefits Realized

by Vikas Hazrati on Oct 31, 2011 |

An alarming study by Flyvbjerg and Budzier published in the Harvard Business Review has made everyone stand-up and take notice. The coherent advice being that IT projects are much more riskier than we think.

The study showed that amongst the 1400+ projects surveyed, on an average 27% were over budget. 16% of the projects could easily be categorized as Black Swans. They had a cost overrun of over 200% and were late by almost 70%. These were the projects which could cause stunning failures and bring down companies.

Boris Gloger mentioned that the results were frightening.

These results are significant. But even more frightening are the conclusions of the authors of Harvard Business Review article. They write: If you want to avoid the slow death caused by IT projects you must be prepared not only to spend 400% more than planned on the project, but to reap only 25% of the expected benefits. If you keep this in mind you can possibly prevent a company from being killed by an IT project.

Paul Glen mentioned that one would expect the distribution of project overruns to look like a bell curve, but surprisingly that is not the case. A disproportionate number of projects were huge failures which had the potential to destroy complete organizations.

Ed added that it was amazing to notice that in the last 15 years the industry had not got any better at delivering large IT projects. One of the possible reasons for the results to look the way they look could be that the study pointed to a lot of projects where re-platforming was involved.

My theory is that it is these very large re-platforming projects that are frequently the issue. They touch so many systems, so many discrete business units involved across the globe, and the desire to do customization so they become impossible to upgrade.

Ed mentioned a similar re-platforming project which was successful as it had all the ingredients of a good Agile project.

An example from the article of a successful project by the Emirates bank in 2006, shows a successful way to run a large complex project. The lessons learned from this successful project were to:
1. Stick to schedule
2. Resist changes to the projects scope
3. Break the project into discrete modules
4. Assemble the right team
5. Prevent turnover among team members
6. Frame the initiative as a business endeavor, not a technical one
7. Focus on a single target – readiness to go live, measuring every activity against it.

Boris added that another reason for such disappointing numbers could be that we are addressing wrong people.

But maybe we are addressing the wrong people , like Ken Schwaber says in his new, soon to be published book. Possibly IT customers, software development, and product development need to simply not accept any more that IT departments, QA departments and process people can not create finished, tangible and measurable results every two weeks.

Mary-Ann Massad mentioned that a good governance model could have saved the day for many organizations. According to Mary-Ann,

Real governance means leadership. Real governance means courage-the courage to tell the truth even when it will not be well-received. Too many "well"-run organizations operate under a culture of fear and "group-think". This is why IT projects or any projects for that matter, have the ability to bring an organization to its knees. We have to work on building organizational cultures that welcome honesty, diverse opinions, and integrative thinking.

Thus, one could dive deeper to analyze the shortcomings and learn from them, however the numbers do suggest a serious re-think in the way projects are executed.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Willfull Ingorance by James Watson

I personally believe that the root of the problem is called out in the article but in a subtle way (probably in order to avoid offending their readers). The issue is that in most non-software companies, information technology is treated like something separate from the business. The conventional wisdom of a decade ago dominates. That is, if your company is not producing software for sale, it shouldn't be building software. It should be purchased, like Word or Excel.

This line of reasoning seems pretty bullet-proof until you dig into the details and it leads to a number of problems including the issues in the article.

First, lets be clear, no company should seek to build software for internal use that can be purchased at reasonable cost and can can meet their needs without customization. Secondly, companies should seek to purchase or other wise acquire software that implements the generic functions and allows for powerful customizations.

The issue arises when companies purchase something like SAP assuming that their needs are sufficiently supported by the software and that anything they need that is special will be supported by 'configuration'. What then happens is that the company realizes (after investing large amounts of money) that they can either make their business completely generic and lose all strategic advantage over competitors or they can pile on a lot more money to customize the hell out of it. Given the options, most high-level execs prefer more investment over losing strategic advantage. Some companies (I suppose) choose the other option and a lot of IT people think this is a good idea (it isn't.) You might be more likely to succeed at the project but the company will often be ruined in the process.

The error really is taking on a project based on false assumptions. Once that mistake is made, there are no good options. Companies need to stop thinking that information technology is separate from their core functions. Most information technology is only worth having if it is tightly coupled to the design of the business and often technology imposes fundamental constraints on business.

No project management methodologies or helpful tips will address this problem. Companies need to embrace technology and make IT a stakeholder at the highest levels of the organization for the kinds of problems described in the article to be fully addressed.

Black Swans? by Marc Stock

I take issue with this article for two reasons:

1) Seriously, who does large IT projects anymore? Did they not get the memo that they are, at best, doomed to be way over budget and behind schedule? At worst, they are an abject and complete failure. Hell, the article starts off with a story of Levi Strauss bringing in Deloitte consultants. Even without knowing what they were working on, I could have told you it would fail right there.

2) Black Swans are unpredictable. What's happening on these projects could be seen from a mile away. I guarantee you that any competent developer with 10+ years experience under their belt could have told you these would fail long before they did (and often, before they even started).

The take away shouldn't be that large projects are risky. It should be that you just don't do them in the first place. They have so many problems with them that aren't even technical that it's pointless to even begin them. I've seen companies that always execute small projects well try a big project and have it be a huge failure.

Non-sequitur by Dean Schulze

I don't see how the 7 bullet points listed are the hallmarks of agile any more than they are the hallmarks of heavyweight processes.

If 92% of these projects are government projects and many of them are replatforming projects then this study doesn't have much to say about software development, which is agile's domain.

Re: Black Swans? by Jason Yip

I agree about the Black Swan thing. It would be interesting to get a description of all the projects without providing the final result and get people to predict what they think would happen.

Re: Non-sequitur by Vikas Hazrati

Dean, you are right that the sample list might seem skewed but this is what they follow it up with (from the article), which makes it relevant for all of us

Our sample drew heavily on public agencies (92%) and U.S.-based projects (83%), but we found little difference between them and projects at the government agencies, private companies, and European organizations that made up the rest of our sample.

Re: Black Swans? by Vikas Hazrati

True Black Swans are unpredictable however in the hindsight, they do not seem improbable. Like they described the Nike project ..
A $5 million project that leads to an almost $200 million loss is a classic “black swan.” The term was coined by our colleague Nassim Nicholas Taleb to describe high-impact events that are rare and unpredictable but in retrospect seem not so improbable.

Project failure or planning failure? by andrej koelewijn

Why is a project failure if it doesn't live up to the initial plan? It's more likely that the initial plans were wrong. Why do we assume we can predict such complex projects with so many unknowns? Please pass me the crystal ball.

Re: Black Swans? by Amit Kumar

Correction - The article mentions Levis and not Nike.

Re: Black Swans? by James Watson

True Black Swans are unpredictable however in the hindsight, they do not seem improbable. Like they described the Nike project ..


I think that Marc's point is that a lot of people could (and probably did) predict these issues long before they occurred. If something occurs as frequently as large project failures do, calling it a 'black swan' doesn't seem inappropriate.

Aside from my comments above, the dynamic of such projects is that middle management cannot openly express doubts for fear of demoralizing the team. It's not that the failures are improbable based on history (quite the opposite) but that the people involved are simply unaware of the risks.

Interesting article by Dave Nicolette

Very interesting stuff. Seems large-scale IT is in even worse shape than indicated by the past studies people have cited over the years. Unclear why the item is categorized under the heading, "agile," though. I see no connection.

Skeptical of this Article by Joe Fecarotta

I got the hbr article, read it and did some google queries. It turns out that Flyvbjerg is a consultant that supports the "reference class forecasting" position of his firm. His firm specializes in a lot of Civil and Construction projects. Nothing wrong with that of course, but reference class forecasting( finding projects like yours and then basing your budgeting on it) is a challenge. Since every company is different, and since software is so behavioral, I doubt that this forecasting method would do much for us. Remember, this is a complex system, and therefore response better to adaptive than control based process models (google Stacey Diagram: Zone of complexity). The whole concept of the Black Swan that they throw around in the article is about the impossibility of prediction.

The background of the fellows in this article explains why the 7 key steps don't resonant completely with me and others on this list. I've been on "stick to the schedule - limit scope" projects, and they were called waterfall. They release something ontime that no one wants that people who use it haven't seen, and who are usually less productive afterwards. In the article he talks about how Emirates went "Big Bang" and how great that was. Call me skeptical. If Scrum is right, then we use adaptive processes for those things, and we learn and adapt towards deployment.

However, the last 5 are very good, especially #7, focusing on a single target. Focus is what is missing from my enterprises that have had tricky large scale projects.

Re: Interesting article by Vikas Hazrati

Could Agile help?

Re: Skeptical of this Article by Vikas Hazrati

There is clarification on # 2 on the blog

Number two may strike us as an Agile faux pas, not allowing change in scope, but I believe the authors are referring to “Mission Creep” rather than changing featues in a requirements spec. Drifting missions or just unclear goals strikes me as a hallmark of any unsuccessful endeavor.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

13 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT