Standish: Why were Project Failures Up and Cost Overruns Down in 1998?

| Posted by Jim Johnson Follow 0 Followers on Oct 03, 2006. Estimated reading time: 4 minutes |

The CHAOS statistics on project failure are frequently cited by those implementing Agile or other forms of process change, so their validity is of importance to the many people who continue to use them.  In August, triggered by Robert Glass's questions in August's Communications of the ACM , InfoQ interviewed Jim Johnson, creator of the CHAOS Chronicles on project failure, to hear how their data is collected.

In the discussion thread that followed the interview, one question surfaced as still unresolved: how does the Standish Group explain the amazing improvement in cost overrun from 189% in 1994 to 69% in 1998? The implication of this question, also posed in several other places on the web, is that Standish may have adjusted their research methods without revealing the change in their reports.  If true, comparisons of some CHAOS data from 1994 and 1998 could be meaningless.

Standish founder Jim Johnson was also concerned by the results he saw between 1994 and 1996, so concerned in fact that the 1996 survey was never published. In this article he shares what their research revealed about major changes in the world of software development in the mid-nineties, which significantly changed the complexion of project planning and execution.  InfoQ brings you an excerpt from this month's CHAOS University newsletter.

Cosmic Intergalactic Change

Excerpt from the November 2006 CHAOS University Newsletter
by Jim Johnson, Founder and Chairman of The Standish Group

Lately a few people have asked us to explain the amazing improvement in project cost overruns, from 189% in 1994 to 69% in 1998.  We have done these explanations on many occasions and in many venues. So, what I am going to say is not new. It is included in both of our recent writings, the CHAOS Chronicles 3.0 and our new book, My Life is Failure, but a direct explanation may not jump off the pages and hit you in the face. So here goes.

It would be both great and self-serving to say that the improvements were the result of our research, but we cannot take credit for such improvement. On the other hand, we did see in many of our focus groups and workshops after our initial published report that user involvement was up, executive support was greater, and people were clearer on their business objectives. Also, many organizations are quickly moving to have their project managers PMI certified. All this adds to improvement, but we doubt it would move the results but a few points. This might have resulted in the improvement from 1994’s 189% to 1996’s 142%.

In this month's DARTS survey we asked SURF participants to rate their skill level on estimating project time and cost. A little less than ten percent thought they were very skilled, while almost two-thirds thought they were moderately to poorly skilled.

How skilled you are at estimating time and cost has a direct impact on time and cost overruns. During our spring focus groups of 1998, we asked our IT participants if they had changed the way they estimated projects. The consensus was, they used to estimate the project and then add a safety amount for contingencies and other items for a more conservative approach. In 1998, they had changed their process so that they were then taking their best estimate, and then doubling it and adding half again. Such a change in estimating procedures would directly result in the improved cost overruns, as we saw from 1994 to 1998.  However, it is our opinion that this is not what caused the overruns to dive from 189% to 69%. So, before you read the next paragraph, I want you to think back on what was going on between 1994 and 1998. Lean back, close your eyes, and think what cosmic intergalactic change happened to the IT industry and computer applications that were developed at that time.

Eyes open now? How about your mind? First a clue! In 1996 we showed a mammoth forty 40 percent of projects were canceled. In fact during that year we were so devastated by the results you will not find a formal 1996 CHAOS report. Nonetheless, if you look at any CHAOS table that contains all the years and you will see these results.

What happened in 1996 to cause so many failures? The answer is the Internet. We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications. They would always get screwed up, and you never knew what was in the user’s PC that would cause conflicts. It was a mess, and organizations dumped their client/server projects as fast as they could. Now with Internet-style development, projects are much smaller, simpler, faster to implement, and easier to manage. This push to browser-based clientless access has been the catalyst to this phenomenon. In addition, now we see that IT has the liberty to use their new way of estimating. Bang, bang – improvement! While we have seen steady improvement since 1998, we think it will take another cosmic intergalactic event like the Internet to see these types of improvements again. However, the lessons learned from this single event are immense and drives much of our thinking on the proper way to develop software.

Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

I'm not sure that is quite right by Jonathan Allen

We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications.

During that same time period, two key tools were in play. Visal Basic and Java revolutionized client-server application design and dropped development times by orders of magnitude.

Creating web applications is actually harder than client-server applications right now. You have to account for a of issues that don't exist in a client-server environment including client-side vs server-side code, session management, cross-site scripting attacks, bad query string and form data (especially when GreaseMonkey comes into play), and url management.

Re: I'm not sure that is quite right by Deborah Hartmann

Isn't Jim's point is that at the time, browser-based development was perceived to be easier, so client/server projects got dumped before completion?

I recall hearing that, don't you? "Heck, we'll do it with a browser front end, piece-of-cake!" (famous last words). Fortunately, we'd also learned to buffer up our estimates!

NOW we know that the browser isn't the magic cure :-) and neither is it appropriate for all applications.
Live and learn, lol.

My take on the Standish position by Gordon Divitt

Let me start by declaring my biases (at least the ones relavent to this subject). I have been an attendee of the CHAOS University sessions for more than 10 years and have a deep affection and trust in Jim Johnson and his team. Deb Hartmann happens to be consulting at my firm at this time and I was instrumental in introducing her to Jim and was actually party to the initial interview.

Anyway - I am not sure I totally agree with Jim's "single cause" argument but clearly there were cosmic shifts at play in the time frame under discussion.

Where I do believe the Internet has been massively influential is in its dissemination of news and ideas. No longer do development teams labour away in dark caves restricted to only those resources, and ideas, they were able, or thought, to bring with them. Now if anyone has even the slightest self doubt there is a vibrant community of people who will tell you where you are going wrong - even if their advice is flawed it will give pause for thought etc etc.

I also believe the stigma of failure was eased by the CHAOS reports as it made Senior Management realize that, despite their protestations to the contrary, IT is/was not infallible. Early recognition of a project going off the rails allows for course correction or cancellation.

Lastly the tools available to development teams are just that much better and easier to use. Assembler was a bitch

I also think that lots of the hard stuff has already been done i.e. creating a banking system on a mainframe using CICS/Cobol for the first time from a bunch of paper ledger cards with users who had never seen, never mind used, a computer is infinately harder than polishing one which has been in use for some time and whose weakenesses are pretty obvious and can be better explaind by knowledgable users who see technolgy as just part of their landscape and not in the least intimidating. the better the user the better the solution.

Yes there is tons of sexy new stuff going on but truly ground breaking requirements and ideas are much fewer

Re: My take on the Standish position by James Crear

I am frequent participant at CHAOS University. I believe since the first CHAOS so many years ago I won't say. I have heard Jim Johnson speak on many occasions. Jim Johnson has repeatedly spoken on this subject and has not waffled in his message. I also agree with Gordon that it is not a single issue that caused the drastic down turn in project overruns. I think the point Jim was addressing is Glass and Jørgensen comments that nothing was going on in IT at that time to cause such a change. I am particularly surprised as it is like saying nothing happened on the maiden voyage of the Titanic. Maybe Glass and Jorgensen is also ignoring the icebergs To say nothing happen in IT from 1994 to 1998 is bizarre. It was the biggest challange and change in the industry. Nothing happened in IT; it is when our IT world changed everything we knew. Everything! Even Al Gore wanted credit for the Internet!

I also disagree with Jonathan Allen. There is much more to a project than just developing code. Did anyone ever here of requirements gathering, planning, negiotiating, unit testing, system testing, user sign-off, user training etc. Testing happens so much quicker and easier when you can focus on doing everything at the server. There are not many applications that can?t be done as thin client. If you can do word processing on thin client you can do most anything.

I also think it is easier to develop browser/server based apps than client/server apps. Hence the move by all major development packages to go browser based, not to mention the ease of deployment.
But maybe I have been doing it wrong for the past 38 years.

What about Y2K? by Mary Poppendieck

Something else was going on in 1996 - the media began its consistent reports about the possible disasters that could be triggered at the turn of the century. In 1997 the crescendo began to build. Companies started canceling everything they were doing and directing all-out efforts to Y2K remediation. In fact, Y2K projects were plumbing projects, the requirements were very clear, and the projects were by-and-large completed on time. In about 2001, I read an article that suggested the widespread adoption of project management in IT was due to the overwhelming "success" of Y2K projects.

I think it's called evolution by Paul H

How many of you guys didn't thought at least once about beeing in college/highschool with your actual wisdom? That happened in software development too, it evolved. The '94 developers, most of them are seniors or architects now, they learnt from their mistakes.
It's "amazing" if you see an old project, like a "simulated" mvc only with jsp's, or a small and simple indoor persistance framework.

Re: I think it's called evolution by Deborah Hartmann

How many of you guys didn't thought at least once about being in college/highschool with your actual [current?]wisdom?
Yes, Paul. I thought of that too... if only we'd known then what we've learned since to make things simpler, safer, more robust. :-)

I see opinions, but no evidence by Magne Jorgensen

I'v read software engineering textbooks, led projects, estimated projects, discussed estimation with practitioners, and conducted research on software cost estimation for a number of years, including a recent review of all published software effort estimation research.

Surprisingly, the idea of that the internet should have this amazing impact of estimation accuracy is new to me. I'v heard nobody else claiming this, and have seen no evidence supporting it. This does not mean that the explanation is impossible, only that there is an important difference between somebody's opinion and scientific evidence.

Personally, I don't think it is likely that the internet should have this impact - in many ways many of the internet applications are very complex. I don't however find it meaningful to start an opinion-based discussion on this. So far, we have the Standish Group's attempt to explain this extremely strong estimation accuracy improvement, but no scientific and independent evidence to back it up. Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.

Re: I see opinions, but no evidence by Jonathan Allen

Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.

Even if we did, I wouldn't fully trust it. I belive that the ability to easily release partial solutions can skew reports of estimate accuracy.

It is easy to implement web applications piecemeal, which gives managers the ability to declare a project "done enough" rather than allowing it to overrun. Thick clients can be built this way, but I suspect they are done so as often.

I would like to see accurate and detailed measurements of project success rates, but I fear that may never be possible. There are too many human factors involved even if we could control for technological changes.

Re: I see opinions, but no evidence by Gordon Divitt

This thread has, for me, taken a fascinating turn. Unless it is just an attempt to impune the itegrity of the Standish group is seems like a bunch of folks trying to discount the obvious, albeit aften anecdotal, evidence that IT does better work more cheaply (in constant $$) and more quickly than 10+ years ago.

Everyone who has contributed to this, in whatever ways, should be justifiable proud and see it as a maturing industry which has moved from an art to, almost, a science.

What about OO? by Stephen Davies

Can this improvement be attributed to the internet alone? I think not. In Australia, the period 1992-1998 represents a transition from structured programming techniques to the adoption of OO languages and incremental/iterative delivery.

I worked on several large, unsuccessful projects in the early 1990's, typically COBOL/C language with waterfall methodology. The transition to an OO language, Smalltalk in my case, con-incided with a change to an interative/incremental delivery approach and a string of successful projects were the result.

Not really trying to bang the OO drum here, its just that in my opinion there was more major technical changes going on during this period than the internet.

Re: What about Y2K? by Deborah Hartmann

Hmmm, interesting. I wonder whether the immovable end date of Y2K also hinted to managers that projects didn't HAVE to slip to "succeed"?

Re: I see opinions, but no evidence by john montes


The question is what evidence backs up the assertion that the Internet was the single greatest contributing factor that reduced software project cost overruns by 120 percentage points between 1994 and 1998.

Jim Johnson states "What happened in 1996 to cause so many failures? The answer is the Internet?".

Does Jim have some statistically significant data to backup that claim?

I did notice that the 1994 CHAOS report has been removed from the Standish website with the following statement :
“The revolutionary 1994 CHAOS and the Unfinished Voyages reports were great a decade ago and they were instrumental in changing the project management world. However they are no longer valid, nor do they reflect the latest research information available today - so we have removed them from our website.”

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

13 Discuss