Why 1994 and 1998 CHAOS Stats Differ Widely

by Deborah Hartmann Preuss on Oct 03, 2006 |
Jim Johnson, creator of the CHAOS Chronicles on project failure, answers a question outstanding after our August interview: How does he explain the amazing reduction in cost overruns between 1994 and 1998?  As it turns out, Standish collected data and planned to publish a CHAOS report in 1996 but held it back due to these unexpected results. Johnson shares what their research revealed happened.

The CHAOS statistics on project failure are frequently cited by those implementing Agile or other organizational process change, so their validity is of importance to the many people who use them.  The implication of this question, posed in several other places on the web as well, is that Standish may have adjusted their research methods without revealing the change in their reports.  If true, comparisons of some CHAOS data from 1994 and 1998 could be meaningless. 

Standish founder Jim Johnson and his colleagues were also concerned by the results they saw between 1994 and 1996, so concerned that the 1996 survey went unpublished while they investigated the source of the unexpected shifts they saw after 1994: project failures grew from 31% to 40% in two years, while cost overruns dropped precipitously from 189% to 142% in 1996.

In this article he shares what their research revealed about shifts in the world of software development in the mid-nineties, which significantly changed the complexion of project planning and execution.

Read the InfoQ article: Standish: Why were Project Failures Up and Cost Overruns Down in 1998?

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

I'm not sure that is quite right by Jonathan Allen

We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications.

During that same time period, two key tools were in play. Visal Basic and Java revolutionized client-server application design and dropped development times by orders of magnitude.

Creating web applications is actually harder than client-server applications right now. You have to account for a of issues that don't exist in a client-server environment including client-side vs server-side code, session management, cross-site scripting attacks, bad query string and form data (especially when GreaseMonkey comes into play), and url management.

Re: I'm not sure that is quite right by Deborah Hartmann

Isn't Jim's point is that at the time, browser-based development was perceived to be easier, so client/server projects got dumped before completion?

I recall hearing that, don't you? "Heck, we'll do it with a browser front end, piece-of-cake!" (famous last words). Fortunately, we'd also learned to buffer up our estimates!

NOW we know that the browser isn't the magic cure :-) and neither is it appropriate for all applications.
Live and learn, lol.

My take on the Standish position by Gordon Divitt

Let me start by declaring my biases (at least the ones relavent to this subject). I have been an attendee of the CHAOS University sessions for more than 10 years and have a deep affection and trust in Jim Johnson and his team. Deb Hartmann happens to be consulting at my firm at this time and I was instrumental in introducing her to Jim and was actually party to the initial interview.

Anyway - I am not sure I totally agree with Jim's "single cause" argument but clearly there were cosmic shifts at play in the time frame under discussion.

Where I do believe the Internet has been massively influential is in its dissemination of news and ideas. No longer do development teams labour away in dark caves restricted to only those resources, and ideas, they were able, or thought, to bring with them. Now if anyone has even the slightest self doubt there is a vibrant community of people who will tell you where you are going wrong - even if their advice is flawed it will give pause for thought etc etc.

I also believe the stigma of failure was eased by the CHAOS reports as it made Senior Management realize that, despite their protestations to the contrary, IT is/was not infallible. Early recognition of a project going off the rails allows for course correction or cancellation.

Lastly the tools available to development teams are just that much better and easier to use. Assembler was a bitch

I also think that lots of the hard stuff has already been done i.e. creating a banking system on a mainframe using CICS/Cobol for the first time from a bunch of paper ledger cards with users who had never seen, never mind used, a computer is infinately harder than polishing one which has been in use for some time and whose weakenesses are pretty obvious and can be better explaind by knowledgable users who see technolgy as just part of their landscape and not in the least intimidating. the better the user the better the solution.

Yes there is tons of sexy new stuff going on but truly ground breaking requirements and ideas are much fewer

Re: My take on the Standish position by James Crear

I am frequent participant at CHAOS University. I believe since the first CHAOS so many years ago I won't say. I have heard Jim Johnson speak on many occasions. Jim Johnson has repeatedly spoken on this subject and has not waffled in his message. I also agree with Gordon that it is not a single issue that caused the drastic down turn in project overruns. I think the point Jim was addressing is Glass and Jørgensen comments that nothing was going on in IT at that time to cause such a change. I am particularly surprised as it is like saying nothing happened on the maiden voyage of the Titanic. Maybe Glass and Jorgensen is also ignoring the icebergs To say nothing happen in IT from 1994 to 1998 is bizarre. It was the biggest challange and change in the industry. Nothing happened in IT; it is when our IT world changed everything we knew. Everything! Even Al Gore wanted credit for the Internet!

I also disagree with Jonathan Allen. There is much more to a project than just developing code. Did anyone ever here of requirements gathering, planning, negiotiating, unit testing, system testing, user sign-off, user training etc. Testing happens so much quicker and easier when you can focus on doing everything at the server. There are not many applications that can?t be done as thin client. If you can do word processing on thin client you can do most anything.

I also think it is easier to develop browser/server based apps than client/server apps. Hence the move by all major development packages to go browser based, not to mention the ease of deployment.
But maybe I have been doing it wrong for the past 38 years.

What about Y2K? by Mary Poppendieck

Something else was going on in 1996 - the media began its consistent reports about the possible disasters that could be triggered at the turn of the century. In 1997 the crescendo began to build. Companies started canceling everything they were doing and directing all-out efforts to Y2K remediation. In fact, Y2K projects were plumbing projects, the requirements were very clear, and the projects were by-and-large completed on time. In about 2001, I read an article that suggested the widespread adoption of project management in IT was due to the overwhelming "success" of Y2K projects.

I think it's called evolution by Paul H

How many of you guys didn't thought at least once about beeing in college/highschool with your actual wisdom? That happened in software development too, it evolved. The '94 developers, most of them are seniors or architects now, they learnt from their mistakes.
It's "amazing" if you see an old project, like a "simulated" mvc only with jsp's, or a small and simple indoor persistance framework.

Re: I think it's called evolution by Deborah Hartmann

How many of you guys didn't thought at least once about being in college/highschool with your actual [current?]wisdom?
Yes, Paul. I thought of that too... if only we'd known then what we've learned since to make things simpler, safer, more robust. :-)

I see opinions, but no evidence by Magne Jorgensen

I'v read software engineering textbooks, led projects, estimated projects, discussed estimation with practitioners, and conducted research on software cost estimation for a number of years, including a recent review of all published software effort estimation research.

Surprisingly, the idea of that the internet should have this amazing impact of estimation accuracy is new to me. I'v heard nobody else claiming this, and have seen no evidence supporting it. This does not mean that the explanation is impossible, only that there is an important difference between somebody's opinion and scientific evidence.

Personally, I don't think it is likely that the internet should have this impact - in many ways many of the internet applications are very complex. I don't however find it meaningful to start an opinion-based discussion on this. So far, we have the Standish Group's attempt to explain this extremely strong estimation accuracy improvement, but no scientific and independent evidence to back it up. Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.

Re: I see opinions, but no evidence by Jonathan Allen

Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.

Even if we did, I wouldn't fully trust it. I belive that the ability to easily release partial solutions can skew reports of estimate accuracy.

It is easy to implement web applications piecemeal, which gives managers the ability to declare a project "done enough" rather than allowing it to overrun. Thick clients can be built this way, but I suspect they are done so as often.

I would like to see accurate and detailed measurements of project success rates, but I fear that may never be possible. There are too many human factors involved even if we could control for technological changes.

Re: I see opinions, but no evidence by Gordon Divitt

This thread has, for me, taken a fascinating turn. Unless it is just an attempt to impune the itegrity of the Standish group is seems like a bunch of folks trying to discount the obvious, albeit aften anecdotal, evidence that IT does better work more cheaply (in constant $$) and more quickly than 10+ years ago.

Everyone who has contributed to this, in whatever ways, should be justifiable proud and see it as a maturing industry which has moved from an art to, almost, a science.

What about OO? by Stephen Davies

Can this improvement be attributed to the internet alone? I think not. In Australia, the period 1992-1998 represents a transition from structured programming techniques to the adoption of OO languages and incremental/iterative delivery.

I worked on several large, unsuccessful projects in the early 1990's, typically COBOL/C language with waterfall methodology. The transition to an OO language, Smalltalk in my case, con-incided with a change to an interative/incremental delivery approach and a string of successful projects were the result.

Not really trying to bang the OO drum here, its just that in my opinion there was more major technical changes going on during this period than the internet.

Re: What about Y2K? by Deborah Hartmann

Hmmm, interesting. I wonder whether the immovable end date of Y2K also hinted to managers that projects didn't HAVE to slip to "succeed"?

Re: I see opinions, but no evidence by john montes


The question is what evidence backs up the assertion that the Internet was the single greatest contributing factor that reduced software project cost overruns by 120 percentage points between 1994 and 1998.

Jim Johnson states "What happened in 1996 to cause so many failures? The answer is the Internet?".

Does Jim have some statistically significant data to backup that claim?

I did notice that the 1994 CHAOS report has been removed from the Standish website with the following statement :
“The revolutionary 1994 CHAOS and the Unfinished Voyages reports were great a decade ago and they were instrumental in changing the project management world. However they are no longer valid, nor do they reflect the latest research information available today - so we have removed them from our website.”

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

13 Discuss

Educational Content

General Feedback
Editorial and all content copyright © 2006-2013 C4Media Inc. hosted at Contegix, the best ISP we've ever worked with.
Privacy policy