Standish: Why were Project Failures Up and Cost Overruns Down in 1998?
The CHAOS statistics on project failure are frequently cited by those implementing Agile or other forms of process change, so their validity is of importance to the many people who continue to use them. In August, triggered by Robert Glass's questions in August's Communications of the ACM , InfoQ interviewed Jim Johnson, creator of the CHAOS Chronicles on project failure, to hear how their data is collected.
In the discussion thread that followed the interview, one question surfaced as still unresolved: how does the Standish Group explain the amazing improvement in cost overrun from 189% in 1994 to 69% in 1998? The implication of this question, also posed in several other places on the web, is that Standish may have adjusted their research methods without revealing the change in their reports. If true, comparisons of some CHAOS data from 1994 and 1998 could be meaningless.
Standish founder Jim Johnson was also concerned by the results he saw between 1994 and 1996, so concerned in fact that the 1996 survey was never published. In this article he shares what their research revealed about major changes in the world of software development in the mid-nineties, which significantly changed the complexion of project planning and execution. InfoQ brings you an excerpt from this month's CHAOS University newsletter.
Cosmic Intergalactic ChangeExcerpt from the November 2006 CHAOS University Newsletter
by Jim Johnson, Founder and Chairman of The Standish Group
Lately a few people have asked us to explain the amazing improvement in project cost overruns, from 189% in 1994 to 69% in 1998. We have done these explanations on many occasions and in many venues. So, what I am going to say is not new. It is included in both of our recent writings, the CHAOS Chronicles 3.0 and our new book, My Life is Failure, but a direct explanation may not jump off the pages and hit you in the face. So here goes.
It would be both great and self-serving to say that the improvements were the result of our research, but we cannot take credit for such improvement. On the other hand, we did see in many of our focus groups and workshops after our initial published report that user involvement was up, executive support was greater, and people were clearer on their business objectives. Also, many organizations are quickly moving to have their project managers PMI certified. All this adds to improvement, but we doubt it would move the results but a few points. This might have resulted in the improvement from 1994’s 189% to 1996’s 142%.
In this month's DARTS survey we asked SURF participants to rate their skill level on estimating project time and cost. A little less than ten percent thought they were very skilled, while almost two-thirds thought they were moderately to poorly skilled.
How skilled you are at estimating time and cost has a direct impact on time and cost overruns. During our spring focus groups of 1998, we asked our IT participants if they had changed the way they estimated projects. The consensus was, they used to estimate the project and then add a safety amount for contingencies and other items for a more conservative approach. In 1998, they had changed their process so that they were then taking their best estimate, and then doubling it and adding half again. Such a change in estimating procedures would directly result in the improved cost overruns, as we saw from 1994 to 1998. However, it is our opinion that this is not what caused the overruns to dive from 189% to 69%. So, before you read the next paragraph, I want you to think back on what was going on between 1994 and 1998. Lean back, close your eyes, and think what cosmic intergalactic change happened to the IT industry and computer applications that were developed at that time.
Eyes open now? How about your mind? First a clue! In 1996 we showed a mammoth forty 40 percent of projects were canceled. In fact during that year we were so devastated by the results you will not find a formal 1996 CHAOS report. Nonetheless, if you look at any CHAOS table that contains all the years and you will see these results.
What happened in 1996 to cause so many failures? The answer is the Internet. We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications. They would always get screwed up, and you never knew what was in the user’s PC that would cause conflicts. It was a mess, and organizations dumped their client/server projects as fast as they could. Now with Internet-style development, projects are much smaller, simpler, faster to implement, and easier to manage. This push to browser-based clientless access has been the catalyst to this phenomenon. In addition, now we see that IT has the liberty to use their new way of estimating. Bang, bang – improvement! While we have seen steady improvement since 1998, we think it will take another cosmic intergalactic event like the Internet to see these types of improvements again. However, the lessons learned from this single event are immense and drives much of our thinking on the proper way to develop software.
I'm not sure that is quite right
We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications.
During that same time period, two key tools were in play. Visal Basic and Java revolutionized client-server application design and dropped development times by orders of magnitude.
Creating web applications is actually harder than client-server applications right now. You have to account for a of issues that don't exist in a client-server environment including client-side vs server-side code, session management, cross-site scripting attacks, bad query string and form data (especially when GreaseMonkey comes into play), and url management.
Re: I'm not sure that is quite right
I recall hearing that, don't you? "Heck, we'll do it with a browser front end, piece-of-cake!" (famous last words). Fortunately, we'd also learned to buffer up our estimates!
NOW we know that the browser isn't the magic cure :-) and neither is it appropriate for all applications.
Live and learn, lol.
My take on the Standish position
Anyway - I am not sure I totally agree with Jim's "single cause" argument but clearly there were cosmic shifts at play in the time frame under discussion.
Where I do believe the Internet has been massively influential is in its dissemination of news and ideas. No longer do development teams labour away in dark caves restricted to only those resources, and ideas, they were able, or thought, to bring with them. Now if anyone has even the slightest self doubt there is a vibrant community of people who will tell you where you are going wrong - even if their advice is flawed it will give pause for thought etc etc.
I also believe the stigma of failure was eased by the CHAOS reports as it made Senior Management realize that, despite their protestations to the contrary, IT is/was not infallible. Early recognition of a project going off the rails allows for course correction or cancellation.
Lastly the tools available to development teams are just that much better and easier to use. Assembler was a bitch
I also think that lots of the hard stuff has already been done i.e. creating a banking system on a mainframe using CICS/Cobol for the first time from a bunch of paper ledger cards with users who had never seen, never mind used, a computer is infinately harder than polishing one which has been in use for some time and whose weakenesses are pretty obvious and can be better explaind by knowledgable users who see technolgy as just part of their landscape and not in the least intimidating. the better the user the better the solution.
Yes there is tons of sexy new stuff going on but truly ground breaking requirements and ideas are much fewer
Re: My take on the Standish position
I also disagree with Jonathan Allen. There is much more to a project than just developing code. Did anyone ever here of requirements gathering, planning, negiotiating, unit testing, system testing, user sign-off, user training etc. Testing happens so much quicker and easier when you can focus on doing everything at the server. There are not many applications that can?t be done as thin client. If you can do word processing on thin client you can do most anything.
I also think it is easier to develop browser/server based apps than client/server apps. Hence the move by all major development packages to go browser based, not to mention the ease of deployment.
But maybe I have been doing it wrong for the past 38 years.
What about Y2K?
I think it's called evolution
It's "amazing" if you see an old project, like a "simulated" mvc only with jsp's, or a small and simple indoor persistance framework.
Re: I think it's called evolution
How many of you guys didn't thought at least once about being in college/highschool with your actual [current?]wisdom?Yes, Paul. I thought of that too... if only we'd known then what we've learned since to make things simpler, safer, more robust. :-)
I see opinions, but no evidence
Surprisingly, the idea of that the internet should have this amazing impact of estimation accuracy is new to me. I'v heard nobody else claiming this, and have seen no evidence supporting it. This does not mean that the explanation is impossible, only that there is an important difference between somebody's opinion and scientific evidence.
Personally, I don't think it is likely that the internet should have this impact - in many ways many of the internet applications are very complex. I don't however find it meaningful to start an opinion-based discussion on this. So far, we have the Standish Group's attempt to explain this extremely strong estimation accuracy improvement, but no scientific and independent evidence to back it up. Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.
Re: I see opinions, but no evidence
Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.
Even if we did, I wouldn't fully trust it. I belive that the ability to easily release partial solutions can skew reports of estimate accuracy.
It is easy to implement web applications piecemeal, which gives managers the ability to declare a project "done enough" rather than allowing it to overrun. Thick clients can be built this way, but I suspect they are done so as often.
I would like to see accurate and detailed measurements of project success rates, but I fear that may never be possible. There are too many human factors involved even if we could control for technological changes.
Re: I see opinions, but no evidence
Everyone who has contributed to this, in whatever ways, should be justifiable proud and see it as a maturing industry which has moved from an art to, almost, a science.
What about OO?
I worked on several large, unsuccessful projects in the early 1990's, typically COBOL/C language with waterfall methodology. The transition to an OO language, Smalltalk in my case, con-incided with a change to an interative/incremental delivery approach and a string of successful projects were the result.
Not really trying to bang the OO drum here, its just that in my opinion there was more major technical changes going on during this period than the internet.
Re: What about Y2K?
Re: I see opinions, but no evidence
The question is what evidence backs up the assertion that the Internet was the single greatest contributing factor that reduced software project cost overruns by 120 percentage points between 1994 and 1998.
Jim Johnson states "What happened in 1996 to cause so many failures? The answer is the Internet?".
Does Jim have some statistically significant data to backup that claim?
I did notice that the 1994 CHAOS report has been removed from the Standish website with the following statement :
“The revolutionary 1994 CHAOS and the Unfinished Voyages reports were great a decade ago and they were instrumental in changing the project management world. However they are no longer valid, nor do they reflect the latest research information available today - so we have removed them from our website.”