Why 1994 and 1998 CHAOS Stats Differ Widely
The CHAOS statistics on project failure are frequently cited by those implementing Agile or other organizational process change, so their validity is of importance to the many people who use them. The implication of this question, posed in several other places on the web as well, is that Standish may have adjusted their research methods without revealing the change in their reports. If true, comparisons of some CHAOS data from 1994 and 1998 could be meaningless.
Standish founder Jim Johnson and his colleagues were also concerned by the results they saw between 1994 and 1996, so concerned that the 1996 survey went unpublished while they investigated the source of the unexpected shifts they saw after 1994: project failures grew from 31% to 40% in two years, while cost overruns dropped precipitously from 189% to 142% in 1996.
In this article he shares what their research revealed about shifts in the world of software development in the mid-nineties, which significantly changed the complexion of project planning and execution.
Read the InfoQ article: Standish: Why were Project Failures Up and Cost Overruns Down in 1998?
I'm not sure that is quite right
We went from a client/server development and implementation style to an Internet style. Client/server applications are a magnitude more complex to develop and implement than Internet applications.
During that same time period, two key tools were in play. Visal Basic and Java revolutionized client-server application design and dropped development times by orders of magnitude.
Creating web applications is actually harder than client-server applications right now. You have to account for a of issues that don't exist in a client-server environment including client-side vs server-side code, session management, cross-site scripting attacks, bad query string and form data (especially when GreaseMonkey comes into play), and url management.
Re: I'm not sure that is quite right
I recall hearing that, don't you? "Heck, we'll do it with a browser front end, piece-of-cake!" (famous last words). Fortunately, we'd also learned to buffer up our estimates!
NOW we know that the browser isn't the magic cure :-) and neither is it appropriate for all applications.
Live and learn, lol.
My take on the Standish position
Anyway - I am not sure I totally agree with Jim's "single cause" argument but clearly there were cosmic shifts at play in the time frame under discussion.
Where I do believe the Internet has been massively influential is in its dissemination of news and ideas. No longer do development teams labour away in dark caves restricted to only those resources, and ideas, they were able, or thought, to bring with them. Now if anyone has even the slightest self doubt there is a vibrant community of people who will tell you where you are going wrong - even if their advice is flawed it will give pause for thought etc etc.
I also believe the stigma of failure was eased by the CHAOS reports as it made Senior Management realize that, despite their protestations to the contrary, IT is/was not infallible. Early recognition of a project going off the rails allows for course correction or cancellation.
Lastly the tools available to development teams are just that much better and easier to use. Assembler was a bitch
I also think that lots of the hard stuff has already been done i.e. creating a banking system on a mainframe using CICS/Cobol for the first time from a bunch of paper ledger cards with users who had never seen, never mind used, a computer is infinately harder than polishing one which has been in use for some time and whose weakenesses are pretty obvious and can be better explaind by knowledgable users who see technolgy as just part of their landscape and not in the least intimidating. the better the user the better the solution.
Yes there is tons of sexy new stuff going on but truly ground breaking requirements and ideas are much fewer
Re: My take on the Standish position
I also disagree with Jonathan Allen. There is much more to a project than just developing code. Did anyone ever here of requirements gathering, planning, negiotiating, unit testing, system testing, user sign-off, user training etc. Testing happens so much quicker and easier when you can focus on doing everything at the server. There are not many applications that can?t be done as thin client. If you can do word processing on thin client you can do most anything.
I also think it is easier to develop browser/server based apps than client/server apps. Hence the move by all major development packages to go browser based, not to mention the ease of deployment.
But maybe I have been doing it wrong for the past 38 years.
What about Y2K?
I think it's called evolution
It's "amazing" if you see an old project, like a "simulated" mvc only with jsp's, or a small and simple indoor persistance framework.
Re: I think it's called evolution
How many of you guys didn't thought at least once about being in college/highschool with your actual [current?]wisdom?Yes, Paul. I thought of that too... if only we'd known then what we've learned since to make things simpler, safer, more robust. :-)
I see opinions, but no evidence
Surprisingly, the idea of that the internet should have this amazing impact of estimation accuracy is new to me. I'v heard nobody else claiming this, and have seen no evidence supporting it. This does not mean that the explanation is impossible, only that there is an important difference between somebody's opinion and scientific evidence.
Personally, I don't think it is likely that the internet should have this impact - in many ways many of the internet applications are very complex. I don't however find it meaningful to start an opinion-based discussion on this. So far, we have the Standish Group's attempt to explain this extremely strong estimation accuracy improvement, but no scientific and independent evidence to back it up. Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.
Re: I see opinions, but no evidence
Do we, for example, have indpendent research indicating that internet-applications have an extremely much better estimation accuracy than other types of applications? Not as far as I know.
Even if we did, I wouldn't fully trust it. I belive that the ability to easily release partial solutions can skew reports of estimate accuracy.
It is easy to implement web applications piecemeal, which gives managers the ability to declare a project "done enough" rather than allowing it to overrun. Thick clients can be built this way, but I suspect they are done so as often.
I would like to see accurate and detailed measurements of project success rates, but I fear that may never be possible. There are too many human factors involved even if we could control for technological changes.
Re: I see opinions, but no evidence
Everyone who has contributed to this, in whatever ways, should be justifiable proud and see it as a maturing industry which has moved from an art to, almost, a science.
What about OO?
I worked on several large, unsuccessful projects in the early 1990's, typically COBOL/C language with waterfall methodology. The transition to an OO language, Smalltalk in my case, con-incided with a change to an interative/incremental delivery approach and a string of successful projects were the result.
Not really trying to bang the OO drum here, its just that in my opinion there was more major technical changes going on during this period than the internet.
Re: What about Y2K?
Re: I see opinions, but no evidence
The question is what evidence backs up the assertion that the Internet was the single greatest contributing factor that reduced software project cost overruns by 120 percentage points between 1994 and 1998.
Jim Johnson states "What happened in 1996 to cause so many failures? The answer is the Internet?".
Does Jim have some statistically significant data to backup that claim?
I did notice that the 1994 CHAOS report has been removed from the Standish website with the following statement :
“The revolutionary 1994 CHAOS and the Unfinished Voyages reports were great a decade ago and they were instrumental in changing the project management world. However they are no longer valid, nor do they reflect the latest research information available today - so we have removed them from our website.”
Tom Gilb & Kai Gilb Jan 26, 2015