Standish CHAOS Report Methods Questioned

Posted by Deborah Hartmann on Aug 25, 2006 04:58 PM

Community

Agile

Topics

Delivering Value,

Methodologies

Tags

Statistics,

Waterfall

Robert L. Glass, in August's Communications of the ACM, asked "The Standish report: does it really describe a software crisis?"  Glass sets out to reconsider the relevancy of this frequently cited report on software project failures, but in absence of hard data, it seems Glass and others can't get beyond asking questions.

Glass is a well-respected author, active in the field of computing and software for over 45 years, largely in industry, but also as an academic. He is the author of over 20 books including Software Folklore, Computing Catastrophes, Computer Shakeout, Software 2020, Software Runaways, Computing Calamities, and Facts and Fallacies of Software Engineering. He was for 15 years a Lecturer for the ACM, and was named a Fellow of the ACM in 1998.

Glass feels there is something seriously flawed in current "software crisis" thinking. He refers to the assumptions underlying decades of rhetoric about software projects always being over budget, behind schedule, and unreliable - largely based on research compiled by the Standish Group under the name The CHAOS Chronicles, first published in 1994 . This is undisputedly the most-quoted source of information supporting the idea of what Glass calls the "software crisis", and is the source of now-familiar statistics such as: "84% of projects fail or are significantly challenged", and "45% of developed features are never used".  The Standish SURF database, source of CHAOS statistics, currently holds data on over 50,000 projects from around the world - information understandably attractive to software researchers.

Numerous academic studies have been designed to address this crisis and methdology evangelists often frame their topics as solutions to this crisis.  But, Glass asks, how can such a crisis exist, in "an era that simply wouldn’t be possible if we didn’t have astoundingly successful software to make all those computers do the wonderful things they do."  Clearly, something is not adding up for Glass.

He states that researchers who have queried Standish, asking for a description of their research process, a summary of their latest findings, and in general a scholarly discussion of the validity of the findings, have been rebuffed, calling the Standish methods into question.  The researchers he referenced did note in their report that it's not uncommon for commercial research firms to decline to share their detailed findings publicly, so their 2006 Simula Research report questioning the correctness of Standish's method is based on the initial 1994 CHAOS findings, which are available free of charge.  (Glass questions the value of using such old data.)

Glass, said that "most research studies conducted by academic and industry researchers arrive at data largely inconsistent with the Standish findings." and referred to the recent Simula Research paper, "How Large Are Software Cost Overruns?".  That study compared CHAOS with other research, citing three scientifically reviewed studies which in turn reported on between 89 and 191 North American software projects.  Findings of these three studies differed greatly from the 1994 CHAOS survey, which reported on 8380 international software projects.  In addition, the Simula researchers found a sentence in the 1994 report suggestive that the Standish findings may, in fact, be biased toward failure, but Glass acknowledges "it is not at all clear how much of the study was based on the initial contact that sentence describes."

In his recent Communications of the ACM column, Glass asked two questions of the Chaos Report: "Does it represent reality?" and "Is it supported by research findings?"  Glass has been asking similar questions for years.  In 2002, writing in his Loyal Opposition column (IEEE Software, Jan/Feb 2002), he reportedly argued that the words chaos and crisis may no longer apply, based on shifts reported by Standish since 1994.  In his 2005 article IT Failure Rates—70% or 10–15%?, he questioned the 10-year trend toward improvement in the Standish results and related his frustration with repeated unsuccessful attempts to engage Standish on the subject of their data. 

Glass answered his own questions this time: "The truth of the matter is, we don't really know."  In the absence of satisfying answers, beyond what's on the Standish site, Glass concluded this latest column with a challenge: "Standish, please tell us whether the data we have all been quoting for more than a decade really means what some have been saying it means."

Blogger Dave Nicolette notes that his own experience in IT over the past 28+ years is generally consistent with Standish's findings, and cautions against jumping to conclusions.  He notes that "Standish is in the business of doing industry research and selling the resulting reports. Customer confidence in the validity of the results is absolutely critical to their business. To deliberately bias the results would surely break that confidence and cause their business to suffer. It would be illogical for them to do so."

---

Related content: InfoQ interviewed Jim Johnson of the Standish Group this week, on how their research is done, and the impact of Agile on the IT industry.

• Older >