BT

InfoQ Interview: Jim Johnson, Creator of the CHAOS Chronicles

by Deborah Hartmann Preuss on Aug 28, 2006 |
Jim Johnson, the founder and chairman of the Standish Group is probably best known for his research on why projects fail, published in The CHAOS Chronicles. The Chronicles comprise 12 years of independent commercial research on project performance of over 50,000 completed IT projects.

The objectives of CHAOS research are to document the scope of application software development project failures, the major factors for failure, and ways to reduce failure.  In 1994, The Standish Group made public its first CHAOS Report, documenting the billions of dollars wasted on software development for projects that were never completed. That report included statistics such as: "84% of projects fail or are significantly challenged", and "45% of developed features are never used" and is among the most oft-quoted in the industry since then.

Taking time out from his vacation, Johnson spoke with InfoQ in this interview about how this research came to be, how it is done, and the role of Agile in his findings.  Johnson's work with project failure has reinforced his belief that small is best - small teams, small projects, and the Agile approach to software development.



Related News: 
    Standish CHAOS Report Methods Questioned, August 2006.
    Standish: Why were Project Failures Up and Cost Overruns Down in 1998? October 2006

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Two questions to Jim Johnson of the Standish Group by Magne Jorgensen

Hi!

The interview with Jim Johnson was interesting, but some important questions were not addressed sufficiently, I think. I would like Jim Johnson to answer the following two questions related to the Standish Group's CHAOS reports:

1) How would the Standish Group explain the amazing improvement in cost overrun from 189% (displayed as 180% in the figure in the interview text) in 1994 to 69% in 1998? Something really interesting must have happened - not observed by me or other people I have discussed this with - or can it be that the initial data set is biased towards problem projects and, consequently, not representative?

2) How was (and is) the cost overrun in the report calculated? Why is this essential information not described in the reports (and emails requesting this information not answered)?

This questions are not merely of historical interest, e.g., whether the image of a software crisis was created by a misleading report or not, but also of great interest for the current process improvement in software cost estimation. If we believe that there were an immense improvement, we should also understand why. If we want to use more recent reports by the Standish Group, we should know how they calculate their number. An answer of the above two questions would therefore be warmely welcomed.

More on the background for these questions (and a critique of the 1994 CHAOS report) can be found in my publication in Information and Software Technology downloadable from www.simula.no/departments/engineering/publications/Jorgensen.2006.4).

regards
Magne Jørgensen
Simula Research Laboratory

My managers ask a different question by j c

Not "why do we fail?"
But "How do we succeed?"

I think because software has more failures than successes failure analysis is highly warranted. But given the state of the discipline, the tools of the trade arent much help.

A block of metal has properties which can be understood to withstand the environment in which it is used to support something.

Software failure analysis is constantly focused on the foreman and construction worker and rarely the building materials. Maybe the materials we are working with are awful.

Only includes big clients? by j c

Isnt that a little skewed when the basic tools that these companies "use" when they fail are IBMs and Microsofts and Oracles?

What about the impact of negative consultant relationships that only exist to sell more licenses?

What about shelfware that is sold by the ton by vendors? That all falls under the failure category.

"Adopt agile and our crappy direction we sold you for millions might work out" Hear that in a focus group lately?

I dunno this sound like a vendor approach to getting a better tasting hamburger when the cook spit in it when you were waiting at the drive through.

How about some innovation from the supposed innovators instead of ivory tower theory?

Re: Two questions to Jim Johnson of the Standish Group by Cameron Purdy

How would the Standish Group explain the amazing improvement in cost overrun from 189% (displayed as 180% in the figure in the interview text) in 1994 to 69% in 1998?


A new computer language? ;-)

Peace,

Cameron Purdy
www.tangosol.com/



Peace,

Cameron Purdy
www.tangosol.com/

Re: Two questions to Jim Johnson of the Standish Group by Deborah Hartmann

[editor's note: Jim was just starting a vacation when we did this interview, so I'll check in with him when he gets back - perhaps after Labour Day? I'm not sure. - deb ]

Re: Two questions to Jim Johnson of the Standish Group by Deborah Hartmann

Hello all. Now returned from vacation, Jim has asked me to post this reply, which is in two parts - a general reply and a more specific one at the end.

General answer to Magne Jørgensen, Robert Glass, et al.

We are an advisory research firm much like a Gartner or Forrester. What sets us apart from them is that we use our primary research to form our opinions, whereas they use their individual consultants. Neither they nor we can afford to give our opinions away for free. We have facilities, utilities, and personnel and we must, the same as you, be able to pay our bills. Just because someone asks a question, does not mean we will respond with an answer. In fact, we most likely will not. It is not rebuff or slight, it is just our business model to survive.

Our current standard answer to a CHAOS inquiry is, first: please purchase our new book, "My Life is Failure" in our online store. If that does not satisfy you, then you need to join CHAOS University. If you do not find your answer or answers there then you need to purchase our inquiry services. Then we will work to answer your questions.

Besides being a client you can get CHAOS information by participating in the research forum. If you would like to participant in our research you can join the Standish User Research Forum. You need to be an organization that develops software for your own use to qualify. This year we are offering a free reviewers copy of a book we just completed titled "The Pubic Execution of Miss Scarlet" as the honorarium. It is a tale of a fictional project in peril, but uses much of our lessons learned.

Every year since 1995 we hold a CHAOS University and discuss many of these issue. It is strange that both Jørgensen and Glass have never applied or professed interest in joining us. Some answers can be found if you join us at CHAOS University 2007 or one of the many outreach events. So you can contribute to the CHAOS research by providing funding or sweat, but short of that you will and must be ignored by design.

We do provide much of our research to the legitimate press and I have personally written many articles on our CHAOS and other research. We have given many talks to public and private organizations. In June, I presented some of our research to an IEEE conference in Philadelphia and the Delivering Project Excellence conference in Phoenix. This month I am doing a keynote talk to the National Association for Justice Information Systems in Boca Raton, Florida.

As to Jørgensen's two questions:

Question 2: How was (and is) the cost overrun in the report calculated? The simple answer to this is that we ask "what was the estimated cost verses the actual cost?" We use a number of different instruments to collect the data and derive our overrun assessment. While we really do have a rocket scientist on staff, this is basic math.

On the first question and any other questions you will have to read "My Life is Failure" to find those answers.

Re: Only includes big clients? by Deborah Hartmann

jc, have you looked at the standish website? look around, you might find indications that they are concerned about the same factors you mention.

Did the Standish Group (Jim Johnson) answer my two questions? No. by Magne Jorgensen

Hi again!

The answers on my two questions were as expected. No answer and answer without information. I'm confident that Jim Johnson understands that none of his answers are really answers on my questions.

1) My first question was the most important, i.e., "How would the Standish Group explain the amazing improvement in cost overrun from 189% (displayed as 180% in the figure in the interview text) in 1994 to 69% in 1998?" Not surprisingly, this question was not answered at all. If there existed a good answer on this question (other than a poor research method) I guess that Jim Johnson would have told us. That there should be a big secret explanation for this amazing improvement only avaiable at his "university", not discovered by anyone else, is higly unbelievable. I would welcome to hear from people who has participated at the Standish Groups "university" to join this discussion and tell us whether this really is explained there (mail me at magnej@simula.no). I have personally read several of Jim's papers and bought the full (very expensive) CHAOS report. I have found nothing that explains the extreme improvement there. Until this has been explained, we simply cannot accept the validity of the Standish Group results from 1994. They are too different from data from other studies in that period. By the way, I will buy his book "My life is failure" and mail this forum a message if I find an explanation there.

2) My second question was about how the Standish Group measured the cost overrun. Jim Johnson's answer was "what was the estimated cost versus the actual cost". This answer is without information. Obviously, their measures has something to do with estimated and actual cost, but type of input data (wich from the template I have seen, seems to exclude project underruns and be based on categories of overrun, not the exact numbers), the formula used to calculate the overrun, how to interpret the numbers, how they include or not include projects never completed, etc. is not explained. If the Standish Group want to communicate with people (different from than those visiting their "university"), they should also ensure that the communication is meaningful. A number without explanation of what is measure can easily be misleading information.

The lack of answers strenghten my previous critique, see: www.simula.no/departments/engineering/publications/Jorgensen.2006.4



regards
Magne Jørgensen
Simula Research Laboratory

Re: Two questions to Jim Johnson of the Standish Group by Charlie Trainor

Having seen Magne Jørgensen's critique of the 1994 Chaos report, I checked for myself and discovered it does indeed have internal inconsistencies and a lack of clarity around exactly what numbers are being quoted. The report had a lot of value at the time in making people aware of the abysmal success rate of software projects (at least from the traditional project management perspective), and the Standish Group has a lot of good advice on how to run software projects; but the actual numbers from 1994 (such as the dubious 189% overrun figure) should not be used as if they were from a peer reviewed academic paper.

Re: Did the Standish Group answer my two questions? WELL ONE, PERHAPS by Deborah Hartmann

Standish: Why were Project Failures Up and Cost Overruns Down in 1998?

www.infoq.com/articles/chaos-1998-failure-stats

Business - Technology Disconnects by John Franks

Good article. Getting the best bang for your buck is never out-of-fashion, and given the acceleration of business challenges and choices, requires a leading qualification for planning. Something has been of tremendous help to us recently: I urge every business person and IT person, management or staff, to get hold of a copy of "I.T. Wars: Managing the Business-Technology Weave in the New Millennium." Our CEO has read it, our project managers are on their second reading. Our vendors are required to read it (they can borrow our copies if they don't want to purchase it). Any agencies that wish to partner with us: We ask that they read it. Do yourself a favor and read this book - then ask your boss to read it - then ask your staff and co-workers to read it. If nothing else, read this interview - quite interesting, and revelatory: www.businessforum.com/DScott_02.html

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

11 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT