Author and consultant Scott W. Ambler has conduct an annual survey examining IT project outcomes since 2007. He makes the survey responses and all his evaluation freely available so others can conduct their own analysis of the content.
He summarised the 2011 survey results in a Dr Dobbs article titled "How Successful are IT Projects, Really?".
He explored the impact of "development paradigm" on the outcome of projects, with a view to identifying which approaches work best. He describes the five approaches he examined as:
On an ad hoc software development project, the team does not follow a defined process.
On an iterative software development project, the team follows a process that is organized into periods often referred to as iterations or time boxes. On any given day of the project, team members may be gathering requirements, doing design, writing code, testing, and so on. An example of an iterative process is RUP.
On an agile software development project, the team follows an iterative process that is lightweight, highly collaborative, self organizing, and quality focused. Example of an agile process is OpenUP, Scrum, or XP.
On a traditional software development project, the team follows a staged process where the requirements are first identified, then the architecture/design is defined, then the coding occurs, then testing, then deployment. Traditional processes are often referred to as "waterfall" or simply "serial" processes.
Lean is a label applied to a customer value-focused mindset/philosophy. A lean process continuously strives to optimize value to the end customer, while minimizing waste that may be measured in terms of time, quality, and cost. Ultimately, the lean journey is the development of a learning organization. Examples of lean methods/processes include Kanban and Scrumban.
When analysing the results he used the following definitions for successful, challenged and failed projects:
In the case of this survey, a project was considered successful if a solution was delivered and it met its success criteria within a range acceptable to the organization. A project was considered challenged if a solution was delivered, but the team did not fully meet all the project's success criteria within acceptable ranges (e.g., the quality was fine, the project was pretty much on time, but ROI was too low). A project was considered a failure if the project team did not deliver a solution at all. In future surveys, I think I will rework the failed definition to exclude projects that were purposefully cancelled with the full agreement of the stakeholders.
Using these definitions he found the following results:
- Iterative and agile approaches had statistically the same success rates, with 69% of iterative projects being successful and 25% challenged; whereas 67% of agile projects were considered successful and 27% challenged.
- Traditional and ad hoc teams also statistically had the same levels of success, which in turn, were statistically lower than those enjoyed by agile or iterative teams. In this case, traditional teams were 50% successful and 36% challenged, and ad hoc teams were 49% successful and 38% challenged.
- 62% of lean projects were considered successful and 30% challenged, putting it dead center between the agile/iterative and traditional/ad hoc pairings.
The article goes on to examine the relationship between various factors that contribute to project success and the approach used. He looks at success as a multi-dimensional construct:
- Time/schedule: 20% prefer to deliver on time according to the schedule, 26% prefer to deliver when the system is ready to be shipped, and 51% say both are equally important.
- Return on investment (ROI): 15% prefer to deliver within budget, 60% prefer to provide good return on investment (ROI), and 25% say both are equally important.
- Stakeholder value: 4% prefer to build the system to specification, 80% prefer to meet the actual needs of stakeholders, and 16% say both are equally important.
- Quality: 4% prefer to deliver on time and on budget, 57% prefer to deliver high-quality systems that are easy to maintain, and 40% say both are equally important.
He compares the outcome of each dimension to the development approach used, and finally concludes:
Once again, this survey has shown that agile and iterative software development methods appear to be more successful than either traditional or ad hoc strategies. This is consistent with previous surveys and is likely part of the explanation for the general trend in the IT community toward adopting agile strategies
Ambler is definitely not the only one conducting project success surveys. The most commonly cited is the Standish Group Chaos Survey, which uses a much more restrictive definition of success as on-time, on-budget and on-scope. The latest Standish Group Chaos Report was released in March 2011. Standish chairman Jim Johnson stated
"This year's results represent the highest success rate in the history of the CHAOS Research, We clearly are entering a new understanding of why projects succeed or fail."
Is Ambler's definition of success valid, and do his results reflect the reality in your organisation?
Participation in this survey is self selected - does that influence the results?
/filters:no_upscale()/sponsorship/topic/2c0d72d2-1768-47c2-8f5b-24f3ef0ae0e0/QconNYrsb-1679560094156.png)
/articles/rebuild-tech-culture-layoffs/en/smallimage/how-to-rebuild-tech-culture-for-those-who-survived-the-layoffs-small-1677236107330.jpg)
/articles/secure-delivery-workflow/en/smallimage/logo-1675435091128.jpg){kind=logo}
/articles/energy-consumption-bugs/en/smallimage/logo-1673281703209.jpg){kind=logo}
/articles/motivational-diagnostic/en/smallimage/increasing-team-and-individual-motivation-with-motivational-diagnostic-small-1672932888794.jpg)
/articles/group-coaching-beyond-individual/en/smallimage/group-coaching-extending-growth-opportunity-beyond-individual-coaching-small-1671614956735.jpg)
/articles/platform-engineering-roadmap/en/smallimage/logo-1670836548184.jpg){kind=logo}
/articles/software-testing-energy-management/en/smallimage/logo-1670618011406.jpg){kind=logo}
/articles/product-mindset-devops/en/smallimage/logo-1668102246270.jpg){kind=logo}
/articles/remote-agile-governance/en/smallimage/using-remote-agile-governance-to-create-the-culture-organisations-need-small-1666608335110.jpg)
/articles/secure-software-development-gitops/en/smallimage/accelerating-secure-software-delivery-lifecycle-with-GitOps-small-1679658720410.jpg)
/presentations/refactoring-rust/en/smallimage/qcon-plus-s-1679694299933.jpeg)
/presentations/aws-apache-airflow/en/smallimage/qcon-plus-s-1679694143127.jpeg)
/articles/billions-messages-minute/en/smallimage/LOGO-1679471830613.jpg){kind=logo}
/articles/ci-cd-observability/en/smallimage/logo-1679470476672.jpg){kind=logo}
/articles/visual-thinking/en/smallimage/LOGO-1679339789328.jpg){kind=logo}
/articles/framework-architectural-decisions/en/smallimage/logo-1679060242842.jpg){kind=logo}
Community comments
Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is better
by Udayan Banerjee,
Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is better
by Udayan Banerjee,
Re: Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is be
by Rulle Rask,
Re: Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is be
by Mark Levison,
Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is better
by Udayan Banerjee,
Your message is awaiting moderation. Thank you for participating in the discussion.
However, closer look at the 3 surveys (2008, 2010 & 2011) shows 2 interesting facts:
1. The gap between “Iterative + Agile” and “Traditional + Ad-hoc” has been increasing
2. Iterative approach has consistently performed better than Agile
setandbma.wordpress.com/2012/01/03/agile-projec...
Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is better
by Udayan Banerjee,
Your message is awaiting moderation. Thank you for participating in the discussion.
However, closer look at the 3 surveys (2008, 2010 & 2011) shows 2 interesting facts:
1. The gap between “Iterative + Agile” and “Traditional + Ad-hoc” has been increasing
2. Iterative approach has consistently performed better than Agile
setandbma.wordpress.com/2012/01/03/agile-projec...
Re: Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is be
by Rulle Rask,
Your message is awaiting moderation. Thank you for participating in the discussion.
Worth to mention is also the criticism of the Standish group concerning the description of how they conducted their studies, and the difficulty to evaluate the validity of their 1994 study. See simula.no/research/se/publications/Jorgensen.20...
It would be interesting to see if later studies, including Ambler's study, survive a similar evaluation.
Re: Scott Ambler Survey 2008, 2010 & 2011 shows Agile / Iterative is be
by Mark Levison,
Your message is awaiting moderation. Thank you for participating in the discussion.
Observe that Scott doesn't build survey's for a living and so his data should probably be taken with at least two grains of salt. Nonetheless from my personal experience in the field Scott's numbers do reflect the trend. Agile/Iterative projects tend to better.
Cheers
Mark Levison
Certified Scrum Trainer | Agile Pain Relief Consulting