BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles The Most Common Reasons Why Software Projects Fail

The Most Common Reasons Why Software Projects Fail

Bookmarks

 

When launching a new software project, best practices suggest enlisting the help of a subject matter expert, who is knowledgeable about software development and can assist in the early stages of project planning. This strategy has proven to greatly improve the project outcome, yet at the end of the project you’re staring at a failure. How did this happen?

Project failure can be defined as one or a combination of cost overruns, late deliveries, poor quality, and/or developing a product that does not get used. Regardless of their involvement during the planning stages, more often than not, software developers bear the brunt of the responsibility for such situations; after all, they’re the ones who built the application. However, closer examinations of the projects do not always show evidence of incompetence.

When assessing these failed projects some of these perform “reasonably” when compared with industry trends, yet to the organization they are deemed failures. The reason is that overwhelming majority of the problems can be tied to flawed estimation or poor business decision making at the very outset of the project.

To avoid this, first organizations need to use a standardized set of estimation terms. We often find that individuals and organizations use a number of key terms interchangeably when they each have a unique meaning.

  • Target – A goal, what we would like to do or achieve.
  • Constraint – Some internal or external limitation on what we are able to do.
  • Estimate – A technical calculation of what we might be able to do at a defined level of scope, cost, schedule, staff, and probability.
  • Commitment – A business decision made by selecting one estimate scenario and assigning appropriate resources to meet a target within a set of constraints.
  • Plan – A set of project tasks and activities that will give us some probability of meeting a commitment at a defined level of scope, budget, schedule, and staff.

Being clear on these definitions ensures projects get off on the right foot with realistic targets and an understanding of the project’s constraints. Not doing so can send a project on a death march from the start due to one of more of the following factors.

1. Accepting a forced schedule or mandated completion/ milestone dates without substantial data and analysis.

Someone in your organization publicly speculates that the project will be done by a particular date, thus unintentionally commits the team to that deadline. Perhaps your budget cycle dictates that the money allocated to this project must be spent by the end of the year or the next release will not get funded. Maybe the stakeholder wants the project finished by Christmas so that he/she can enjoy the holiday in peace, knowing that the project is complete. Or maybe the stakeholder just really likes round numbers and wants the project to release the 1st of the month. There are numerous reasons why a development team would be given an arbitrary project completion deadline. The unfortunate reality of an overzealous schedule often results in overstaffing the project, the next reason why software projects fail.

2. Adding excessive personnel to achieve unrealistic schedule compression.

How do project managers deal with an overly optimistic schedule? One common response is to staff up the project, often adding way more people than necessary to complete the project. Not only does this drastically increase the cost of a project, but it also decreases the quality. Having more people involved in the project increases opportunities for miscommunications and also makes it more challenging to integrate the different sections of code together. Additionally, Frederick Brooks (1975) would argue that “adding people to a late project only makes it later,” which makes sense. The people have to come from somewhere, often from other projects. This puts the other projects further behind schedule and requires that the new staff get caught up by veteran staff, thus decreasing productivity across the board.

3. Failing to account and adjust for requirements growth or change and making necessary adjustments to the schedule and budget forecasts.

“Wouldn’t it be great if…?” Those five words can be some of the most dreaded, especially when heard halfway through the construction portion of a project. While there is certainly a time and a place for brainstorming, those activities should take place in Phases 1 and 2. In fact, the purpose of these two phases is to determine whether or not a project is feasible, and what features the application should have. If you think of it this way, Phase 2 helps you figure out what you are building, while in Phase 3, you build whatever you decided upon in Phase 2. While some overlap may exist between the two high-level phases, by the time you’re in the Phase 3, the scope of required functionality for a given product release should be clear.

  • Requirements growth is a problem if you’re adding functionality without allotting more time and budget for development. In essence, you’re asking for more work in a shorter amount of time, a technique that we’ve already found does not work.
  • Changes to existing requirements can also be a problem depending on the nature and timing of the change. Projects using Agile methods can manage changes to requirements details, as long as they occur before construction of a given iteration. However, any changes to requirements of software architecture that cause rework of code already written will almost certainly have an impact to the schedule and budget.

4. Emotional or “intuition-based” stakeholder negotiation that ignores facts and statistics.

At some point or another we’ve all become attached to a particular project that we’ve worked on and become emotionally invested in its outcome. For many, this may be the project where your reputation is on the line; the one that is too-big-to-fail, and often we let our emotions get the best of us.

When the pending success or failure of a software project puts an individual’s career on the line, it’s likely that any related business decisions will be impacted. Stress can cloud one’s thinking, especially when the stakes are high. A stakeholder may call for a 12-month schedule in order to impress a customer despite the fact that reports from previous projects of similar sizes all show a 15 month lifecycle. The stakeholder may dismiss advice from team members, stating that he has “a feeling” that the team will be able to pull through. In such situations, following your gut could be extremely detrimental and potentially lead to project failures.

5. False, but common belief that the proverbial IT silver bullet alone can solve project throughput or process issues.

When all else fails, a common next approach is a change of strategy. “We must be doing it all wrong” and “What are our competitors doing?” are common thoughts at this point. This is when musings of an “IT silver bullet” may start circulating around the office. For instance, someone may suggest that the organization needs to adopt the latest bandwagon development approach. While that might be a great direction in which to take the organization, a decision like this should not be taken lightly. Whichever development methodology your organization decides to use, it’s only as good as your implementation. Merely switching development methodologies is not enough to transform your development operation. Regardless of what you decide, in order to make the implementation successful, there needs to be buy-in from all sides, individuals need to be trained, and everyone needs to be held accountable to the same standards. Otherwise, your development strategy will be roughly equivalent to waiting for the stars to align perfectly and hoping for a miracle each time you begin a project. If implemented without thought, this strategy is not only incredibly risky, but also reduces opportunities for team members to give feedback mid-project.

In short, software projects fail for a number of reasons. Take a moment to reflect on whether any of the above reasons may have been the cause of a project failure in your organization. Now what can you do about it? As an executive leader you can do a lot, but it will require some courage to support your development operation. They need to function within a reasonable set of expectations. Obviously, they will still have to be held accountable, so you will also need historic performance data as evidence of their capability.

You will need to collect data on several different metrics, including the project’s schedule duration, the effort expended, the scope of the work performed, the project’s overall quality, and the productivity level achieved. In this case, productivity (PI) was measured in index points, a calculated proprietary QSM unit which ranges from 0.1 to 40. It allows for meaningful comparisons to be made between projects and accounts for software development factors, including variables such as: management influence, development methods, tools, experience levels and application type complexity.

Once a repository of an organization’s completed projects has been established, custom trend lines can be calculated to use for creating the baseline. These trend lines serve as a reference point that can be used for comparing projects within your organization. Where your projects fall relative to the trend line will indicate better or worse performance than the average. This will give insight into your organization’s current capabilities.

Figure 1. Company project portfolio plotted against their baseline productivity trend.

We can learn a lot about what we do well and what we can improve upon from looking at our projects relative to the baseline. Examining how far a project deviates from the various trends can help isolate best- or worst-in-class performances. Project outliers can also provide great insight into this. Figure 1 displays the company project portfolio against their baseline average productivity. Two of the projects stand out since they fall outside two standard deviations, one above the average and one below respectively. Further examining the factors that influenced these projects (i.e. new technology, tools and methods, personnel, or project complexity) will help shed some light on why these projects performed so well or so poorly. Mimicking what went well for best-in-class projects and avoiding what didn’t for the worst-in-class projects can help improve the performance of future projects.

Understanding the baseline for your current development operation can help set reasonable expectations for future projects by showing what has been accomplished in the past. If the desired project parameters push the estimate into uncharted territory, you can use the historical baseline to negotiate for something more reasonable. This baseline can also be used for contract negotiation, evaluating bids and vendor performance, and navigating customer constraints, thus allowing you to achieve your cost reduction and process improvement goals.

Doing this will put you in a much more powerful position to negotiate with customers and business stakeholders. The ideal outcome is one in which every entity can be successful. In all likelihood this will require some compromise, but through this process you will be sowing the seeds of success.

About the Authors

C. Taylor Putnam-Majarian is a Consulting Analyst at QSM and has over seven years of specialized data analysis, testing, and research experience.  In addition to providing consulting support in software estimation and benchmarking engagements to clients from both the commercial and government sectors, Taylor has authored numerous publications about Agile development, software estimation, and process improvement, and is a regular blog contributor for QSM.  Most recently, Taylor presented research titled Does Agile Scale? A Quantitative Look at Agile Projects at the 2014 Agile in Government conference in Washington, DC. Taylor holds a bachelor’s degree from Dickinson College.

Doug Putnam is Co-CEO for Quantitative Software Management (QSM) Inc. He has 35 years of experience in the software measurement field and is considered a pioneer in the development of this industry. Mr. Putnam has been instrumental in directing the development of the industry leading SLIM Suite of software estimation and measurement tools, and is a sought after international author, speaker and consultant. His responsibilities include managing and delivery of QSM software measurement services, defining requirements for the SLIM Product Suite and overseeing the research activities derived from the QSM benchmark database.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Bad Programmers

    by Marc Clifton,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I've rarely encountered the above reasons. The #1 reason why I see projects fail is very simple: unskilled, untrained, inexperienced programmers.

  • Re: Bad Programmers

    by Bin Wu,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    do you really mean programmers not project managers?

  • Re: Bad Programmers

    by Graham Pickard,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I have never encountered a project that has failed due to bad programmers.
    However the following have been observed.

    Requirements creep.
    Poor timescale estimation
    Inflexibility of the plan.
    "Requirements" that don't actually capture the requirements.
    Expecting the design to capture requirements.
    Misunderstanding of the hardware and it's limitations.
    Using a design methodology without correct training.
    Using a programming language without correct training.
    Big bang delivery
    Lack of communication with customer.
    Lack of communication within the team.
    Relying on a Super Programmer
    Relying on a contractor
    etc etc etc

  • Re: Bad Programmers

    by Bin Wu,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    @Graham Pickard, I completely agree with you...

  • BAD PROGRAMMERS

    by Francis F. Amunde,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Though, training is of essence, but i think communication with the customer/client, team players, and most importantly the business needs both at the present at future can not be left out. Above all, Professionals/Practitioners must take into account the evolving pace of technology and keep on updating themselves as needed.

  • BAD PROGRAMMERS

    by Francis F. Amunde,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Though, training is of essence, but i think communication with the customer/client, team players, and most importantly the business needs both at the present at future can not be left out. Above all, Professionals/Practitioners must take into account the evolving pace of technology and keep on updating themselves as needed.

  • Re: Bad Programmers

    by Travis Spencer,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Perhaps in your situation you've seen bad programmers. I've seen all of the above reasons and they had nothing to do with the skill level (good or bad) of the programmers.

    I'm not saying your wrong, I'm just saying that "bad programmers" aren't the only reason projects fail. Even good programmers can't compensate for the issues listed above.

  • Bad I.T. Management

    by Marco Ramirez,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Usually the reasons projects fails, it's due to "Bad I.T. Management".

    Sometimes, those Managers are bad Programmers, sometimes don't.

    Sometimes, those Managers are people without any I.T. skills, that make I.T. decisions.

    Sometimes, the Managers recruit or allow othes, to recruit bad Programmers,
    or Programmers that may have some skills in one area, and fail, in another,
    yet it's their responsability.

  • Bad I.T. Management

    by Marco Ramirez,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Usually the reasons projects fails, it's due to "Bad I.T. Management".

    Sometimes, those Managers are bad Programmers, sometimes don't.

    Sometimes, those Managers are people without any I.T. skills, that make I.T. decisions.

    Sometimes, the Managers recruit or allow othes, to recruit bad Programmers,
    or Programmers that may have some skills in one area, and fail, in another,
    yet it's their responsability.

  • What's New Here?

    by Daniel Toczala,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The article is basically stating what software development professionals have known for the past 10-20 years. It outlines some of the major reasons/drivers for the Agile and DevOps movements.
    What the article fails to do is to highlight the main theme that runs through the 5 areas identified. The main theme that is not highlighted is that in order to have a successful project, people need to make informed decisions up front (is it the right thing to build? What is the estimated cost?), and then continue to revisit these decisions based on transparent metrics coming from the development team.
    If a project ends up being more technically difficult than first estimated, and that the additional time/effort/cost invalidates the business model, then doesn't it make more sense to kill it once you discover this? It's nobody's fault - it's just that your organization has learned more, and has better information. No need to fire the developers, the PM's, or the Executives. Just lick your wounds and move on to something more productive.

  • What's New Here?

    by Daniel Toczala,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The article is basically stating what software development professionals have known for the past 10-20 years. It outlines some of the major reasons/drivers for the Agile and DevOps movements.
    What the article fails to do is to highlight the main theme that runs through the 5 areas identified. The main theme that is not highlighted is that in order to have a successful project, people need to make informed decisions up front (is it the right thing to build? What is the estimated cost?), and then continue to revisit these decisions based on transparent metrics coming from the development team.
    If a project ends up being more technically difficult than first estimated, and that the additional time/effort/cost invalidates the business model, then doesn't it make more sense to kill it once you discover this? It's nobody's fault - it's just that your organization has learned more, and has better information. No need to fire the developers, the PM's, or the Executives. Just lick your wounds and move on to something more productive.

  • Re: Bad Programmers

    by Arockiaraj Durairaj,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    agree

  • g

    by tctec ptyltd,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    ..

  • Fail early.

    by tctec ptyltd,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It is my belief that a percentage , more than half, of new software projects will fail.
    Regardless of management, technical expertise, and planning.

    The failure can't be predicted or mitigated.

    I believe a new product (software application) is an NP-Hard problem,.. i.e. 'The traveling salesman problem'.

    The traveling salesman wishes to find the shortest route between many points.
    The shortest route is the product, an ideal fit for the application.

    We cannot know the best fit without iteration (measuring a route and comparing with the others), trial , error, modification.

    Attempting to bypass the unavoidable cycle of Test, failure, Iteration, with up-front planning can be a waste of money.
    I believe a good approach is to make modification/test cycles as short as possible.
    With a plan and budget only to the next milestone no more than a few weeks away.
    To fail early, and reassess regularly.

    Even if the project is huge:
    If we only have a week to deliver a first test;
    Features are teased-out and identified by priority.
    The 'nice-to haves', are set aside for later.
    The project scales up as a set of independent features from most important to least important and problems are discovered early..

  • Re: What's New Here?

    by Taylor Putnam-Majarian,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Daniel -- Thank you for your insightful comment. When I originally wrote this article, it was intended to be a part of a much larger work that addresses using estimation to help make better business decisions. Some of that gets lost in the stand-alone format, so thank you for summarizing the common theme! You're absolutely right that the key to project success is informed decision-making. At QSM, we believe that the ultimate purpose for estimation is to help organizations make better decisions.
    What makes an estimate most valuable is its degree of usefulness to the organization. If an estimate can identify high risk scenarios before they happen, then that's much more valuable than having precision to three decimal places. Like you said, if the project's actual performance deviates from the estimate, due to requirements growth/ lower productivity levels/ etc., what this really tells us is that we need to modify the parameters from which we based our estimate. In the end, project failures provide a wealth of information on how to run projects more successfully in the future.

  • Re: What's New Here?

    by Ben Linders,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I agree with you Daniel that you should kill a project, or simply stop and release what you have developed so far, when there is no business case to continue.

    But I don't believe that you can make informed decisions up front. You can make some, but other have to wait until you've learned from your customers using your product. That's one of the main benefits of delivering iterative with agile.

  • PMs' mistakes

    by James Allan,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It is usually considered that a project fails due to a project manager’s inability to lead their team properly, rather than team members who couldn’t do their job right. Some of the most common mistakes that project managers make are related to lack of communication (rare meetings and update follow-ups), not enough transparency on a project, unrealistic deadlines, and other. The following article shows some of the most common mistakes are that PMs do and it also gives tips on how to fix them - www.bgosoftware.com/blog/8-common-mistakes-proj...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT