BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

Why Agile Adoption Fails in Some Organizations

Posted by Christopher Goldsbury on Nov 04, 2009 |

Introduction

How often do you hear that a company attempting to adopt agile practices fails? This article attempts to examine and explain the often overlooked key organizational reasons that agile fails, why it isn't obvious to most of us and some potential strategies for coping with organizational impediments. The article's target audience is managers with budgetary responsibility although technical groups might also find interest.

Historical Perspective on Agile

Where did agile practices originate and why? Asking this question takes us directly to large software development efforts in companies that market and sell software as their primary revenue generator or companies that see their custom software as a primary competitive differentiator for their business....we'll call this company X. Agile did not originate on a fixed bid development effort in an IT department at company Y.

Why is this important?

The answer lies in how the development effort is viewed by company X and Y financially. In company X the software development effort is viewed as an investment, indeed the primary investment, in the company's future. In company Y the development effort is a small application and is viewed as an expense to be time bound and tracked. In company X the team is funded. In company Y the project is funded. Read those last two sentences again.

In company X agile will succeed. In company Y agile will fail.

In a fixed bid development effort the software development is intended to end at some point. Securing funding for the project requires that you define it up front, estimate it, resource it, develop it, test it, implement it, and then turn it over to support. This is company Y.

In a time and materials funding scenario the company determines it has need for a software development team as there are many projects that require development well into the future. An estimate of how big a development team they can afford is created for the budgetary time frame (1 year, 2 years), it is resourced, and then project scoping and scheduling are done. This is company X.

See the difference? In company X there will always be software development. There is no end and the team is funded with that intention. So putting that work in a backlog, prioritizing it, and estimating and reviewing it in time boxed iterations makes sense.

In company Y the effort can only afford to be funded for some subset of the budgetary period (say 3 months). After that there is no more money or the company is unwilling to allocate additional money. They don't want a long term development team because they can't afford it and besides there wouldn't be enough development work to keep them busy anyway. So deep controls and strong project management is required to ensure that something is delivered in that 3 month period.

Viewed this way.....financially, agile is a luxury. It assumes that you'll always have a software team and there will always be development work. It assumes you're team is funded year after year and you, as a manager, don't have to worry about funding individual projects. As an agile manager you're primarily concerned with schedule, scope and capacity. Budgets are an annual or bi-annual thing. You flex up or down depending on the economic realities facing the company. The criteria for success are largely centered on functionality delivered over time.

In company Y you might have $50,000.00 that is set aside to complete you're project in 3 months. Budgeting and expense tracking are critical and will determine whether the project is a success or not. A manager here gets funding on a project level at various times throughout the capital cycle. You may deliver all functionality on time and over budget, but that won't be seen as a successful project. As a manager in this company you are concerned with all 3 legs of the iron triangle. Your team is likely temporary and staffed by contractor labor. Adoption of agile in this situation is a mistake.....even superficially. Why?

Estimation

The key lies in estimation.

In an agile software team you don't estimate your work till right before you begin. And you only estimate, in the case of Scrum, the next iterations work. So how do you know how long it will take? You don't. Furthermore, you really don't care. You'll continue to deliver functionality every iteration. As soon as product management and QA say you have a good enough product; it'll be released as a production version. You might have a guess, but until the team estimates it....you really don't know long it will take.

In a fixed bid situation....your estimation needs to take place up front. The company is asking you how much it will cost to build the application because they are unwilling to fund it forever. They want it to end.......preferably sooner rather than later. Funds are limited and your project, although perhaps necessary, is not viewed as critical to the company's future. Its ROI may even be negative. Returning to the leadership of this company with the answer; “I'm not sure how long it will take....just fund the team for a year and we'll see how it goes every iteration” would be a mistake that would likely result in your dismissal.

In the 2nd scenario, if I told the team, after securing funding and hiring them, “we're using Scrum”: they'd estimate the work the next iteration. They would assume their estimates would be taken seriously and you'd give them time to complete the work as it unfolds regardless of whether their estimates fit your original project funding or not. That's only fair. Unfortunately, that puts you as the manager in the uncomfortable position of submitting budget/schedule variances and/or cutting scope when you're original estimate is proven to be inaccurate. Hence, you've failed at managing the project and therefore.......”this agile thing doesn't work”.

The mistake was to assume the company's leadership understood and was organizationally committed to Scrum and agile principles. The mere fact that they are asking you to estimate the funding you'll need to complete the project tells us otherwise. If they had asked us, “How much does it cost to fund a software development team for the next 3 years?” Then we'd be wise to approach it from an agile perspective.

The Real Problem

So, what is the real problem with agile adoption in organizations? It can be boiled down to these points:

  • Agile assumes that the company wants a long term software development effort not a short term project.
  • Agile is often assumed by company leadership to be a development process with no impact on budgeting. This is not the case.
  • Development teams assume leadership understands the implications of adopting agile at the budgetary level.

The complexity of these points can't be underweighted. Developers and development teams often have zero visibility into budgeting so they are unaware of how their agile efforts are being accounted for in monetary terms. This is evident in so many agile articles on the web. Likewise, management is often ignorant of development and specifically agile development practices. Agile adoption requires education to ease the clash and misunderstanding of these two worlds.

So what are some of the consequences of attempting to adopt agile practices on a fixed bid project...essentially laying an agile façade over the waterfall project?

Story Points

Story points are often used on agile teams to determine the complexity of the work being done. The number of points completed each iteration determines their velocity ( points per iteration ) and gives management an approximation on how much work can be completed in a given iteration.

If you come from a fixed bid shop like company Y your immediate question is, “How does this relate to hours? How do I project costs and ROI?” Truthfully, it doesn't. It isn't intended to. Again, the agilist is assuming you're funding a software development team not a software development project.

In company X you could estimate things by hamburgers or cigarettes in each iteration. It doesn't matter. You're going to get the product done at some point ( +/- functionality requested ). The only real question is at what point to do we call it complete and release it to production. Funding for the team is not contingent on estimation of effort.

In company Y project funding is directly related to estimation of effort. It is critical that we tie this to time because our cost driver is often an hourly rate. Story points have no meaning here.

ScrumMaster vs. PM

“Agile does away with the need for a project manager.” Ever heard this before? It's scary for traditional PMs and unintentionally threatening. However, it is correct. If you assume that a team is funded year after year regardless of projected effort then the needs for organizing and managing the development effort are more centered on technical leadership, task and risk management. Timelines and budgets go out the door. A ScrumMaster is sufficient, preferable for getting the job done.

However, if you're in a non-agile situation, like company Y....then traditional PM practices are not only valid, but essential to making sure the effort is kept within budgetary and schedule tolerances. In this situation the leader of the development effort is being entrusted with precious company resources that can't be wasted and needs to have the skills of a CEO-Lite.

A funded development team does change the project manager role. A fixed bid project does not.

Daily Stand Up Meetings

Agile uses daily stand up meetings for a variety of reasons: motivation, risk mitigation, status, accountability, team building, etc. It's a good idea that is equally at home on a fixed bid waterfall project. There is no reason this practice can't continue to be used, but the team on that fixed bid effort has to realize that you're not really doing agile and there is a deadline looming. You also might want to weigh the time needs. The daily stand ups should be short, but 15 minutes a day adds up when you only have 3 months of funding.

Iteration Reviews

With fully funded software teams that use agile the question of when something is done is answered incrementally. Functionality is reviewed at the end of each iteration ( say 30 days ) and evaluated for readiness for production release. Again, this is a good idea that could still be used in a fixed bid situation, but the business owners need to be coached to understand the iteration review as a risk mitigation and accountability technique and not a demonstration of the completed product.

Iteration Planning

Iteration planning really does assume you're using agile in a team funded scenario. It necessitates your costs are known and fixed for the budgetary cycle and that any estimates the team comes up with won't play havoc with your budget. Doing iteration planning in a fixed bid situation will almost certainly result in confusion, budget variances, and/or loss of functionality.

Burn Down Charts

Burn down charts show the progress of completing functionality in a given iteration. They are a measure of team performance over time. They do not illustrate how close the project is to completion. If we were to sum all of them up.....they might show this, but given that they are usually used strictly for the development effort of a project; this won't always be the case.

In fixed bid scenarios the question is not usually one of how much functionality the team is doing per time period. It really doesn't matter. They need to get all the functionality done within the time frame that the money allows.

So using burn down charts and iterations in a fixed bid project sends the wrong signal to the team and your customers.

Summary

Firstly I would suggest that trying to adopt agile in a fixed bid scenario/project funded situation is not recommended. Instead, as a manager, you should make the assessment of whether the company can support an agile practice/fully staffed development team through time. If the company can; then you should use agile...it works well. If the company can't....then you'd be wise to stick with traditional project management practices.

If you determine that your company has the resources and the workload to support a software development effort but isn't using agile then it comes down to convincing the leadership that agile makes sense for this effort. Put on your change management and sales hats....you'll need them.

Secondly, project manager and Scrummaster are not static roles and titles. In one company a PM/SM may have budgetary responsibilities. In another they may not. In some they may have direct reports in a traditional organizational structure. In another the structure may be more matrixed and the PM/SM might have no direct reports. I mention these differences because agile articles, like all writing, is written from the viewpoint of the author. Too often what worked in one situation is attributed to being a superior process or technique.......when in actuality the organization and roles fit the process and technique. Change the roles and organization and it might not work as well. Context is often missing in the agile vs. waterfall blogosphere.

Lastly, the agile debate is often likened to a philosophical war. But from my experience and vantage the confusion is largely an outgrowth of misunderstanding. Too often a technical manager hasn't thought through the business implications of adopting agile. Likewise business folks frequently misunderstand agile as being ‘some development thing' with no relevance to how they just funded the effort. I've been fortunate in my career to walk across the bridge of misunderstanding and see both sides. Doing this has given me insight into the background budgetary assumptions that so frequently go unrecognized as the cause for agile adoption failure.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

What about company Z? by Artem B

Great article! It has given me some valuable insights on suitability of Agile in different situations.

But I think there is one type of company missing. An software development outsourcing company. Such a company that makes its living out of software development (like X), has stable in-staff teams (like X) but implements software projects for other companies which tend to be more like Y. What approach would be suitable in this case?

You seem to assume that non-agile will succeed for Y - why? by Richard Cohen

Nice article, persuasive reasoning. But in my experience projects are projects and work best with motivated teams. The financing and control seems to be in the realm of the product owner. A good product owner who knows what is needed and can make good priority decisions will probably do well with agile in either case.

Agile Can Be Done with Fixed-Bid Project by Robert Dempsey

I agree with many points of this article, especially that typical development teams are not provided insight into the financial aspects of their projects. I do not agree with a few things though.

"Agile assumes that the company wants a long term software development effort not a short term project."

I don't believe this to be true. I've seen agile used by software and non-software dev firms both for short-term projects where the requirements were emerging, rather than set in stone up front. If you don't know what you're going to build, or it's changing rapidly, if it's a short or long project, agile can help with this.

"Firstly I would suggest that trying to adopt agile in a fixed bid scenario/project funded situation is not recommended."

I would tend to agree with this, depending on how the scope of the project is handled. If the scope is not 100% fixed in stone (by a manager or client that thinks they have defined exactly what they want with 100% certainty), then agile can definitely work in fixed-bid, or rather fixed-budget projects. Perhaps we need to start asking, "how much can we get for $X" rather than stating, "I want all this for $Y. Make that happen."

Other Reasons for Failure.. by Sandeep S

Chris,

I agree with many points in your article. However, I think a Project that claims to "use" Agile to tempt the Godesss of Success, can fail due to other "hidden" reasons :-

1. Experts :- An Agile Project requires expertise in the applied technology - quite often, firms neglect this cardinal rule and decide to patch together a team that can "learn" on the fly.

2. People :- Agile mandates that a Team Member have a strong personality, communicate issues forcefully and boldly voice dissent against unpopular opinions. Unfortunately, there are firms out there that discourage the usage of the word "No" in all of it's manifestations - the reasons for this could vary from culture to an individual's ego.

I can enumerate many other reasons and horror stories, but my point is - there are many other reasons Agile ( or, something done with the label "Agile" ) can fail.

I think I have lit a fuse fodder for another article from you - Why Agile Adoption Fails : Part II ? ;)

Just my 2 cents.

Re: What about company Z? by Fidel Chavarria

Hi,
using the attributes of your definition of a "Z" company, to me it looks the same as the "Y" company. The company Z does not have a product to develop in a long term, it doesn't own a product(software) regardless it offer a service in a budgeted time frame. company "Z" always needs to be in schedule.

Re: Agile Can Be Done with Fixed-Bid Project by Fidel Chavarria

Perhaps we need to start asking, "how much can we get for $X" rather than stating, "I want all this for $Y. Make that happen."

you hit the nail head bro! :-D

The old story of project success being reduced to the budget by Kai-Uwe Schaefer

Nice article for creating constructive discussions. However, in some points you reduce the idea of Agile Development in an unrightful way. Agile methods - let‘s take Scrum as an „abbreviation“ - are not primarily about management but about customer expectations and team efficiency. This applies to fixed bid projects in the same way as it applies to product development, perhaps even more. Scrum does not imply a never-ending project but instead getting the most out of the development time available. It is not true to assert that in Scrum you would calculate only for the next iteration and only in vague measures. As soon as your team has a good feeling for an average velocity your predictions are pretty precise and can easily be translated into money and time. One true risk in using Scrum is when doing short-time projects (e.g., 3 months) with an unexperienced Scrum Master and a team that needs months for achieving a stable routine and forecast. By the way, how many „traditional“ fixed-bid projects do not exceed their budget in comparison to Scrum projects? Don‘t lie, you know the statistics! And I don‘t even talk about ROI. Agile methods work best with a long-term effort, that's true. But your implications are short-sighted.

Hostile Agile by tribhuwan negi

I coined this word - "Hostile Agile" for me while working on an integration product development with a big partner in a product development company. This Integrated product development happened in context of transition time in the company from a tradionally managed product development processes to Agile adoption (may be because all companies are trying it). We wanted to be “Y”  “X” soon.

I had fair amount of experience working as Scrums from my previous development work with good Scrum teams and started this new development with great Scrum Kickoffs. Things became difficult after 3-4 iterations as I had little support from the management and the Project Manager (whom I thought would be Product owner for us) had no experience in agile development environment. I started seeing huge project plans with milestones finalized in board rooms and enforced strictly due to budgetary and time lines set by partners. I wondered who created them and how without development team inputs. But they were there, and discussed for weeks in SCRUM meetings. I hardly saw any value in them for development activities.


Though the management failed to recognize the team SCRUM efforts. I was seeing very good productivity, quality and motivation in team for first few iterations, we delivered as planned. There was work rotation, daily scrum meetings, self committed and motivated team. BUT no management support. This did not last long and finally the project management control caught us. People needed visibility. They wanted daily updates. To the extent that one of the daily meeting with the project manager was called as daily SCRUM (to showcase agility in management). Which was nothing but OLD reporting meeting, renamed for being Agile. I had to spend more time in reporting and management activities rather then as a Scrum master for scrum iteration facilitation.

I tried squeezing the scrum iterations and deliveries in project plans and delivering on preplanned project deadlines to show case scrum success, and bring a change upside. We in the scrum team wanted to continue the fun of being agile.

It did not last long………… Project Control took over.
I got an email from management. “Please ensure you are present in daily SCRUM meetings for daily progress updates”. We moved back to “Y”

Conclusion: Agile is people driven; it’s driven by principles of people leadership not project management.
It’s not something you can change overnight. Tools are means to achieve it. It requires lot of un-learnining before learning these new agile principles.
It’s successful when team members are having FUN in daily development tasks and burning backlogs for potentially deliverable self committed scrum items.
It requires management commitment and suport otherwise it can not survive.

Not quite so clear cut by Iain Forsyth

Interesting and thought provoking article - but I'm not sure these are the only factors - or that the dichotomy between X and Y is so clear.

"Agile assumes that the company wants a long term software development effort not a short term project."


But isn’t agile supposed to be better at delivering quickly? Sounds like Agile expects companies to give up on cost control?

My previous employer had a disastrous project supposedly run on agile lines with a time and materials cost model, it went way over budget, cost over £1m and after 3 years was wound up without delivering anything. But the company had an in-house development team and was committed to long term software development. But it also wanted a major software change delivered within a reasonably predictable timescale and budget. The two are not manually exclusive.

The project went wrong because of poor technical leadership, a lack of requirements management and inexperienced staff. But it was the budget bleed of paying for additional contract staff on a time and materials basis that resulted in it being killed.

If Agile requires companies to write a blank cheque (or more accurately keep writing cheques indefinitely) for no assured outcome then it won’t (and shouldn’t be) adopted. But I don’t believe that’s the case.

Re: Agile Can Be Done with Fixed-Bid Project by Iain Forsyth

Perhaps we need to start asking, "how much can we get for $X" rather than stating, "I want all this for $Y. Make that happen."


Spot on - its the old time/cost/quality triangle, where quality = features. If budget and perhaps time are fixed then features have to go.

I do not agree by Juan Pablo Olguin

With all due respect your reasoning is completely flawed. You don't give any valid reason or statistics for supporting your conclusion, indeed the statistic I know supports exactly the opposite (www.ambysoft.com, ).

1 - In Scrum and any other agile methodologies you can do up-front estimating and planning when you start a project, as Kai-Uwe Schaefe well state, the difference with traditional methods is that you keep restimating and recalculating your teams velocity every iteration, you keep measuring and replanning as you go, all the times, acknolewdeging the fact that THINGS WILL and ARE GOING TO CHANGE, 100% Absolutely sure.

2 - Agiles is about being closer to the end users, it s about early and continous feedback, It is about improving and learning all along the development process so it 's always better to do agile in both kinds of organizations that you mentioned. Indeed is much more important when your budget is fixed and your schedule tight, to get the most out of the resource's shortage.

3 - The type of company x that you mentioned does not really exists exactly as you say. I mean a software company that just funds a team for making software, the budget are still allocated to individual projects and has a more o less deadline to accomplish. And yes, as you say is a good scenario to use agile as is also the company Y because agile is based on , It is worth mention again, acknowledging reality, facing the facts that the user is going to change requirement or the market or change in regulations BEFORE the projects ends.

I would suggest reading more about agile methodologies, values and practices.

Thanks you for letting me express my disagree,

Re: Agile Can Be Done with Fixed-Bid Project by Volker von Einem

I also don't see the point, why Agile shouldn't work for Fixed-Bid Projects.
To my experience there's no better methodologie for estimating and getting project work done than SCRUM.

Is there any approach that does a better job?

What I often heard as reason for SCRUM to fail is visibility in organizations and teams that do not honor this.

Re: What about company Z? by Konstantin Zagaynov

Not exactly. We are software developing company who works only by projects-funding base and we use Scrum and Agile methodologies pretty much in all the cases (so we are not of a "Y" company, rather, indeed "Z"). Only prototype/demo projects which are done by not more then 1-2 people usually and are strongly focused on fast dedicated development of features are exception.

Scrum indeed organizes and motivates and I consider it as a right investment even in small-size projects. I would agree with the points below that success of Agile projects is based on people/dedication.

Btw, there are several articles in the web how to use Agile in fixed-term projects. Check out, for example slides 27-28 of this presentation about Agile Planning: www.tinypm.com/blog/?p=59

Re: What about company Z? by Konstantin Zagaynov

Sorry, I meant presentation agileee.org/schedule/AM/. But article from tinyPM as well ;)

Worth discussing: conciliating agile development and project management by Renata Andrade de Melo

This article is valuable due to the worth discussion it raises.

When mentioning Agile, it's relevant to remember that there are different Agile Methods available, each of them fitting different purposes.

I have been successfully applying the Agile Scrum method on different Organisations/projects. I am a Scrum method supporter, certified Scrum Master but also a PMP/Prince2 Project Manager, and I do appreciate the level of maturity in the Scrum's proposal.

The main strength on Scrum is the clever and practical **conciliation of agile development and project management**.

Scrum reinforces the technical team communication, promoting a highly-productive, organised and healthy "technical brotherhood" (e.g. through the daily Scrum Meetings). At the same time, Agile Scrum cares about organising the team’s potential and efforts through an iterative approach, based on estimates, prioritisation and frequent progress and planning reviews (does it recall you of RUP?). In summary, it promotes a manageable project, defendable under the “budget and schedule” point of view.

Thanks to its maturity in recognising the relevance of project management, Scrum brings Agile to the table of discussions on large corporations, as a feasible alternative: while focusing on the technical management, it **supports and integrates with high-level management**.

I appreciated the comment posted by Kai-Uwe Schaefer, and recommend you to get in touch with Agile Scrum (you'll find great value at any of the Schwaber's books about Scrum).

Best regards,
Renata Andrade de Melo.

A case which leads the Agile to the Failure State by Roe Mjn

Agile fails when developers are not well known to the Platform they are working with because it will take heck lot of time to resolve or to complete the task which is obviously creating piles of tasks in pending status.
what do you say?
Regards,
Roe

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

16 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT