If Agile is So Good, Why Isn't Everyone Doing It?

CIOs of most Fortune 500 companies don't know their Agile from their as*. Instead they rely on mass appeal propaganda middle managers to find what works in their organization without upsetting the "delicate balance" of the large dysfunctional machine. The trick for most new adoptions in large companies is to push an innovative idea or methodology like Agile from the trenches up to buy in from the middle managers. Then there's pressure to successfully implement the idea/methodology for a project despite several other existing inefficiencies and bloat.

I see your point of view Shane, however, just think, what a wonderful world it would be if CIO's were Agile evangelists - and there are some - and dream of the quality software they produce. Take ThoughtWorks for example.....

In my opinion, much of the agile community focuses on stuff at the project level. This is important (obviously), but it completely ignores how to introduce agile processes and make them work at the program or product level. You may criticize those stupid CIOs all you like, but it's easy to call the game from the cheap seats.

Changing a corporate culture is hard. Identifying the right communities to build and then building them is hard. Figuring out how to get multiple agile teams at multiple levels working together and with the product and sales and marketing teams in a sane way is hard.

Everyone likes to bag on Dilbert-esque pointy haired bosses, but my experience in organizations transitioning to agile is that more often than not, the upper-level managers get it. Many like the idea of planning only as much as needed, being able to more easily change priorities, and having better visibility into the *real* progress being delivered. Frankly, most of the resistance I get is from the trenches - sub-par developers and project-managers who can no longer hide behind organizational inefficiencies.

In my opinion, much of the agile community focuses on stuff at the project level. This is important (obviously), but it completely ignores how to introduce agile processes and make them work at the program or product level. You may criticize those stupid CIOs all you like, but it's easy to call the game from the cheap seats.

Agreed.

One aspect that has bitten me on occasion is the aspect that while adoption of Agile practices may be a strategic move, it's not strategic in scope; meaning Agile methods are tactical in nature. Unfortunately this seems to provide some impedance mismatch. A CIO is (should be) planning long-term, and helping set the strategic goals of an organization.

While adopting Agile may be a strategic goal - it is implemented at a tactical level. And adopting it tactically can be at odds with other strategic goals. How do we communicate with our partners what our product will be capable of one year from now if the requirements are not fixed? How will our tech docs people start to produce the documentation web site and feed the marketing people with materials - do they all switch to Agile as well (doubtful)? If they don't - how does the portion of the organization (IT) that is Agile interface with the portions that aren't?

How does one "black box" Agile - so that it can continue to interface with the rest of the organization and external partners in traditional ways? Can that even be done?

All this non sense about agile is stupid. What it comes down to is good developers. Get good developers and the methodology largely doesn't matter. It's the retard managers who have zero clue about software development that feel they need some "magic" methodology to make it work. This is usually because they instill hatred in developers and have a hard time keeping the talent. Good managers be it middle or C level, can keep the talent and maintain a competative edge over their competitors. People hawking agile or some magical methodology don't have any secrets. Just use common sense and be practical.

peter

Like most of the things in our industry there are so many factors that make one's point of view unique and worth listening. Rather than judging, to try to understand seems more positive as an attitude.

I work in a technology company. There are more than 150 of us and 95% of this number is directly involved with development. We don't have an army of salespeople. We don't have layers and layers of management. So it is not surprising that our agile process has been working marvelously well. It started with Java and now we're implementing it with our .NET projects as well.

My point is, to appreciate how being agile benefits a development team, to see how a process shouldn't get in the way of the developers, one has to have his hands dirty. One has to experience the difference between a process that breaks a developer's coding flow versus a code-centric approach. So I understand how agile development may look different to someone who has a 30,000-foot view on the code that's being produced. I think that's the case with the CIOs in question.

Not everybody's doing it, because it exposes people's and team's deficiencies.

It is much easier and more comfortable to organise meetings and company-paid pub discussions, and talk babble through six months of "planning and design" before moving on to screw up another project and leaving a bunch of poor sods to try and make sense of your copied-and-pasted from another failed project design documentation.

It is much easier for a developer to sit around waiting for the great minds in the ivory tower to come up with their cunning design, which they'll have to work around later, than actually do something.

It is called procrastination. Let's see how long we can get away with talking about things, without actually doing something. It will go on for as long as people are willing to pay for it.

Interestingly, it is the richest, the happiest and most successful companies that adopt the agile style of development, not the ones where every other email is about cost-cutting.

How many companies do you think have ANY defined process at all?

All you need to do is attend a conference and talk with a few people at random to understand that, for a significant chunk of the developer world, the process is called "ad-hoc". They're not jumping on the agile bandwagon, or any bandwagon for that matter, because they aren't even aware of the problem.

I still run into people who work at shops that don't have revision control systems in place, for god's sake.

All this non sense about agile is stupid.

What it comes down to is good developers. Get good developers and the methodology largely doesn't matter.

It's the retard managers who have zero clue about software development that feel they need some "magic" methodology to make it work.

People hawking agile or some magical methodology don't have any secrets. Just use common sense and be practical.

These are good questions. I also hear them in my coaching work.

It's interesting that Toyota, rather than "black boxing" their new way of working, instead put effort into coaching their partners and suppliers, when their "next" bottlenecks turned up outside their own organization.

According to the 2006 “State of Agile Development” survey by The Agile Alliance and Version­One, respondents said only 29 percent of traditional projects were “somewhat successful” or “very successful.” (Conversely, respondents said 81 percent of their agile projects achieved that level of success.)

The Standish Group, which famously compiles its Chaos data on software project failures, reported in its 2006 research that just 16 percent of waterfall projects succeeded as opposed to 41 percent of agile projects.

> The managers on this project aren't open to a transition to agile because what we are doing now works.

This is entirely appropriate, isn't it?

Change isn't easy: how would you get buy-in from the many people affected by a methodology change, if there is no benefit to them? (i.e. the change will not solve any perceived problem or provide a preceived advantage).

One reason for the resistance to agile maybe that agile proponents have oversold their wares. The statistics above bear that out. The hype surrounding agile is obvious and I think that contributes to management avoiding the move to agile.

There's a rule about surveys that says that a survey taken by someone with an interest in the results is likely to be skewed in the direction they favor. I think that explains the difference and so I take the Agile Alliance results with a grain of salt. I have to wonder how well their survey was constructed and conducted.

Matt,

It isn't FUD that taints survey results. It's having an interest in the outcome of the survey you are designing or taking. And why does the Agile Alliance know when a project used Agile any more or less than Standish?

The fact that there is a factor of two between the two surveys when it comes to Agile projects shows that one or both of these surveys is bad, or that they were measuring different things.

According to Johnson, Standish has been careful about what they counted as "Agile" - for this reason the 2002 and 2004 surveys' data on Agile was not reported... they were not sure answers could be clearly interpreted.

From the August 2006 InfoQ interview:

InfoQ: Have you collected any data in relation to Agile in your research?

Jim Johnson: Yes, we've tried that. We hope we can show some of that. We've been putting it in our surveys, but, not everyone fills it out - it's difficult to get clean data on that. We started to try 2 surveys back... hopefully this time we can. In the next few days we'll be doing a last push to SURF members, to close the 2006 survey.

If they reported on Agile in their 2006 stats, it suggests to me that they've adjusted their questions and collected enough data to report (what they consider) reliable stats on Agile project failure/success.

Remember, Standish's data comes from a huge database - over 50,000 projects in the past 14 years... and they survey the same organizations over the years, looking at multiple projects, year after year. I can't imagine the VersionOne sample was comparable, I can't find information on that at the moment.

It would be interesting to know how Standish distinguished Agile projects from traditional ones, wouldn't it? Let's see what InfoQ can do to find out... :-) deb

Good stuff, Deb.

The factor of two difference in the agile success rates between the Standish data and the Agile Alliance data are a red flag. I have to wonder if the Agile Alliance data reflects the "projects that are truly agile don't fail" mentality of some agile proponents and therefore led them to consider some failed projects to be non-agile just because they failed.

The "Agile projects don't fail" mantra leads very quickly to a circular definition.

I think that's a good point. I think metrics in real software projects must be really hard to do. How *do* you measure whether a project is agile? Someone just labelling the project agile is certainly not enough. In that kind of scenario, I can understand people saying, yes it failed, but it wasn't really agile. Nothing is ever perfect, but I imagine it must be possible to define fairly clear criteria for determining whether a project is agile or not, at least for the purposes of publication. After defininf agile, we also need to define success - again for the purpose of objective evaluation, I think it ought to be possible to define this, even if it won't ever be perfect. Some combination of how well received it was on completion and how close it came to the original budget and schedule ought to work.

Version One sample details, from the last page of the "State of Agile" survey:

722 individuals responded (though it wasn't recorded how many organizations or projects this represented).

Role breakdown:
20% Project Manager
12% Developer
12% Team Lead
11% Consultant
10% VP / Director of Development

Has your company adopted Agile development practices anywhere within your software organization? 84% Yes, 16% No.

> how well received it was on completion and how close it came to the original budget and schedule ought to work.

Some observations on this:

-> "how well received" would replace the typical "scope" question (since our aim is to produce what's needed as we go, not what was promised up front.

--> "how close it came to the original budget and schedule" this is tricky for Agile teams. Projects used to contain fluff requirements (features not really needed or used). With emergent requirements, this could go two ways: the team ends the project earlier, or they deliver more valuable features on the same (or a different) schedule. When they decide to deliver a different featureset on a different schedule (three short releases rather than one long project for example) how on earth do we compare plan to actual delivery schedule? Of course, it can be done for a given project, with the appropriate explanations, but how do we come up with meaningful stats to compare across projects? A further complication is: over time, many Agile teams move away from "projects" to "products" - more closely reflecting the actual software lifecycle in which new features, enhancements and fixes all happen in parallel, according to priorities. This makes the "project" essentially equal the life of the product!

That's an interesting point. I do think that for the vast majority of software out there, the software can't go into production in a meaningful way until some critical set of features has been implemented. I would want to compare the time it actually takes to get to that first 'real' release vs. the original estimates. I suspect that there will always be room to argue over what constitutes the first meaninful release to production, but perhaps there is a way to pin this down enough for most stakeholders to agree.