Tracking change and innovation in the enterprise software development community
Scaling Scrum
in the enterprise?
ScrumWorks Pro
can help.
Start a trial today
Free Book Offer
SOA Demystified
$40 Value
QCon London 2010!
March 10-12, 100
Speakers, 80 sessions,
"THE" team leads and,
architects conference.
Posted by Ben Hughes on May 24, 2007
On CIO.com Thomas Wailgum wrote a comprehensive commentary on the adoption of Agile processes in large organisations, and the sometimes frosty greeting they have at the Board level, with the arrival of yet another delivery methodology. He voiced the question - "If agile development is so darn good, then why hasn’t it been universally adopted?"“To say that companies or CIOs are reluctant to embrace agile is like saying they wouldn’t take aspirin for a headache....And they’re not only not taking the aspirin, they’re banging their heads against the wall and wondering why it hurts.” - Jim Johnson, Standish Group ChairmanThe author went on to explore, using case studies and interviews with CIO's of a number Fortune 500 companies, how preconceptions of Agile have caused uncomfortable feelings in their businesses:
Misapprehensions about agile still run rampant in IT organizations. Eugene Nizker, a former financial services CIO and current consultant, ticks off the most infamous ones: Agile teams do not plan. Agile teams skip design. Agile teams do not test. Agile means no documentation.But where there is darkness there is light, and the author covered some success stories of Agile delivering to the enterprise:
It is “a culture and timing change,” Waghray (CIO Verizon Wireless, Midwest) says of the process, “but I have never gotten more accolades from a business project.”In summary, Wailgum pondered: given the broad range of evidence, why are so many CIO's cool to Agile Software Development?
Learning to be a Good Product Owner: Foundation Skills
VersionOne is recognized by Agile practitioners as the leader in Agile project management tools. Companies such as Adobe, BBC, CNN, Dow, HP, IBM, Sony and 3M have turned to VersionOne to help deliver greater value to their customers.
CIOs of most Fortune 500 companies don't know their Agile from their as*. Instead they rely on mass appeal propaganda middle managers to find what works in their organization without upsetting the "delicate balance" of the large dysfunctional machine. The trick for most new adoptions in large companies is to push an innovative idea or methodology like Agile from the trenches up to buy in from the middle managers. Then there's pressure to successfully implement the idea/methodology for a project despite several other existing inefficiencies and bloat.
I see your point of view Shane, however, just think, what a wonderful world it would be if CIO's were Agile evangelists - and there are some - and dream of the quality software they produce. Take ThoughtWorks for example.....
In my opinion, much of the agile community focuses on stuff at the project level. This is important (obviously), but it completely ignores how to introduce agile processes and make them work at the program or product level. You may criticize those stupid CIOs all you like, but it's easy to call the game from the cheap seats.
Changing a corporate culture is hard. Identifying the right communities to build and then building them is hard. Figuring out how to get multiple agile teams at multiple levels working together and with the product and sales and marketing teams in a sane way is hard.
Everyone likes to bag on Dilbert-esque pointy haired bosses, but my experience in organizations transitioning to agile is that more often than not, the upper-level managers get it. Many like the idea of planning only as much as needed, being able to more easily change priorities, and having better visibility into the *real* progress being delivered. Frankly, most of the resistance I get is from the trenches - sub-par developers and project-managers who can no longer hide behind organizational inefficiencies.
In my opinion, much of the agile community focuses on stuff at the project level. This is important (obviously), but it completely ignores how to introduce agile processes and make them work at the program or product level. You may criticize those stupid CIOs all you like, but it's easy to call the game from the cheap seats.
I don't think the question of introducing agile processes is being ignored. There is plenty of advice out there how to do it. The bottom line is that there is no silver bullet in adoption or execution of agile. People just have to do the leg work of learning and adapting. There is no free ticket.
If agile were easy, in deed, everyone would be doing it. But those who can do it, gain to benefit from it.
Agreed.
One aspect that has bitten me on occasion is the aspect that while adoption of Agile practices may be a strategic move, it's not strategic in scope; meaning Agile methods are tactical in nature. Unfortunately this seems to provide some impedance mismatch. A CIO is (should be) planning long-term, and helping set the strategic goals of an organization.
While adopting Agile may be a strategic goal - it is implemented at a tactical level. And adopting it tactically can be at odds with other strategic goals. How do we communicate with our partners what our product will be capable of one year from now if the requirements are not fixed? How will our tech docs people start to produce the documentation web site and feed the marketing people with materials - do they all switch to Agile as well (doubtful)? If they don't - how does the portion of the organization (IT) that is Agile interface with the portions that aren't?
How does one "black box" Agile - so that it can continue to interface with the rest of the organization and external partners in traditional ways? Can that even be done?
All this non sense about agile is stupid. What it comes down to is good developers. Get good developers and the methodology largely doesn't matter. It's the retard managers who have zero clue about software development that feel they need some "magic" methodology to make it work. This is usually because they instill hatred in developers and have a hard time keeping the talent. Good managers be it middle or C level, can keep the talent and maintain a competative edge over their competitors. People hawking agile or some magical methodology don't have any secrets. Just use common sense and be practical.
peter
Like most of the things in our industry there are so many factors that make one's point of view unique and worth listening. Rather than judging, to try to understand seems more positive as an attitude.
I work in a technology company. There are more than 150 of us and 95% of this number is directly involved with development. We don't have an army of salespeople. We don't have layers and layers of management. So it is not surprising that our agile process has been working marvelously well. It started with Java and now we're implementing it with our .NET projects as well.
My point is, to appreciate how being agile benefits a development team, to see how a process shouldn't get in the way of the developers, one has to have his hands dirty. One has to experience the difference between a process that breaks a developer's coding flow versus a code-centric approach. So I understand how agile development may look different to someone who has a 30,000-foot view on the code that's being produced. I think that's the case with the CIOs in question.
Not everybody's doing it, because it exposes people's and team's deficiencies.
It is much easier and more comfortable to organise meetings and company-paid pub discussions, and talk babble through six months of "planning and design" before moving on to screw up another project and leaving a bunch of poor sods to try and make sense of your copied-and-pasted from another failed project design documentation.
It is much easier for a developer to sit around waiting for the great minds in the ivory tower to come up with their cunning design, which they'll have to work around later, than actually do something.
It is called procrastination. Let's see how long we can get away with talking about things, without actually doing something. It will go on for as long as people are willing to pay for it.
Interestingly, it is the richest, the happiest and most successful companies that adopt the agile style of development, not the ones where every other email is about cost-cutting.
How many companies do you think have ANY defined process at all?
All you need to do is attend a conference and talk with a few people at random to understand that, for a significant chunk of the developer world, the process is called "ad-hoc". They're not jumping on the agile bandwagon, or any bandwagon for that matter, because they aren't even aware of the problem.
I still run into people who work at shops that don't have revision control systems in place, for god's sake.
All this non sense about agile is stupid.
Why do you feel that agile is stupid? Have you had some bad experiences? Please explain.What it comes down to is good developers. Get good developers and the methodology largely doesn't matter.
Of course good developers are essential. However, if you just had good hockey / baseball / football / basketball / soccer players on a team without a coach and system that maximizes that the players' effectiveness, what is their overall likelihood of being successful? In some rare cases they may indeed succeed. However, having a system that is tailored to the skills and personalities of the team members, and a coach that knows how to get the best out of his or her team will greatly increase the chances for success.
Do you think that Adobe has bad developers? If not, why are they singing the praises of how the introduction of agile practices helped considerably with the Creative Suite 3 development (www.infoq.com/news/2007/03/adobe-photoshop-agil...). Why are companies like Hewlett-Packard using Agile, when they have good developers? (BTW, HP is 14th on the Fortune 500).It's the retard managers who have zero clue about software development that feel they need some "magic" methodology to make it work.
First of all, 'retard' is offensive. Please don't use that term. Secondly, yes, there are managers out there like that. They shouldn't be managers. They will find a way to fail, regardless of whatever process is used. That isn't a process problem, that's a people problem.People hawking agile or some magical methodology don't have any secrets. Just use common sense and be practical.
Agile isn't magic, and there are no secrets! The practices used by the common agile processes have been around for decades. To my knowledge, no one has ever said that this is magic. With XP, for instance, I understand that when Kent Beck was faced with leading the team that worked on the Chrysler C3 project he looked back over his career at what practices had worked, and how his team could maximize the benefit of those practices. Testing is good, so do it all the time. It's hard to write tests after the fact on a large code base, so write a small test, then write some code to make it pass. Restructuring code is difficult and risky, so do it in very small steps, and use the tests you've written as a safety net. Having developers together in a war room has historically been a good thing, so why not put the whole team in one place. Having access to the Customer is good, so provide access all the time. Peer reviews of code are good, so do them continuously. Integrating code frequently is a good thing, so do it all the time in very small steps to minimize the risk of conflicts and outdated code. Working iteratively and incrementally is a good thing (and has been around since the late 50's on life-critical software), so work in very small iterations and releases. Planning is good, so do it all the time, adjusting estimates and priorities as required based on reality rather than what was known when the project started.
Do any of those points sound like magic or are they impractical?
Dave Rooney
Mayford Technologies
These are good questions. I also hear them in my coaching work.
It's interesting that Toyota, rather than "black boxing" their new way of working, instead put effort into coaching their partners and suppliers, when their "next" bottlenecks turned up outside their own organization.
According to the 2006 “State of Agile Development” survey by The Agile Alliance and VersionOne, respondents said only 29 percent of traditional projects were “somewhat successful” or “very successful.” (Conversely, respondents said 81 percent of their agile projects achieved that level of success.)
The Standish Group, which famously compiles its Chaos data on software project failures, reported in its 2006 research that just 16 percent of waterfall projects succeeded as opposed to 41 percent of agile projects.
I find these statistics interesting. The survey conducted by the Agile Alliance found the success rate for agile projects to be twice as high as the survey by Standish. There's a rule about surveys that says that a survey taken by someone with an interest in the results is likely to be skewed in the direction they favor. I think that explains the difference and so I take the Agile Alliance results with a grain of salt. I have to wonder how well their survey was constructed and conducted.
If you consider both surveys, somewhere between one in five and three in five agile projects still fail. That shows that the claim made by agile proponents like Ron Jeffries that no project that is "truly agile" has failed is simply false.
One reason for the resistance to agile maybe that agile proponents have oversold their wares. The statistics above bear that out. The hype surrounding agile is obvious and I think that contributes to management avoiding the move to agile.
As others have pointed out, there are a lot of projects that don't qualify as either waterfall or agile. The project I'm working on now has a BDUF stage followed by an agile stage. We write long documents that have to be approved before we start design and code, but after approval we get feedback, collaborate, and make changes as needed without revisiting the documents. All the docs are out of date almost immediately. It's a strange hybrid, but it works in spite of itself probably because of the quality of people on the team.
The managers on this project aren't open to a transition to agile because what we are doing now works.
> The managers on this project aren't open to a transition to agile because what we are doing now works.
This is entirely appropriate, isn't it?
Change isn't easy: how would you get buy-in from the many people affected by a methodology change, if there is no benefit to them? (i.e. the change will not solve any perceived problem or provide a preceived advantage).
One reason for the resistance to agile maybe that agile proponents have oversold their wares. The statistics above bear that out. The hype surrounding agile is obvious and I think that contributes to management avoiding the move to agile.
Right on. I'm a big proponent of agile methods, but I think any time you have a lot of people excited about something (tool, technology, process or otherwise) you run the risk of having those folks over-promise and under-deliver. I don't think there are any bad guys here; it's just human nature.
Keeping a project from going off the rails is hard. Agile just gives you some tools that help make the job easier. To Peter Lin's point, this problem goes away somewhat if you can just go find 10 outstanding developers (that is, if I've got 10 hotshot developers telling me how amazingly smart and great they are, then they had better damn well self-organize and knock my socks off). But for large organizations, that's not always a realistic proposition.
There's a rule about surveys that says that a survey taken by someone with an interest in the results is likely to be skewed in the direction they favor. I think that explains the difference and so I take the Agile Alliance results with a grain of salt. I have to wonder how well their survey was constructed and conducted.
However with all the mis-information and FUD being spread about Agile, you'd have to assume that the other survey results are just as tainted. At least with the Agile Alliance the folks they're surveying know when a project used Agile and when someone was just looking for something to point the finger at.
Matt,
It isn't FUD that taints survey results. It's having an interest in the outcome of the survey you are designing or taking. And why does the Agile Alliance know when a project used Agile any more or less than Standish?
The fact that there is a factor of two between the two surveys when it comes to Agile projects shows that one or both of these surveys is bad, or that they were measuring different things.
According to Johnson, Standish has been careful about what they counted as "Agile" - for this reason the 2002 and 2004 surveys' data on Agile was not reported... they were not sure answers could be clearly interpreted.
From the August 2006 InfoQ interview:
InfoQ: Have you collected any data in relation to Agile in your research?
Jim Johnson: Yes, we've tried that. We hope we can show some of that. We've been putting it in our surveys, but, not everyone fills it out - it's difficult to get clean data on that. We started to try 2 surveys back... hopefully this time we can. In the next few days we'll be doing a last push to SURF members, to close the 2006 survey.
If they reported on Agile in their 2006 stats, it suggests to me that they've adjusted their questions and collected enough data to report (what they consider) reliable stats on Agile project failure/success.
Remember, Standish's data comes from a huge database - over 50,000 projects in the past 14 years... and they survey the same organizations over the years, looking at multiple projects, year after year. I can't imagine the VersionOne sample was comparable, I can't find information on that at the moment.
It would be interesting to know how Standish distinguished Agile projects from traditional ones, wouldn't it? Let's see what InfoQ can do to find out... :-) deb
Good stuff, Deb.
The factor of two difference in the agile success rates between the Standish data and the Agile Alliance data are a red flag. I have to wonder if the Agile Alliance data reflects the "projects that are truly agile don't fail" mentality of some agile proponents and therefore led them to consider some failed projects to be non-agile just because they failed.
The "Agile projects don't fail" mantra leads very quickly to a circular definition.
I think that's a good point. I think metrics in real software projects must be really hard to do. How *do* you measure whether a project is agile? Someone just labelling the project agile is certainly not enough. In that kind of scenario, I can understand people saying, yes it failed, but it wasn't really agile. Nothing is ever perfect, but I imagine it must be possible to define fairly clear criteria for determining whether a project is agile or not, at least for the purposes of publication. After defininf agile, we also need to define success - again for the purpose of objective evaluation, I think it ought to be possible to define this, even if it won't ever be perfect. Some combination of how well received it was on completion and how close it came to the original budget and schedule ought to work.
Version One sample details, from the last page of the "State of Agile" survey:
722 individuals responded (though it wasn't recorded how many organizations or projects this represented).
Role breakdown:
20% Project Manager
12% Developer
12% Team Lead
11% Consultant
10% VP / Director of Development
Has your company adopted Agile development practices anywhere within your software organization? 84% Yes, 16% No.
> how well received it was on completion and how close it came to the original budget and schedule ought to work.
Some observations on this:
-> "how well received" would replace the typical "scope" question (since our aim is to produce what's needed as we go, not what was promised up front.
--> "how close it came to the original budget and schedule" this is tricky for Agile teams. Projects used to contain fluff requirements (features not really needed or used). With emergent requirements, this could go two ways: the team ends the project earlier, or they deliver more valuable features on the same (or a different) schedule. When they decide to deliver a different featureset on a different schedule (three short releases rather than one long project for example) how on earth do we compare plan to actual delivery schedule? Of course, it can be done for a given project, with the appropriate explanations, but how do we come up with meaningful stats to compare across projects? A further complication is: over time, many Agile teams move away from "projects" to "products" - more closely reflecting the actual software lifecycle in which new features, enhancements and fixes all happen in parallel, according to priorities. This makes the "project" essentially equal the life of the product!
That's an interesting point. I do think that for the vast majority of software out there, the software can't go into production in a meaningful way until some critical set of features has been implemented. I would want to compare the time it actually takes to get to that first 'real' release vs. the original estimates. I suspect that there will always be room to argue over what constitutes the first meaninful release to production, but perhaps there is a way to pin this down enough for most stakeholders to agree.
Brian Marick takes us through a quick tour of the most important values and challenges to adopting Agile successfully (they aren't the typical challenges and values we hear in the community).
The line between development and architecture is tricky. Does it exist at all? Is an ivory tower actually needed? There's a balance in the middle, but how do you move from developer to architect?
The word 'authority' sometimes produces an allergic response in hard-line agilists. Freedom and authority – both are bad if misused and both are good if used in right spirit for a noble cause.
"Getting Started with Grails" brings you up to speed on this modern web framework. Companies as varied as LinkedIn, Wired, and Taco Bell are all using Grails. Are you ready to get started as well?
Those familiar with only ITIL V2 often scoff at the thought that ITIL could serve as a governance framework for SOA. With ITIL V3, the focus of the framework shifted towards service-orientation.
SpringSource CTO Adrian Colyer discusses AspectJ, SpringSource's dm Server and tc Server products, OSGi and Scrum.
Heroku's Adam Wiggins talks about Rails, Background Jobs, Add-Ons, Ruby, and how Heroku manages to work around Ruby's inefficiencies using Erlang and other languages.
For Grady Booch the foundation of a good architecture is patterns, SOA being just one of many patterns. In this Second Life presentation, Booch attempts to bring more clarity on what architecture is.
22 comments
Watch Thread Reply