BT

Agile Hybridization - Novel Experimentation or "They just don't get it."

Posted by Christopher Goldsbury on Oct 05, 2011 |

Introduction

"They don't get it. They aren't doing it right. They aren't really agile until they follow all the principles and practices." Have you ever heard these comments before?  With the increasing hybridization of the agile world there is growing disdain for non-pure agile thinking.  Ironically, and paradoxically, agile's chief accomplishment was to get us away from a one-size fits all development philosophy in waterfall.  Now, it seems the zealous lieutenants of agile are dooming themselves to repeat the same mistake ......holding agile up as the only way.  This article will attempt to show that not only is hybridization necessary but even preferred if we as software professionals seek continued innovation and better methodologies.  

Waterfall just doesn't work

Agilists are famous for saying this and indeed they have a great deal of evidence to support their views.  But their rhetoric would probably be more accurate if it was reworded to say:  "Agile tends to work better on certain types of projects and in certain types of organizations with certain types of technologies."   It's not the case that waterfall never worked.  It just didn't work consistently on all projects.  You can find plenty of veterans in the industry to tell you they've done just fine with their waterfall process.  Are they lying?  Are they resisting change?

The truth is both techniques tend to work depending on the mix of variables. I’ll discuss some of these variables below, but there are others. My point in highlighting these is to show how context plays an important role in how a software development effort is approached and perceived:

1. Project Type

Is this a fixed bid effort or an ongoing strategic software product that will form the core of the business?  Fixed bid efforts are usually better suited to traditional waterfall process, while Scottrade would find it more advantageous to use agile. Why? In the first case the project is seen as an expense to managed, tracked and controlled. At Scottrade their online trading platform "IS" their business. A more involved explanation of this distinction can be found in one of my previous articles: Why Agile Adoption Fails in Some Organizations

2. Technologies Used

Web based technologies usually require less deployment time then more traditional heavy client applications and because browsers follow (for the most part) industry standards it’s easier to implement.  They are the optimal technology to employ with agile methodologies.  If you're deploying a heavy client product that requires installation on 16,000 laptops, desktops, with a mix of operating systems (or virtual machines) across an organization spanning 15 divisions....you're development process will be much more waterfallish.   You may still develop in time-boxed iterations and deploy your code to a TEST environment every 2 weeks, but delivering that same product to production on an iterative 2 week basis is complex and fraught with risk. It requires much more planning.  Installing something new on everyone's desktop every 30 days is not a good idea. By the time all the associated issues and problems with the first installation were resolved, one would be ready for another 30 day installation. The product would be in a constant state of flux. Delivery of functionality is important, but so is stability, consistency and performance, one has to weigh the technologies used against the intent of the product. If more change and rapid feature delivery are important does the current technology mix support it? If what’s prized is a stable, and reliable system that hardly ever goes down then reworking your entire software delivery process may not be necessary.

3.  Project Complexity

When the entire project is a new social media startup agile is a natural fit. The complexity of the project is probably contained within the developers/founders minds and their needs for information and requirements clarification exist nearby (maybe even in the same room). Their development effort will probably not require any integrated hardware. They will not interface with another division of the company with its own development/release cycle. They will probably not be integrating their software with another vended software team, and lastly there will probably not be any business process improvement effort that they need feedback and information from.  

Contrast this with a Six Sigma Black Belt business process reorganization project spanning 6 divisions of the company. The software development piece of this project may be small in respect to the entire effort. Their needs for information and requirements clarification will hinge on the "steering committee" which may not be able to meet more than once a month. There may be other vended software that they are required to interface with and pull information from (or send information to), so negotiation and timing of those integrations is crucial.

Sitting around these examples are differing levels of complexity. While polar opposites are portrayed here, the point should be clear that the higher the complexity of the project the more likely that deeper planning is required......even necessary. The context of complexity gives us pause to consider our approach.

4.  Culture

Is the company open to new ideas and ways of thinking?  Do they want to explore agile software development?  How do they adjust to change in general?  If the company is conservative then your introduction of agile may not find a strong following and may create enemies.  Caution may be required.  A more liberal organizational would probably see change as liberating and empowering. The flip side of this coin is that an organization could be too fad driven. In this culture new ways of thinking become the flavour of the month and rapidly find fertile ground before discernment and reflection have a chance to decide what’s best.

The Third Way

Sitting between agile and waterfall is pragmatism  These are the shops that have taken pieces and parts of agile and waterfall and cobbled together their own process/movement. They focus on techniques and results and skip the dogma and rhetoric.

The pragmatists are often scorned by their agile peers as "unwilling to go all the way".   I would contend, instead, that they're doing exactly what the original Agile founders did:  experiment with new ideas on how to develop and build systems.  The pragmatists see that a dogmatic approach to software development that doesn't foster flexibility, taking account of the contexts mentioned above, is doomed for failure.

Truly, new ideas come from the pragmatists.

Far from "not getting it", they are seeking out those pieces and parts that work consistently for them and maybe even for all projects.  This willingness to experiment and tinker and not follow the well worn path is invention at its purest form.  It should be encouraged and watched.  If history is any guide, something revolutionary could spawn.

Here are some pragmatic approaches I'm seeing and hearing about in the market place that seem to make sense:

1) Use pair programming on tough issues/defects or critical architectural systems.  Rather than enshrine this XP technique as the norm, pragmatists are using it as needed.  Further, I've seen this evolve into group programming where a team gets together in a room and hashes out the code and key integration pieces needed up front.

2) Layer more traditional PM practices over a Scrum approach.  Vehement agilists will tell you this is bad all the way through.  But in my experience this does have its place.  For example a project that has a heavy business process component that finds the software development project is only part of the overall effort. One may be able to get away with using scrum to develop the software, but that scrum process will need to meld tightly and neatly with the overall business project and its practices, procedures, timelines, risks, and budget. In most enterprises, scrum isn’t the methodology chosen to run a business process project.

Another example is a project with an offshore vendor. These projects can be difficult to coordinate owing to time zone differences and the limited resource capacity of the vendor. Additionally, the vendor may ask that the 15 minutes daily stand-ups, iteration review time and iteration planning time be billed in addition to the product costs. One has to weigh whether the additional communication is worth the extra premium in project costs. You can often achieve the same quality with a weekly touch base meeting and strongly defined specification. Additionally, the vendor is usually contractually obligated to deliver what the customer asked for. Doing otherwise, means no payment. The incentive to ‘get it right’ is very high in this situation and this may make a waterfall approach more palatable, even acceptable.

3) Change management in agile -  agile embraces change, but embracing any and all change can set a team up for failure.  We’ve all been part of projects where a customer will change their minds frequently throughout the development process to the point of revising a set of stories many times over even though the original requirement was developed and tested several iterations ago. This can occur for good reasons, but bad reasons for change come about too: product owner that can’t make up their mind, is not aware or accountable for a budget and feels the project is endless, or needs to "see it built" before he/she can say it’s right The scope creep in these situations can be harmful to the overall project and may find leadership doubting agile. Stronger change management techniques orchestrated by a scrum master or more traditional PM, are becoming necessary to document the progression of the system and justify budget and schedule variances.

4) User Story Manager - The agile approach is that the team and product owner put these together during a sprint planning session.  However, this often leads to poorly written stories that don't account for all the details.  Agile requirements misunderstanding is a common problem.  Designating a full-time user story writer, while expensive for a project, is worth the money invested.  The clarity provided leads to easier coding, better testing, and ultimately stronger quality.

These are just some of the things I've seen.  But, I'm interested in hearing from others.  What innovations in agile and scrum are you seeing?

Summary

The pragmatists don't see either approach, agile or waterfall, as a 'take or leave it' package.  I'd liken them to independent thinkers who see the world as a supermarket of ideas.  

"Let's use daily stand-ups, but I still need that Gantt chart to integrate with other departments." 

"I like showing our work to our clients every 30 days, but we need some kind of strong change management because this client has some truly whacky ideas."

Agile hybridization is good.  It's fostering innovation in the field.   Without this, we'll surely repeat the mistakes of the pre-agile world.  In a sense extreme agilists are right:  pragmatists aren't following the methodology correctly.  But, that's not a bad thing.

About the Author

Christopher R. Goldsbury is a software development professional who has played the roles of developer, architect, scrum master, development manager, project manager and quality assurance manager  throughout his career.  Chris writes on his experiences and ideas at his blog.

 

 

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Waterfall never worked by Serge Bureau

I have been in the software industry for over 30 years now.
I have never seen a waterfall project work, it is a flawed approach.

There is never a moderate to big projects that you can model all before coding.
I do not say you start without designing.
A real project works well when you combine top down with the bottom up approach.

Waterfall ask for total design and prediction of timing from the start. It is and has always been ludicrous.

Another aspect: Size by Adam Nemeth

I've seen that Agile just doesn't scale well in certain organizations. I've seen a company of multiple ten thousands of people, most part engineers, going fully Agile, and failing like a stoned duck.

Why Agile doesn't scale:
Communication
While Agile may work with an enclosed team of 10-20 people, it does not necessarily work when thousands have to interact. It also does not help, if you do 100 cross-functional teams out of 1500 people.

Levels of Scale in human friendships
Agile-style communication works by being nice to each other, trying to be friends. It's insanely hard to get somebody from the other side of the world, participating in a project, trying to be nice, and trying to get the information / request ASAP, just because there's no documentation

Lack of documentation
Agile doesn't document, not because it's not needed, but because it's not fun, and it isn't needed in teams of 10-20. But if you are outside of that team, it'd be easier just to read some docs. Also, I'd prefer public meeting notes over relying on gossips about the meetings you weren't on, only if they existed. They don't.

Oral history also slips better than written one.

The customer is not a user, in fact, (s)he's just your manager
Agile says nothing about users. And in an enterprise setting, nothing is said about them. A system has many stakeholders, but the customers (or, say what they are: the managers named POs) have no direct relationships in their interests. Developers don't have either this way: they fill customer (manager) needs in order to feel satisfied (if they do)

No free code

Whoever said to you, that construction is free, it's not true with enterprise developer salaries. It's just horribly expensive to do a full solution based on a specification only to realize that it's not something you wanted. Even if you don't ask to deploy it (you will, because it was horribly expensive), thinking through with paper-based idea representations tend to be cheaper.

Lack of formalism doesn't lower chaos, it ensures chaos
While it helps for 10-15 people, an army of people just simply needs some order in order to function healthy. There were written laws and etiquettes 4000 years ago for similarly-sized communities.

All in all, I came to the belief, that if more than 100 people are involved in the project development in sum, clean Agile is just not the way. This doesn't mean you shouldn't have iterative development, yet adapting RUP elements makes that possible without avoiding chaos, and having some control over the interactions.

Effective Retreat by Chris Chapman

If I understand your thesis, you're effectively arguing for holding on to a portion of the status quo of project delivery: If it's too hard to do completely with agile practices, there's no shame - and in fact, even honour - in retreating to the warm security of planning everything up front, moving work products through successive phases and dining a la carte from the agile buffet.

While there's certainly nothing stopping you from doing so, I have to ask "What's the motivation?" Why not be totally pragmatic and skip agile completely and just go all-in on waterfall? It's easier, right? Promising everything up front, hoping that things go 100% according to plan and you've padded the estimates enough to cover overages? Not bothering to surface problems or issues daily - skipping out on reviews. It's all a bit much, isn't it?

Trying to marry a defined process with empirical process practices isn't pragmatic: It's bizarre. It ignores the science behind project complexity and the rational reasons why BDUF/waterfall practices fail in favour of providing a politically-comfortable and "palatable" solution. Building software solutions is a complex, difficult effort made doubly so because of the amorphous elements that go into it: Vaguely-defined user requirements, all-intellectual effort, unpredictability, emergent answers, intermediate products that are marginal representations of thoughts involved and most importantly, people. The situation demands us to choose a strategy for delivering value against this chaos, and practical experience across tens of thousands of projects demonstrates that empirical processes trump defined.

But don't take it from me or the other irrational and impractical "zealots": It might be worthwhile to "shop" your revolutionary ideas on a pragmatic hybrid approach directly to folks like Fowler, Jeffries, Schwaber, Sutherland, Poppendieck, Beck, et. al. to get their feedback.

Re: Waterfall never worked by Jonathan Allen

I honestly don't understand why people keep presenting waterfall as a viable option. Since the 1960's there heve been nearly a dozen well known methodologies and countless lessor known ones and every single one of them has been some sort of iterative approach.

"It's not the case that waterfall never worked." Citation needed by David Hillier

"It's not the case that waterfall never worked." Citation needed

Change your company to adopt best practices. Don't change the practices or they are no longer best practice.

blog.objectmentor.com/articles/2011/05/24/bambo...

Enlightening Article by John Quincy

Just to inject a little humor into the discussion, watch the following videos:
www.youtube.com/watch?v=nvks70PD0Rs
www.youtube.com/watch?v=l1wKO3rID9g

Re: Waterfall never worked by John Russell

There is no explicit methodology called "waterfall" and there never has been. It is nothing more than a straw-man used to criticise well established and proven analysis, design, and development practices.
No "rules" are being broken by going back and changing requirements models or artefacts when something is discovered during later phases for example. Neither is there any "rule" which says you must design everything in minute detail before starting to code, or that you can't use OOAD, UI prototypes, feature backlogs (prioritised lists), or that developors can't have direct end-user contact when using a "waterfall" process.

Re: Waterfall never worked by Alex Panzin

I'll say that it's not waterfall that failed, it's the "design upfront"(architecture is not design) and deferred testing that have failed.

Design has to be fluid and documented when it is confirmed. I have never, ever, seen an upfront design that wasn't fundamentally rewritten as a result of development process and technological discovery... Hell, even simple designs like "We'll use a thread pool, message queue, XML messages and a DB" tend to be transformed during development.

Re: Waterfall never worked by Serge Bureau

I agree with you.

And unfortunately it is still very much in use today.
Many firm are still pushing for strict date prediction and full design from the start.
And yes testing mainly after coding most of the applications.
I am more a fan of functional testing than Unit testing.
But for sure I want the tests to be applied as we go, not mostly at the end.

"Waterfall" philosophy is still very much alive.
Software is not a manufacturing industry !

Good article! by E Travis

Chris,

Nicely done. I am one of those ScrumMasters who has seen waterfall work just fine over my 30 years. However, agile can and often is a better approach for certain situations. I can never understand though why when there is a project failure many people want to blame the methodology that was used and not the practitioner!

In sum, as PM professionals we need to recognize that Waterfall, Agile, Rational, Kanban, and all the other methods are tools of our trade. There is no substitute for having a clear understanding and mastery of these tools. Once this is achieved, you have a sound basis to determine when and how to use a particular tool or to borrow techniques from several to address a business situation and client needs.

Conversely, if you don't have the required expertise with a given tool you may want to be very careful about how you attempt to use it.

As for waterfall, if I have a tool that has proven to work in some situations in the past and have no idea what new situations will come up in the future, why would I toss it out of my tool bag? Having said that, I recognize that if the tool never worked right for you, then by all means toss it out.

et

E. Travis PMP

Re: Good article! - A response... by Paul Boos

There are conditions where waterfall can work; David Bland does a great job of summing this up in his post: www.scrumology.net/2011/07/19/stop-blaming-wate...

Waterfall can work well when both the requirements and solution are known upfront. For example, I need to project the ballistic path of this ball thrown at X pounds of force using the known science behind that. Waterfall can work really well here. Everything is known - the requirement, the algorithm, and you can choose the technology upfront. Agile will work here also...

Add any uncertainty though and things don't look so hot for waterfall.

I also do not subscribe to the line of thinking in this article. Agile is a set of values and principles. It is NOT a methodology. If hybridizing it still holds to the values and principles, then you are Agile; if you don't, then you aren't. Example: if your approach focuses on delivering working software often and accepting and working with the feedback, then you are Agile. If you are focused on interim deliverables and not responding to change, then you aren't.

If not Agile, then what? by Adam Nemeth

What if you have a lot of documentation, and working software?
What if you have a lot of processes, and working software?
What if your customers need only one release, but that shall be fine (like, Mars-exploration)?

The greatest problem of software development (and especially Agile) is that some developers concentrate on how to bring working software instead of actually bringing it.

Re: by Chris Goldsbury

Are the agile methodologies best practices or just an option? I'd like to think that Scrum, Evo, or Crystal are the "BEST", but is it? How do you know? Can you prove it? My view is they are a collection of tools. Follow my blog and you'll see the evolution of that point in the coming months.

Re: Enlightening Article by Chris Goldsbury

Nice...thanks. :)

Re: Waterfall never worked by Chris Goldsbury

Hmmm....you're on to something. Good thoughts. I definitely sense I've opened a debate here. My opinions & ideas usually have that effect. Dig deeper....should we move in the opposite direction from BDUF to MDUF??? :)

Re: Waterfall never worked by Chris Goldsbury

"Software is not a manufacturing industry!" I agree with this. But to the agile practitioner's credit they are borrowing ideas in search of improvement. Nothing wrong with experimentation...that's pragmatic.

With regard to testing...here's my beef: why so much testing?? What is the fundamental flaw in our tools and process today that yields such a high probability for defects? Requirements? OOD as a practice? Communication? Design? On the surface it seems like a naive question....but is it?

Re: Good article! by Chris Goldsbury

Exactly.....what we need in SD are tools. I don't care what religious stripe those techniques and tools come from ( agile, waterfall, whatever )...I just care about the tools & techniques and the result!!!! That's pragmatic and a real sale to your business owners. Watch my blog for the evolution of that thinking.

Re: Good article! - A response... by Chris Goldsbury

Yep David Bland is good guy. Like his thinking.

Re: If not Agile, then what? by Chris Goldsbury

Interesting. Are we just using process as a scape-goat for a fundamental problem in our tools to deliver software? Or is the process as much a problem as OOD?

Re: Another aspect: Size by Chris Goldsbury

Good points.

Re: Waterfall never worked by Chris Goldsbury

Never? Really? So how were all those software programs in the 70s, 80s, and 90s built then? Was that just a mirage???

Re: Effective Retreat by Chris Goldsbury

I'd love to get their feedback. Something tells me they'd agree with everything I just wrote. In fact if they're reading this now....I'd ask them to offer their opinion. Am I right or wrong? Do the original agile signatories believe that BDUF never worked or that "It has it's place."?

Re: If not Agile, then what? by Adam Nemeth

A software has resemblance of the organization which created it, that's the statistical Conway's Law[1], therefore a bad organization created bad software. Or at least, bad practices lead to bad software.

It's not the process but the values.

Re: If not Agile, then what? by Adam Nemeth

btw, what is working software?

If a startup fails to gain traction, is it working software?
If all the managers and senior devs participating are elevated, but the software is painfully slow, is it working software?
If some other software would better fit the problem, is it working software?
If users do a lot of things manually / the software isn't omnipresent for them, is it working software?

Re: Waterfall never worked by Serge Bureau

"Software is not a manufacturing industry!" I agree with this. But to the agile practitioner's credit they are borrowing ideas in search of improvement. Nothing wrong with experimentation...that's pragmatic.

With regard to testing...here's my beef: why so much testing?? What is the fundamental flaw in our tools and process today that yields such a high probability for defects? Requirements? OOD as a practice? Communication? Design? On the surface it seems like a naive question....but is it?


If you read properly I am in fact advocating less useless testing (Unit Testing)
And more useful testing (Functional), because most problems are related to timing in multithread/multicore programs.
Problems in transactions, time responses, etc... Unit testing is useless in all those cases that are the bulk of todays programming.

And I did not talk about Agile at al, which I really like. I am referring to the waterfall that unfortunately is still used in the majority of projects ????

Re: Waterfall never worked by Serge Bureau

Never? Really? So how were all those software programs in the 70s, 80s, and 90s built then? Was that just a mirage???

Find me how many where on time and on budget ? And what percentage of started projects actually where completed ?

You must be young. Nothing wrong with that, but experience tell us it was a catastrophy.

Re: Waterfall never worked by Adam Nemeth

Testing: Have you ever seen a coder who knew how to properly use UML? When I came out of university, I thought the world is full of these, then I realized, I'm nearly the only one, who understood UML-based design and uses it in practice as well. I wouldn't call what people call "design in code" as design. Therefore, as there's no design at all, the code should be tested, period.

Waterfall: the customer doesn't need incomplete functionality. If there's a tax law change, the customer isn't interested in a module which does half of the things right. Sometimes the minimal complete functionality is simply larger than a single sprint. This shall be managed well.

Once some people would have the taste to at least try to think forward, suddenly they would realize, that oops, the industry is predictive, things could be designed up-front. It takes brain only.

Once upon a time, we realized we're too young to see everything beforehand. Now we don't even try to see anything beforehand. The truth lies in the middle.

Re: Enlightening Article by Bryan Cooper

Enjoyed these videos very much. Great Post!

Agile is called agile for a reason by Kevlin Henney

A curious article. It seems to be based on a strawman and very rigid view of Agile development and a basic misunderstanding of what is meant by pragmatism, pitting it against Agile development as if there were some dichotomy to be resolved.

Much of the point about Agile is that it is, well, agile. In other words, it is adaptive and responsive with respect to context. If two weeks (30 days, 6 months, 1 day, etc.) is not a good delivery schedule, then you change it, right? That's agility. If you don't change it, that's not a problem with agility, it's simply not being agile, no matter what capitalisation or branding you give the word. This is not about manifestos, it's about knowing the basic meaning of the words being used.

An agile process is one that you adapt to your needs. By definition that makes it pragmatic.

Re: Waterfall never worked by Mark Harris

Even when we said we did Waterfall, it was really iterative. More like the "spiral" approach. The truth is that people don't *really* know what they want until they see it. So there are always changes requested after you think you're done. Agile just embraces that truth and helps you deal with the changes as they come.

Re: Waterfall never worked by Arialdo Martini

This is the most amazing reply ever. I couldn't agree more

Re: Waterfall never worked by Arialdo Martini

I used your comment in my post arialdomartini.wordpress.com/2011/11/29/why-you...

Forgive me, it was too much inspiring :)

Re: Agile is called agile for a reason by Arialdo Martini

Interesting. I still don't know if I agree, you have found some very good points.

Actually, the lack of a design up-front (that is a deliberate decision, in agile) seems to me a very controversial element. I'm quite sure a lot of teams are using agile AND producing up-front estimation and design, since that's what their corporate company asks.

At the same time, I strongly believe John Russell's comment above is somehow very true. Read it if you didn't, it's amazing.

I wrote about this here arialdomartini.wordpress.com/2011/11/29/why-you...

Re: by Assaf Stone

Look up the Standish Group's Chaos Reports. There is ample empirical evidence that Agile projects are declared a success more often than Waterfall projects.

Re: Waterfall never worked by Assaf Stone

According to the 2001 Chaos Report (Standish Group) Roughly 30% of 10000 sampled projects were considered successful (on time, on budget and on scope). Another 35% or so were considered "contested", meaning they were completed but broke either the deadline, scope or budget or all of them. The rest simply failed.

Agile projects have significantly higher success statistics.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

35 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT