BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles The Secret Sauce of Highly Productive Software Development

The Secret Sauce of Highly Productive Software Development

This item in japanese

Bookmarks

Professional trainers and coaches have seen it again and again: it's a pattern that too many Agile teams get hung up on - getting stuck in the just-average "norming" stage, rather than progressing into the exciting, high "performing" stage of team growth [1]. We ask readers to consider that there may be one common element of all software development projects which, when maximized, could help make productivity soar. In fact, we believe that many of the most successful teams (both Agile and traditional) are already leveraging the seemingly simple but too-often forgotten "secret sauce" of software development: frequently making time to reflect and learn. Learn about what? Everything: each other, the technology, the domain, the customer, etc. A team that learns quickly succeeds. Read on for more about the invisible "learning bottleneck" that stunts team performance.

A Hypothetical Experiment

What can we do to make software development better?  What are the typical bottlenecks in software development?  Is there any one commonality?

The days where one could run a true experiment are long gone.  To run a true experiment you need build the same project twice.  This is, unfortunately, prohibitively expensive in today’s business environment.  Therefore, it is currently unrealistic to run real experiments to figure out exactly what the common weak points are.

At the same time, we all have experience developing software as practitioners.  So here is a hypothetical situation that we have presented to many students:

Suppose I was your client and I asked you and your team to build a software system for me. Your team proceeds to build the software system.  It takes you a full year – 12 months – to deliver working, tested software.

I then thank the team and take the software and throw it out.  I then ask you and your team to rebuild the system.  You have the same team.  The same requirements.  The same tools and software.  Basically – nothing has changed – it is exactly the same environment.

How long will it take you and your team to rebuild the system again?

When we present this hypothetical situation to students – many of them with 20+ years experience in building software – they typically respond with anywhere between 20% to 70% of the original time.  That is, rebuilding a system that originally takes one year to build takes only 2.5 to 8.5 months to build.  That's a huge difference!  It's hard to identify another single factor that could affect software development that much!

So, what's the problem?  What was different the second time around? The team has learned.  They learned about each other as a team, and have gelled over the year.  They learned about the true requirements – not just those written down.  They also learned to use the toolset, the idiosyncrasies that come up during all software development; basically they worked through all the unknowns until they built and delivered a successful software solution. Learning is THE bottleneck of software engineering. [2]

The learning that occurred costs a significant percentage of the work.  In fact, we'd guess that it accounts for about 30% to 80% of development time.  And here's the main reason that Agile practices work so well – they are all about recognizing and responding to change. 

Agile practices, from Test-First Development and Continuous Integration to Iterations and Retrospectives, all consist of cycles that help the team learn fast.  By cycling in every possible practice, Agile teams accelerate learning, addressing the bottleneck of software engineering.  Call it "scientific method," "continuous improvement" or "test everything." 

Examining Agile through
“Learning is the Bottleneck” Lenses

Agile software development works – that is an established fact.  Hundreds of successful war-stories have been documented over the past eight years.  One of the interesting things to note about Agile development is that most of the practices are not new - in fact they are very old.  Agile has simply distilled many of the most successful software development practices and brought them together.  In fact, the Agile manifesto was not creating something new, but finding commonality among a number of light-weight methods that had already been successful throughout the 90’s (two different accounts by Jim Highsmith [3] and Uncle Bob [4] are available).  Reflecting on many of the most effective practices reveals some very interesting commonalities…

Cycles for Recognizing and Responding to Change

So how do we learn?  We learn from our mistakes (if we are paying attention, many do not).  How do we learn faster?  Make more (smaller) mistakes faster – "fail fast".  Immediate application of what we've learned is a key to cement learning and make it "stick".  By immediately applying what we've learned in the next cycle, we compound the effect of learning (like compound interest!).

A very common cycle we find again and again looks like this:

1) set a goal, 2) perform an action to achieve that goal, 3) compare the outcomes of the action and the original goal, 4) change the action accordingly and going back to (2).  Learning happens in step (3) and is applied in step (4).

This simple cycle is the basis of many of the practices in Agile development:

  • Test-First Development (TDD):
    1) write a failing test, 2) write code to make test pass, 3) run test – does it pass? 4) if test is still failing then learn why and go to (2).
     
  • Daily cycle:
    1) set tasks to be done as a team for the day and report at the daily stand up meeting, 2) go do it, 3) report back the next day on past progress, obstacles, and 4) apply what’s been learned to the plan for the next day.
     
  • Test Driven Requirements (TDR):
    1) define the requirements as tests, 2) develop the software, 3) run the tests; if they pass you are done, if not then 4) find out why and go back to (2).
     
  • Iteration:
    1) define a ‘done state’, 2) commit to a set of requirements that form that iteration’s backlog, 3) work throughout the iteration to build the backlog, 4) close the iteration by testing each item against the ‘done state’ and mark them done or put them back on the backlog for later consideration.
     
  • Demo:
    usually performed at the end of the iteration, it gives the customer a chance to  test out if the requirements envisioned when implemented really solve the problem at hand.  1) (implicitly) determine business needs/value addressed by the requirements, 2) define requirements, 3) build requirements during iteration, 4) evaluate implemented requirements against business needs.
     
  • Retrospective:
    1) decide on a way the team will work, 2) work that way for an iteration, 3) reflect on how well the practices worked and 4) come up with actions, with owners and dates, intended to create a more effective process and implement them.
     
  • Release:
    1) create a vision for the release and set the business goals the release is to meet, 2) create a release backlog, 3) perform several iterations to build the release, 4) deploy to customers and collect feedback for the next release.
     
  • Scrum of Scrums:
    in dealing with multiple projects at once, each project has its own set of cycles running independently and at the same time synching up via regular meetings – Scrums of Scrums – where representatives from each project reports to the rest giving an opportunity for recognizing and responding to change at an enterprise level.
     
  • Management tests:
    1) work with a product owner or with business stakeholders to define, at a high level, how they will measure the success of a project or release, 2) keep these "tests" visible as the team works to deliver on these expectations, 3) incorporate evaluation of management tests into retrospectives, 4) team and product owner work together to align each iteration with these tests.

Many of these cycles fit neatly within each other.  Test driven requirements has several cycles of test driven development.  An iteration has several cycles of test driven requirements.  A release has several iterations, etc… 

Furthermore, these cycles feed into each other – in fact they feed into each other constantly as learning occurs that invalidates a higher cycle. The test driven development loop is nested within the test driven requirements - that is for each requirement, several test driven development cycles occur to fulfill that one requirement.  Therefore, for example a broken test may mean bad code, but it also may mean that for the requirement to be built, another previous requirement may need to be invalidated.  So learning in the test driven development loop brings about learning about requirements.  Similarly, iterations are nested within releases.  Therefore, an iteration that fails to meet one or more of its goals because of a technical limitation that could not have been foreseen which significantly increases the effort needed to complete a significant portion of the system.  The team then regroups, considers the release backlog, and does some reprioritization and scope modifications to accommodate the new information.

Cycles: Necessary but not Sufficient

The cycle is one of two basic requirements for recognizing and responding to change - it provides the opportunity.  The second requirement is communication, to magnify the learning by involving everyone, hence the emphasis on "information radiators" in Agile development approaches.

Information needs to be communicated to the rest of the team, and beyond.  Without communication, issues may go unrecognized.  Without communication, those who see issues but can't spot solutions may never get the input of others with different skillsets who can see the solutions.  In a team environment, it is not always obvious who is best suited to solve a given problem - by telling everyone you invite suggestions from any interested party, not just the team "experts" - sometimes novices or outsiders are able to "think outside the box" and surprise us!

Communication accelerates and accentuates learning, and a formal pause for reflection can ensure that this communication does not fall on ears deafened by cries of “hurry, hurry!”.  This pause could be as formal as a retrospective, or as informal as a shared meal at the end of the iteration – as long as learning and improvement are on the agenda.  And while communication should happen in a planned manner at the beginning (communicating goals) and end (test results) of each cycle, we also benefit dramatically from informal, “osmotic” [6] communication as each cycle unfolds.

A significant number of Agile practices directly focus on communication – both formal and “osmotic”:

  • Self-organizing team:  A team works together to respond to changes that happen together.  They collectively do what needs to be done to build the software.
     
  • Co-located team:  The team members sit together and regularly have group conversations, overhear others, and individuals are aware of what is happening by osmosis.
     
  • Cross-functional team:  A team with multiple disciplines works together to solve a problem from end-to-end.  By working together, they share their individual expertise.
     
  • Pair programming:  Two people working together to solve one task is a very deep form of sharing experience and expertise.
  • Information radiators: are "big visible charts" whose sole purpose is to communicate some important data to anyone passing by.
     
  • Evocative documents:  Agile teams build documents together to have conversations.  Those documents then evoke the entire discussion and context when read later.  These are much more valuable than traditional throw-over-the-wall documents which are representational in nature – i.e. because so much is invested in them, there can be a (false) belief that these formal documents actually are the thing they describe.
     
  • Stand up meeting:  Agile teams synchronize daily by communicating tasks just completed, roadblocks met, and tasks planned for completion in the next daily cycle.

Communication, when added to a cycle, allows the entire team to learn faster and respond as a team.  After all, software development is a team effort.  If learning is the bottleneck for the team, then the entire team needs to learn as much and as fast as possible.

If they are using the practices well, Agile teams learn about everything – not just technology - quickly: design (TDD, Pair programming, Cross functional team), requirements (Demo and TDR), the product (iterations, releases), and the people (Retrospective, Stand up meeting).

Why is this important?  Theory to Practice

This is all well and good – learning seems to be a very important part of software development.  It is probably even more important than we currently give it credit for – but so what?  How can this focus on learning help my team or yours produce better software?

Despite all the examples of "learning" practices cited above, Agile teams are not immune to the learning bottleneck. Urgency sometimes presses us to take the short term win by cutting out the learning step ("oh, we'll do that later"). Agile teams that fail to learn may display any or all of the following symptoms:

  • fatigue (not working at a truly sustainable pace) leading to morale problems,
  • repeated inability to "get to done" in a single iteration,
  • consistently under-delivering on promises (some promised features or stories are not even started),
  • so many defects consistently coming back from deployed software that development plans are derailed,
  • de-valuing or eliminating retrospectives ("because we never actually solve the problems we identify").
     

Keep Your Eye On the Bottleneck

Consider the figure below; each step has a speed indicated in parentheses below the step name.  Inventory builds up between two steps when they are of mismatched speed – therefore because step A is faster than step B, an excess product of A waits to be processed by B. Throughput of the overall system will always be limited by that bottleneck [5].  The idea is that if you want to increase the overall throughput of the system you need to focus almost exclusively on improving the performance of the bottleneck.  Efforts spent elsewhere are waste and could even be counterproductive.  That means, for the SSP - in the figure below, the only way we can make an improvement is to fix step D.  If we address anything else, we will not improve.

If learning really is the bottleneck of software development then this should elevate it to our number one priority.  It means that any efforts we spend on anything other than learning will provide only limited productivity gains.  Conversely, it means that anything that does improve our productivity has somehow improved our rate of learning. 

Now, when we're reflecting on our process, contemplating ways to deliver more value, we should remember that we are usually delivering several sorts of value to the businesses we serve: working software, certainly, but also maintainable and changeable  software, and a responsive team to continue that work. Priorities for the first two should be solicited from the business, but the third is simply an expectation of professionalism, and is usually within the control of the development organization. This last form of value, team agility, is actually an asset of the organization, and can outlive individual projects, creating the opportunity for greater profit or speed on subsequent projects.

Agile teams should be relatively stable organisms that live for 6 months or more together, in order to grow into effective teamwork. Building such a team requires a strategic view, which must be considered together with the immediate needs of the business in each iteration. If this balance is not maintained, a time will quickly come when the team fails the business, due to unresolved booby-traps in their process which work against predictable velocity and quality.

To avoid creation of a fragile and inconsistent team, when we reflect on process we should be asking two kinds of questions:

  1. "How will this affect our velocity in the next iteration?"
    and
  2. "How will this affect our learning?" (which in turn affects our velocity and responsiveness in all following iterations and projects).

So, instead of simply asking, “Will pair programming slow us down?” we should also ask, “Will pair programming slow our rate of learning or will it speed it up?”  Instead of asking “should we really have Demos every two weeks, although our stakeholders are only available on a monthly basis?”, we should ask “How will reducing Demo frequency to once a month affect our learning?”  Instead of asking “Which tools should I install to support Agile?”, we should ask “Does tool ABC increase our learning?  Or does it slow it down by reducing our communication bandwidth?" or, better yet, wait until a need is discovered to track information (in order to learn from it), then find a tool that provides it with the least work on your part [7].

When we understand that learning enhances the corporate asset called "the team," we now have a vocabulary to explain the benefits of our learning practices to stakeholders.  "Yes, pair programming looks expensive at first glance - let me explain how it offsets risks and outweighs the apparent cost in the long haul..."  Don't forget - with Agile's short cycles, "the long haul" may be as little as 3 to 6 weeks!

Where to Go From Here

We had one purpose in writing this article. It was not to analyze the theory and mechanics of learning, which can – and in fact has – filled many, many volumes.  Neither was it not to categorize and/or evaluate different Agile practices – those mentioned are simply examples, reminders that we already have many ways to learn, if we choose to.

Our objective was to bring learning to the forefront of our minds and actions, because we believe it is the key to the success of Agile practices.  Don’t just keep learning in the back of your mind - bring it to the forefront.  This article stands as a reminder that the Agile approach already offers many learning practices and mechanisms – are they all being used to best advantage, to serve your team and your business?

See the world through ‘learning is the bottleneck’ glasses.  It can significantly reduce the chance of cargo-culting [8] Agile practices, and focus your efforts for maximum effectiveness. 

Endnotes

  1. The Forming–Storming–Norming–Performing model of team development was first proposed by Bruce Tuckman. http://en.wikipedia.org/wiki/Forming-storming-norming-performing
  2. This hypothetical story and the "Learning is the Bottleneck" idea comes from Ashley Johnson who is is VP of Business Planning and Strategy for Valtech Skill Development.
  3. For one history of commonality among the early light-weight methods see Jim Highsmith http://www.agilemanifesto.org/history.html
  4. For another history of commonality among the early light-weight methods see Uncle Bob Martin http://blog.objectmentor.com/articles/2007/07/10/the-founding-of-the-agile-alliance
  5. "Osmotic" communication on Agile teams: http://www.agilemodeling.com/essays/communication.htm
  6. Readers familiar with Theory of Constraints will find this argument familiar, and in many ways an over-simplification when a non-linear process is considered.
  7. See Appropriate Agile Metrics: Using Metrics and Diagnostics to Deliver Business Value, Deborah Hartmann and Robin Dymond, 2006 http://www.berteigconsulting.com/archives/2005/01/agile_work_reso.php
  8. Cargo Cult: during World War II a number of airbases were built on remote tropical islands inhabited by pre-industrial societies. During the war soldiers built airfields and control towers and engaged in various activities that resulted in large airplanes full of cargo landing and discharging their contents. The native inhabitants shared this cargo. After the war the soldiers departed and no more cargo was available to the natives. So they adopted, as best they could, the superficial form of airstrips, control towers, and ritual behaviors intended to induce the return of planes full of cargo. A cargo cult is any group that adopts form instead of substance and believes that doing so will bring about a desired result.

About the Authors

Amr Elssamadisy is a software practitioner.  Building software is a hard - with a capital H - and Amr is passionate about finding better ways to develop software and sharing it with the community.  Today it is helping his clients build better software by successful adoption of Agile software practices, tomorrow it will certainly be something different as we continue to learn better ways of building software.  Amr is also the author of Patterns of Agile Practice Adoption: The Technical Cluster and is an editor for InfoQ's Agile Community.

Deborah Hartmann is a bilingual Agile practitioner, trainer and coach based in Toronto and working internationally. Deborah is passionate about making work both valuable to the business and enjoyable for the team. She's coached both large and small businesses in Agile adoption, has been Lead Editor for InfoQ's Agile Community since April 2006, and has facilitated OpenSpace conferences for the XP and BarCamp communities in Canada and the US.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Bottleneck?

    by Jim Standley,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I'd rather think it is just what we DO most of our time. Whatever you call it, the "learning lense" does look like a valuable view.

    And this view will show us interesting things about the cost of change and the value of Big * Up Front?

  • Synchronicity

    by Kevin Rutherford,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The claim that learning is the bottleneck, in the TOC sense, is fascinating - and will have many of us scurrying away to draw CRTs to check it! Your article also dovetails neatly with a number of others that have appeared in the last few days - see my mini carnival-of-blame for a survey that ties them together into a story.

  • Re: Bottleneck?

    by Amr Elssamadisy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I'm glad you agree that the "learning lense" idea is valuable. The 'bottleneck', in TOC terms, is counter-intuitive. In fact, the idea that there *is* a common bottleneck even more uncomfortable.

    But, I cannot think of one single thing that can make any team get such a gain. As awkward as that may feel and sound, it points to the fact that it *is* the bottleneck.

    I have shared my thoughts here to test that bold assertion with the community.

  • Re: Synchronicity

    by Amr Elssamadisy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks Kevin! Please share your thoughts on the bottleneck idea. I would be equally happy to have holes blown in the theory as to have agreement/validation. (Well, I would prefer validation to be completely honest, but...)

  • Learning is hidden and underrated.

    by cleve gibbon,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I really enjoyed reading this article because we are currently embracing agile practices and a lot of what you mention we have and are encountering with respect to learning.

    I found that great teams are full of people who want to learn and on learning new thingthrive s. Agile is a great way to build this into everyday project life.

    On the article I found that teams that skip retrospectives suffer badly. I think it is one the most important parts of the learning process. Also, migrating the "whole" team thinking over to TDD is challenging but very important from the feedback cycle perspective. These two have stood out key contributors to addressing the learning bottleneck.

    I'm interested to know if you have any similar experiences with the practices you mention in your article?

  • See also...

    by Matt Savage,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Pat Kua's series on onboarding new team members:

    www.thekua.com/atwork/category/onboarding-strat...

  • Re: Learning is hidden and underrated.

    by Amr Elssamadisy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks Cleave!

    Check out the book on Patterns of Agile Practice Adoption: The Technical Cluster (www.infoq.com/minibooks/agile-patterns) - it is focused on adoption, which necessarily includes learning.

  • To be an agile development org or not to be...

    by Remember Objective,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    that is the question.

    The end result of not learning is fragmented, transient non-organization of temporary specialist workers, who never talk to one another about actual business-related problems or technical creation because they have nothing relevant to say to one another.

    That's pretending you don't have a development organization.

    Nobody has "enough time". They only have time to keep going in their specialist microcosm. "Learning"? What are you talking about--don't you have enough to do? A new language? What for? I am not in pain even though I'm still compiling and "building" web apps...

    Ruby? Ruby on Rails? I've never heard of them before this conversation. He was serious.

    Learning is always "playing" and only in spare time, which there is none of. Why would a person have spare time if they were essential to the organization? To the non-organization...?

    What customer in their right mind is going to pay for me learning something, unless I'm a certified learner of some kind with a demonstrated track record of getting an ROI from learning? Why not just hire someone who already knows? How does the customer know that the developer is willing to jump in? Easier said than done...

    I have 600 fields to display in a form. I do it the hard way and charge the customer for that, and they think they're getting a hard worker. I pass this pile of code on to the next developer, who declares it a unmaintainable pile of rubbish, but it sure must've been a lot of work, yeah.

    The next developer invests some learning time, finds a way to do it much easier with less effort. It's a total rewrite. What do we tell the customer?

    But then, that developer has moved on to AJAX and wants to rewrite it again...because they have since learned that it could be done much simpler. Server-side is just boring now...no matter what the language.

  • Re: To be an agile development org or not to be...

    by Deborah (Hartmann) Preuss,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks for the colourful illustrations, Mr or Ms "Objective" !

  • Re: To be an agile development org or not to be...

    by Greg Helton,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Why teams don't grow? In the case of the agile team I was on, it is because my project lead at the time thinks that every domain object should be responsible for its own persistence. And lots of similar reasons. Is there ANY overlap between Agile and good design?

  • Feedback from Agile 07 Lightning Talk

    by Amr Elssamadisy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    So, I was at Agile 07 this week and attempted to give a lightning talk (5 min max) about this idea. I suggested the hypothetical example in the article and asked the attendees for responses.

    For the very first time in all the groups I have asked, several people seriously said that the second time around will take significantly more time to build. People don't really speed up until the 3rd time around. There were some that had (almost) built the same thing twice also (conferences are great!)

    Hmm.... What does this mean? Does this mean that the hypothesis of learning being the bottleneck is wrong? Or that we, as human beings, need several times 'around the block' to learn correctly?

    I'll try to encourage some of the attendees of that session to comment on this thread.

    Amr

  • Implication is not bidirectional

    by Cliff McCollum,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    While I agree with most everything in the article, and enjoyed the ideas it presents, it contains some very poor logic:

    It means that any efforts we spend on anything other than learning will provide only limited productivity gains. Conversely, it means that anything that does improve our productivity has somehow improved our rate of learning.


    If learning accounts for 25%-80% of a projects time, I agree that spending time on things other than learning may have decreased value.

    However, this does NOT imply the converse, as your quote seems to indicate. There can still be numerous things that will increase productivity that have nothing to do with learning. Nothing in this article, or its evidence, supports the idea that all productivity increases come from learning.

  • Re: Implication is not bidirectional

    by Amr Elssamadisy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I agree that implication is not bidirectional - let me clarify:


    While I agree with most everything in the article, and enjoyed the ideas it presents, it contains some very poor logic.
    <\blockquote>

    If and only if (iff) learning is the bottleneck, then all productivity gains must come through improving that bottleneck. (The article is a hypothesis and not a proof or evidence of an experiment.)

    The assertion made in TOC is that any improvement in throughput can only be through improving the bottleneck.

    So, let's make this easy, all it takes to disprove a hypothesis is one counter-example - could you oblige me with one?

  • Re: Feedback from Agile 07 Lightning Talk

    by David Chia,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    It could mean that the projects that they were talking about just took that long to do.

    The classic example is giving birth. It's not going to be much faster than 9 months, that's just the nature of it. The mother that has given birth already cannot use her knowledge to have a second child in, say, 4 months. In fact, the second pregnancy might prove even more difficult because of any number of external factors. Even though the mother gained knowledge from the first pregnancy, it wasn't applicable the second time around.

    Is learning the bottleneck? Great question. Maybe it's only half the problem. As the GI Joe cartoons of my youth used to point out, "Knowing is half the battle." Perhaps the other half is doing, being able to use the knowledge you've gained. If you can't use it, was it that useful?

    Also, what do we decide to learn? Are we learning the correct things? Let's use a driving example. Let's say I've got a manual transmission car and I'm learning how to go from A to B. I've determined that being able to shift optimally can shorten the time by 10%. That's great, but not when compared to a shortcut through an alley that could've saved me 80% of the time.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT