A Case For Short Iterations

Posted by Dave Nicolette on Oct 31, 2008

Community

Agile

Topics

Agile Techniques ,

Methodologies ,

Change

Tags

Planning ,

Complementary Practices ,

Scrum

It is a commonplace that iteration length should be set according to release cycle length. I disagree; I think the two cycles should be decoupled. Short iterations provide more-frequent feedback from customers than long iterations and afford the team more opportunities to reflect and improve their work practices. Shorter cycles result in a "heartbeat" that occurs frequently enough to be meaningful. Short cycles naturally prevent the team from creating work items that are tediously large, with no "success" point to be had until a great deal of work has been done. These benefits apply even when the release cycle is long.

RelatedVendorContent

Agile Transformation Strategy

Learning to be a Good Product Owner: Foundation Skills

Technical Debt and Design Death

Manager's Introduction to Agile Software Development

Impediments to Agile in the Enterprise

Related Sponsor

ScrumWorks Pro is the only Scrum project management tool designed for the enterprise. Sign up for a free 35-day trial.

Benefits

1. Rapid response to changes in priority without disrupting work in process. It's not unusual for the Product Owner or Customer or Proxy to change priorities or add features in mid-iteration. This can be managed more cleanly if iterations are short enough that the changes may be made in the next iteration instead, since the regular pulse of iterative work need not be disrupted.
2. Problem detection. Mature agile teams recognize process smells and act on them immediately. At the moment, however, most teams using agile methods are still on the learning curve and they haven't reached a level of maturity with agile development that lets them depend on their noses. They need to track trends in project metrics to recognize problems. Given that it takes 3 data points to make a a trend and that project data are collected once per iteration, the implication is that shorter iterations expose problems faster.
3. Scope management. It's easier to move backlog items around when they are small than when they are large. Longer iterations tend to result in larger User Stories. When the Product Owner needs to change the priorities of backlog items, the impact of any given change is greater when the User Stories are larger. Shorter iterations tend to result in smaller User Stories. Following the INVEST principles, the User Stories will be easier for the Product Owner to reprioritize.
4. Iteration planning and tracking. The larger User Stories that usually come from long iterations often require decomposition into "tasks" to break the work down into bite-sized pieces. The tasks then have to be tracked throughout the iteration just so that the team can understand the status of all the Stories, either using a kanban-like system or an iteration burn chart. Many teams pause to re-estimate individual tasks that remain in incomplete status each day. All this internal process overhead can be eliminated by running short iterations so that User Stories are, themselves, small units of work and people can follow the status of the iteration in a simpler way.
5. A foundation for moving to an iterationless process. An iterative approach retains some of the natural overhead of a waterfall process even when we go to some pains to remove it. A cumulative flow diagram may expose this overhead in the form of "lead time" from the start to the end of each iteration. As teams I've worked with have squeezed their iteration length as far down as they could manage, I've noticed that they were able to eliminate significant amounts of this overhead. The shorter the iteration, the less process overhead is needed to keep things on course.

On the other hand, the discipline of working in timeboxed iterations is the source of some of the value-add of the agile approach, including frequent and regularly-scheduled demonstrations and retrospectives, a consistent schedule for delivering incremental results, frequent opportunities for customer feedback, and the sense of a "heartbeat" or "pulse" that seems to keep teams engaged over the long haul. Therefore, it seems reasonable to expect short iterations can provide a low-pain transition to a very lightweight iterationless process without making the mistake of "throwing out the baby with the bathwater" by dropping the value-add aspects of timeboxing the work, as some teams unfortunately have done when adopting an iterationless process.

A common mistake is to assume any and all practices the team had associated with "iterations" should be discarded when they move to an iterationless process. We do well to separate the concept of "iterations" from the specific value-add practices that are commonly associated with agile development, and look for ways to reduce process overhead while preserving beneficial practices.

Potential downsides

Some people have experienced problems with short iterations. Mishkin Berteig, proponent of short iterations, also mentions some potential downsides:

• "Intensity of work can lead to burnout." I think this is a question of how the team chooses to work. Short cycles don't necessarily translate into higher intensity. It can be done in such a way that short iterations only mean smaller timeboxes; that is, you commit to deliver less work per timebox. There needn't be any difference in "intensity." Other agile principles (notably "sustainable pace") are designed to prevent burnout.
• "Strategic thinking can be hard to fit into the schedule." Strategic thinking isn't an iteration-by-iteration matter. Iterations are tactical. Strategic thinking is...well, not tactical. This sounds like a management problem rather than a characteristic of short iterations as such.
• "Overhead tasks that must be done every iteration take a larger proportion of the time in the iteration." This sounds like another issue that boils down to how the team chooses to work. I've observed an interesting effect of squeezing down the iteration length: People "discover" certain overhead tasks were never really necessary in the first place, and stop doing them. They end up doing only what is necessary, which is another way to say they eliminate waste from their process. In fact, this observation leads me to disagree respectfully with Jim Shore's observations in a Java Ranch discussion that longer iterations are less stressful and that longer iterations work better for experienced agile teams. I don't think we want more time for iteration planning; I think we want less iteration planning. I'm for ever shorter iterations, vanishing to none as we move to a customer pull approach with single-piece (read: single feature) flow.
• "Waiting for resources or people outside of the team can make it more likely for work to span iterations." This is an organizational constraint. It isn't practical to try and run iterations shorter than the organization can handle. When you do that, they aren't really "iterations" because it isn't possible to deliver results that quickly; the organization can't absorb the results. To go further than this requires us to address the organizational constraints. I wouldn't assume shorter iterations are impossible because of a temporary organizational constraint - and they're all temporary, if you really want them to be. Easy? No one said that. But if organizational change were easy, it wouldn't be so much fun, would it?

Related InfoQ content: Extremely Short Iterations as a Catalyst for Effective Prioritization of Work

About the Author

An IT professional since 1977, Dave Nicolette discovered Agile in 2002 and found it solved or alleviated many of the problems inherent in traditional IT. Since then, he has been a dedicated practitioner and ardent proponent of change toward Agile and Lean thinking and practices. He enjoys sharing experiences and effective practices with fellow IT professionals and participates actively in the agile community. Dave is currently an agile team coach for Valtech Technologies in the US.

1. Back to top

   "An iterative approach retains some of the natural overhead..."

   Oct 31, 2008 12:57 PM by Mark Levison

   of a waterfall process. I take issue with the importance of this. Many in the Kanban world suggest that Planning, Review and Retrospective meetings are eventually a form of waste.
   
   I differ on this point seeing that much the activity that occurs in these meetings will occur even if the meetings don't exist. Example - I find a large chunk of the planning meeting will be taken up understanding the stories and breaking them down into tasks. This work will happen whether or not there is an explicit planning meeting.

   Reply

2. Back to top

   Multiple problem dimensions - one size doesn't fit all

   Oct 31, 2008 3:35 PM by Kelvin Meeks

   Dave's article is very helpful to stimulate discussion and out-of-the-box thinking - and offers a perspective I had not considered for Agile adoption:
   
   "I don't think we want more time for iteration planning; I think we want less iteration planning. I'm for ever shorter iterations, vanishing to none as we move to a customer pull approach with single-piece (read: single feature) flow."
   
   My own experience with promoting the adoption of Agile principles within different size organizations - and different industries has been somewhat mixed.
   
   The successes have been greater with small teams developing new software for start-ups.
   
   The more challenging experiences have been in mature organizations - with large mission crticial applications that have impact on infrastructure (e.g. Telecom).
   
   I like the idea of "single piece flow" - but the management approval, security, and quality inspection layers that must be passed for a release in large-scale development organizations (in my experience) __do__ often introduce a significant "Overhead tasks that must be done every iteration take a larger proportion of the time in the iteration."

   Reply

3. Back to top

   Cost of context-switching ?

   Nov 4, 2008 2:12 AM by Tathagat Varma

   I think on a comparative basis, an increasingly shorter iteration will eventually reach a point where the cost of context-switch (from one task to other, or one sprint to the next, etc.) will ovetake the direct costs of software development effectively canceling-out any advantages gained because of shortening the iteration.
   
   Tathagat Varma
   managewell.net

   Reply

4. Back to top

   Re: Cost of context-switching ?

   Nov 4, 2008 1:18 PM by Mark Levison

   Interesting point - do you care to hazard a guess where that might happen? Mishkin Bertieg gave a talk at Agile 2008, called: Extremely Short Iterations as a Catalyst for Effective Prioritization of Work (www.infoq.com/presentations/Short-Iterations-Mi...).
   
   Is that beyond the limit?
   
   What about kanban? Does that get around the problem?

   Reply

5. Back to top

   Re: Cost of context-switching ?

   Nov 4, 2008 3:32 PM by Dave Nicolette

   I don't think I understand the problem. When I envision a single-feature pull style of software development, I'm thinking each feature would be built and delivered before the team picks up the next feature, whether they literally use a kanban (card) system or some other mechanism. So, I don't see why there would be a context-switching problem. What am I overlooking?

   Reply

6. Back to top

   Downsides of short iterations and Iterationless feedback

   Nov 5, 2008 7:06 PM by Jack Milunsky

   Nice blog. I completely agree with your comments re Berteigs potential downsides to shorter iterations. Shorter iterations in my opinion are good and are a way to manage risk, to setup the heartbeat in the development organization and ruthlessly focus the organization around the most important things. In good Agile shops, teams would find ways to automate much of the overhead of say deploying to production every week etc.
   
   I can't however see how an iteration-less process could work however. I think you need to have the end of sprint milestones to ensure the team is working towards a common goal. Without deadlines I can't see teams rallying around artificial boundaries. Perhaps I am missing something here but I just don't see this working at least not in my experience. Thought provoking though.
   
   Jack

   Reply

7. Back to top

   Re: Downsides of short iterations and Iterationless feedback

   Nov 6, 2008 10:03 AM by Dave Nicolette

   With an interationless process you still have release milestones, and there's nothing to prevent a team from choosing to hold retrospectives and interim product demonstrations at regular intervals, if they value those things.

   Reply

8. Back to top

   Re: Downsides of short iterations and Iterationless feedback

   Nov 6, 2008 11:12 AM by Jack Milunsky

   I just don't think that release milestones are enough. I think the whole point about timeboxing (at least one aspect of it - there are others) sprints to say 30 days or 10 days (or whatever you decide) is to drive a sense of urgency and completion. I believe there are studies to suggest that individuals will naturally fill time based on who much time they have so if you don't have deadlines I am not sure you can be as effective as you can with them. My 2 cents.

   Reply

9. Back to top

   Re: Multiple problem dimensions - one size doesn't fit all

   Nov 6, 2008 2:06 PM by Dominique Archambault

   I like the idea of "single piece flow" - but the management approval, security, and quality inspection layers that must be passed for a release in large-scale development organizations (in my experience) __do__ often introduce a significant "Overhead tasks that must be done every iteration take a larger proportion of the time in the iteration."

   
   I have to agree that you can't fully embrace short iterations when part of the process is out of your hands. But I'm thinking that there may be parts of the process that are under the development team's (or group's) control.
   
   While deployment in a production environment may not be possible, why not deploy in some test environment dedicated for testers and clients to validate what's been added? There could be at least some feedback and at least part of the entire development process would proceed with short iterations.

   Reply

10. Back to top

    Absences

    Nov 7, 2008 1:53 PM by Adrian Eidelman

    How do you think the absence of a team member affect commitments in short iterations? Do you think this has a higher impact when the length of the sprint is small?

    Reply

11. Back to top

    Re: Multiple problem dimensions - one size doesn't fit all

    Nov 10, 2008 5:05 PM by Dave Nicolette

    The issue of process overhead in a large or
    
    ganization is quite real. I think there are a couple of different situations, broadly speaking. First there's the case of an agile development team working in a traditional organization. In that case, the main impact on agile work will be that the "last responsible moment" (LRM) for decisions that have dependencies on non-agile groups will be earlier than we might prefer. It needn't affect the overall pace of work for the agile team; it only affects issues for which the team has a dependency on a non-agile part of the organization. Something that's worked for me in those situations has been to engage the non-agile groups on which our team had a dependency very early in the project so that they understood what we intended to do. In most cases, it turned out they were just as frustrated with status quo methods as we were, and they were happy to adjust their response time for our teams. In other cases, the non-agile group was adamant about sticking to The Process, and we resorted to contract negotiation over collaboration with them so that all parties knew what they were expected to do. It's manageable. There's no need to avoid all opportunities for improvement just because perfection isn't immediately achievable.
    
    The second case is an agile development team in a supportive organizational culture. In that case, if there are business needs for multiple layers of approvals and inspections, then that's just part of the business. One would expect that because the organization as a whole is cognizant of waste and interested in delivering business value effectively, they won't be creating needless overhead; any approvals and inspections deemed necessary are probably necessary. I don't like waste, but not all overhead is waste; sometimes it serves a purpose. There might be regulatory requirements or customer safety issues that call for a bit of extra care. Frankly, I don't see this as a reason for longer iterations.

    Reply

12. Back to top

    Re: Downsides of short iterations and Iterationless feedback

    Nov 10, 2008 5:11 PM by Dave Nicolette

    I'm not sure who said there would be no deadlines. Anyway, the way your comment is worded it sounds as if you haven't experienced working in both ways, so it's hard for you to compare them. Why not try a project or two with and without iterations and see what really happens? If it turns out that people actually don't deliver anything because they think they have an infinite amount of time, you can always re-introduce iterations in mid-project. I'm satisfied that enough projects have been delivered successfully that way to prove the case, but if you're not satisfied then there's no surer way to reach a conclusion than through personal experience. If you decide to try a project without iterations, I suggest you don't just dump iterations without introducing some other form of control. Read up on "kanban" methods first; see how people have managed the work without explicit iterations, and then give it a try.

    Reply

13. Back to top

    Re: Absences

    Nov 10, 2008 5:13 PM by Dave Nicolette

    I think that if a team member is absent for, say, three days then the impact is three days of one person's time, regardless of the iteration length. The impact to the project is the same.

    Reply

14. Back to top

    Re: Downsides of short iterations and Iterationless feedback

    Nov 11, 2008 11:25 AM by Jeff Santini

    But, if I understand you correctly, you are suggesting that false milestone's will create a sense of urgency. If business value is your goal, and a completed iteration does not deliver business value(i.e. is not deployed) then you have not achieved a real goal at iteration end.
    
    My team a few years ago came to the conclusion that we were fooling ourselves trying to work towards iterations which did not lead to deployments. Of course there are two ways to solve this problem. One is to lengthen iterations as we have assumed here. The other is to shorted deployment cycles.

    Reply

• Older >