BT

The Current Direction of Agile

Posted by Jon Arild Tørresdal on Aug 19, 2009 |

In the last couple of years, Agile adoption has increased quite a bit, letting a number of people experience and challenge different elements of the Agile processes. This increased exposure and usage has made some try out different solutions to different challenges than what was originally proposed by the people that came up with the Agile processes. This is a core part of Agile, to change the process over time and make it more effective. This article will focus on some of the recent trends within the Agile community by briefly describing some alternatives to today’s well known Agile processes. Particularly focusing on estimation, forecasting deliverables and the increased impact Lean manufacturing has had on the Agile community.

Lean

I will not cover Lean specifically, but I will briefly cover some of the Lean principles, practices and tools which are relevant to the other processes and tools covered in this article. Let’s first look at a brief description of Lean from Wikipedia:

Lean software development is a translation of lean manufacturing principles and practices to the software development domain. Adapted from the Toyota Production System, a pro-lean subculture is emerging from within the Agile community.

Lean in software development originated in Mary and Tom Poppendieck’s book, Lean Software Development, where they cover 7 principles and 22 tools of Lean Software Development.

Below is a short list of terms often used in Lean that you should be familiar with reading this article:

  • Waste
  • Just-in-time (JIT)
  • Work In Process (WIP)
  • Pull based systems

Waste

Waste (a.k.a. muda), is about eliminating artifacts from your process that does not add value to your customer. Mary Poppendieck defines waste as:

Anything that depletes resources of time, effort, space, or money without adding customer value.

And value as:

Seen through the eyes of those who pay for, use, support, or derive value from our systems.

In Mary’s article, Principles of Lean Thinking (PDF) and also in the Poppendieck book mentioned earlier, she identifies seven wastes of software development. The wording of these wastes has changed a bit since they first were published and below I’ve listed the ones I saw her use in a presentation recently:

  1. Extra features
  2. Handovers
  3. Work in process
  4. Failure demand
  5. Task switching
  6. Delays
  7. Defects

(Limit) Work in process (WIP)

Limiting WIP is about having feature requests that have been initialized (specification, requirements gathering, etc.) to flow through the system in a continuous way. In other words, when your organization first starts to work on a feature request, limit how long it stays in process before going to release.

Just-in-time (JIT)

JIT is about reducing in-process inventory and avoid/discover bottlenecks in the process. JIT is often used as a mean to reduce WIP (described above). A typical example of inventory in software development is requirements going from specification to development or code going from development to test. JIT sets focus on improving return on investment (ROI), quality and efficiency. One of the processes/tools supporting JIT is Kanban, which I will talk about later in this article.

Pull based systems

The number one identified waste mentioned above is extra features. To pull instead of push is about responding to customer needs, instead of giving the customer what you think he will need (extra features). As with JIT, Kanban is one example of a tool supporting a pull based system.

Iteration Based Processes

Scrum and Extreme Programming (XP) are two well known Agile processes that utilize iterations or sprints as a central part of their planning and deliverable technique. In relation to estimation, you typically find that you estimate User Stories (XP) or Backlog Items (Scrum). In this article I will use User Story to describe both User Story and Backlog Item. User Stories are typically estimated in either Ideal Days or Story Points. Tasks are typically described in ideal hours. Mike Cohn, in his book: Agile Estimating and Planning, has covered the different methods and techniques for estimation and forecasting thoroughly.

The main idea is to have fixed length iterations of 1-2 weeks. Figure 1 shows three iterations from a backlog with stories of varying size. It also shows that the team has a velocity of 20 points, meaning their able to deliver 20 points worth of stories every iteration.

Knowing the team’s velocity is important to be able to forecast when a user story will be finished or predict which stories will be part of a release.

Equal Sized User Stories

When an Agile team first starts out they often start with Scrum or XP. This is the approach covered in most Agile books. A typical scenario for a fresh Agile team might go something like this:

The first planning meetings takes quite some time, with a lot of discussion and thinking to come up with the best estimates possible. First for each story, then for the tasks, and at the end decide on how much to commit to for the iteration. After a while (several iterations) when the team gets more experienced, they know almost by instinct how much they can commit to for a single iteration, and estimation of stories and tasks goes much quicker.

Some teams have taken this to another level and skipped the whole estimation process. Rather than spending time on coming up with estimates, they split their user stories into equal sized stories (see figure 2). The main difference is that you don’t estimate each and every story, but you have each story fit a fixed estimate or size.

By having these fixed sized stories, the team can commit to a certain number of stories for their iterations. Their velocity then becomes the amount of stories they are able to commit to, instead of the amount of Story Points or Ideal Days. This simplifies the planning process quite a lot.

Think about how you would find out how many stories (or which stories) a team will get done in a three months time, with a prioritized backlog. Using the team’s velocity (number of stories), multiply it with the number of iterations for the three month period and you have your forecast.

Fixed sized user stories eliminates waste by not spending time on doing estimates and allow the team to focus on the details of what’s important; implementing the stories.

Some argue that you still estimate, since you have to make all stories equal size. How do you do that without estimating? I find this to be about two things:

  1. Getting each and every story small enough.
  2. The average sized story is what is important.

For the first point it’s important to have as small stories as possible, something that adds value to the business and are deliverable. A two day story is much easier to spot and be correct on its size, than a bigger story that you might think take 8-10 days, but in reality takes 15.

For the second point it does not matter if one story or another is bigger or smaller as long as it’s not extreme deviations. Typically you find an exception from time to time, but all in all your velocity does not fluctuate very much.

Naked Planning

Naked Planning, coined by Arlo Belshee, takes a different approach where not only estimation is removed from the process but also iterations. In relation to estimation Arlo asked himself the question:

Are estimates necessary to decide the order of prioritization and make a guess at how long it’s going to take something to come live?

After talking to the business he found it was not and found another way of accomplishing the same thing.

Inspired by Lean practices, Arlo used a queue of Minimum Marketable Features (MMF’s). Using this queue the development team grabs the top most MMF and start working on it. When finished they grab the next one from the top and so on. Like Lean this is a pulled based system. The queue is managed by the business, always having the most important and valuable MMF at the top. The queue is kept to about seven items, which is about how many a person can keep in his head at once, and also a manageable amount of items to prioritize amongst.

To estimate when an MMF can be delivered they use historical data. When an MMF gets pulled from the queue it’s marked with the date, this is also done when the MMF is finished. Every three to five weeks they calculate the average of the duration, giving the business a number to work with in planning. If the size of the MMF’s put into the queue gets bigger or smaller, the duration estimate adjust itself after a short period of time.

Kanban

From Wikipedia:

Kanban (in kanji 看板 also in katakana カンバン, where kan, 看 / カン, means "visual," and ban, 板 / バン, means "card" or "board") is a concept related to lean and just-in-time (JIT) production.

This description can lead you to believe that Kanban is all about the task board most Agile team’s use. That is not the case. Kanban as a tool uses a white board to visualize the flow of “materials”, but it is not to be confused with the board itself.

Kanban is a way of controlling (limiting) WIP and keeping continuous flow. You use the whiteboard to visualize the different stages an item needs to pass through before being deployed. By monitoring this workflow it’s easier to discover bottlenecks that slows down your process. You typically see this by observing where elements queue up. Often a number is put on columns defining how many items are allowed to be “in process” at the same time. This number is specifically for limiting work in process.

A simple “Kanban” we often see on Agile teams have the following columns: ToDo, In Process and Done (see Figure 4).

Kanban takes this to a different level and visualize the workflow used by the organization and not just the development team (see Figure 5).

Typically you can follow a feature on the Kanban board from the idea and all the way through the different phases in the organization to its deployment into the system. Henrik Kniberg has a nice cartoon like description of a small Kanban system that helps visualize this further.

Kanban does not utilize iterations, but often has regular release cycles. Having a release cycle of e.g. two weeks, you would put whatever has reached the “release state” into production. However, there is nothing preventing you from releasing finished content right into production, as long as your process support one-click deployment, which any Agile team should do.

One thing to note about Kanban: there is no perfect Kanban system that everyone should adapt. It will be different for every organization, because they are different by nature.

David Anderson is one person in the Agile community that has used and documented Kanban in software development. I encourage you to check out his writing and presentations on the subject.

Scrumban

Scrumban was coined by Corey Ladas. Based on his extensive Scrum experience he describes in his book, with the same name, how you can start adapting lean thinking to Scrum focusing on Kanban. Scrumban is interesting because it does not just explain what Kanban is, but takes you through the steps of moving from Scrum towards Kanban in a step-by-step manner. Scrumban is neither Scrum nor Kanban, but if you go all the way you end up with doing Kanban on an Agile team.

Conclusion

There are obviously other examples of where Agile is going not covered here, but I find that these tell a story of which direction Agile is going. Many teams have found iterations to be a limiting factor and looked for ways of removing them. Some wanted to scale the Agile process to not just include the development team and those working closely with that team, but to also include the rest of the organization. Others found that the amount of time spent on estimating was considered waste and found other ways of delivering software and still be able to satisfy the questions and demands from the business.

But all in all I think it shows that the software industry is maturing. Software teams see value in Agile processes, but they continuously want to improve it, which hopefully brings us to better and more effective processes as well as better quality software. Software development is still a young craft and we live in interesting times!

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

is that the way to go? by Steven Mak

I would be very much worried if that is the direction of Agile.

I am more concerned on the lack of focus on the engineering practices, code quality, etc

Moreover, it's not just about being Agile... I think it's more on good ways of developing software.

The article is not insightful enough, too bad.

Re: is that the way to go? by Jon Arild Tørresdal

Steven,

I'm sorry you feel that way and I must say I don't agree with you. I'm not saying this is the way to go. I have observed (from my perspective) new processes and tools emerging in the agile community and used this as an example of the direction agile have moved in, hence current and not future. I do personally feel that the lean inspiration of these "new" processes has made a better agile. What is it with this trend that worries you?

In regards to engineering practices I'm very much an advocate for clean code and better quality software, but that's a different topic in my opinion, and should be addressed on its own.

Re: is that the way to go? by Al Tenhundfeld

I think the article provides a nice summary of the leading edge in Agile. I also understand Steven's point about wanting to see more focus on engineering practices.



In my experience, I'm seeing a trend towards two large focus areas in Agile: process management and engineering principles. I think people have realized that Agile will not save a mediocre team, but even great teams often fail under chaos/waterfall.


This article does a good job describing the new ideas in Agile processes. A nice complement to this article would describe new ideas in engineering: BDD, Continuous Deployment, integrated real-time code metrics, etc.

There is a great missing piece in your review by Agustin Villena

Your article is all about management and maybe teamwork, but forgot the movement that is pushing back the technical quality as a fundamental element of Agile: Software Craftmanship

Re: is that the way to go? by Amr Elssamadisy

Hey Al,

Why don't you write this article? It is a great idea and we're always looking for good contributions to the AgileQ.

Amr

Lean Startups by David Bland

Smaller, ventured backed companies especially are already evolving the above approach. I won't go into details here, but I suggest joining LSC Google Group leanstartupcircle.com/ if interested.

Wrong way to go? by Tim Hodkinson

In my experience, we're barking up the wrong trees. The article is not wrong by any means, in fact its all too true: but in my eyes the real innovation of agile principles was that they addressed the fundamental concept of what most software engineering is, and its still very much like bespoke tailoring, not factory based manufacturing. Methods and concepts from manufacturing (eg Kanban, lean etc) will never really work because the process itself is not akin to manufacturing. Manfacturing is a defined process: Building software is an empirical process. It was this initial revelation that prompted Schwaber to create Scrum, after all.

Re: Wrong way to go? by Jon Arild Tørresdal

I see your concern in regards to manufacturing not directly mapping to software development, though that is not to say that anything found in manufacturing will not work for software development. I think there is a lot of proof and metrics that show the opposite to be true. That Lean/Kanban DO work in software development, and especially on large projects where it's found to scale well.

Investment Commitment?? by chris lafosse

A key issue that I've yet to see discussed is how the corporate (and I use that word deliberately) investment commitment is gained for projects that have no estimates and not even an outline plan of deliverables and timeframes. Perhaps it's only smaller companies that are willing to go ahead "on a hunch" that this works for?
On a separate point - re. the "Pull based approach" (To pull instead of push is about responding to customer needs, instead of giving the customer what you think he will need (extra features). Surely a company marketing a new software product is going to want to incorporate innovative features that differenciate them from the competition? Where the approach is appropriate, surely it's just a case of doing proper collaborative requirements investigation, and not allowing developers to simply make things up?

Re: Investment Commitment?? by Tim Hodkinson

I think this is one of the fundamental misconceptions about what Agile development is: If you start a project with no estimates, outline plan for deliverables or requirements then thats just hacking and its certainly not what "agile" methodologies advocate, certainly not Scrum anyway. If you follow the Scrum you should have an overall release plan, scope is tightly controlled, requirements monitored and at all times you should be able to put an acurate cost on everything. What you are describing is cowboy coding, not agile development.

Re: Investment Commitment?? by chris lafosse

I agree it sounds like cowboy coding! - That's why I want to be better able to defend that it is not. Because, unless I've misunderstood, at, or rather near, the start of the project (once you've produced and estimated your Project Backlog) you only have Story points (which are notional, not 'meant' to be 'days') and no established velocity. So what do you do? Say "well, at velocity X using number of resources Y, we'll finish in Z sprints at a cost of £Ps" Now, that may be no worse than traditional estimating and planning, but it doesn't seem much better! And perhaps it's all we can do. But proposing NOT to estimate at all seems unlikely to help!
Just to give some backgorund, we've been using Scrum for a couple of years now, with pretty successful deliveries. Scrum has many good aspects that are working very well, but others that are still challenging. HHowever, I'm still intrigued by issues like this, and how to deal with them.

Re: Investment Commitment?? by Tim Hodkinson

Well I think you've hit on the $1000000 question. You are entirely right, when you start a scrum project your estimates can be no better or worse than a traditional waterfall one. I think the advantage potentially is the constant re-assessment/planning that scrum forces you to do, so initial estimates are honed pretty quickly and continuously. Having said that, "traditional" project management says you should be replanning every 6 weeks minimum too, so really what we're talking about here is just good practice.

Re: Investment Commitment?? by Essam Badawi

What I really like in this article is the way of thinking itself and the concept of continuous improvement in whatever process you are adopting is an important goal everyone looks for. Of course, getting benefits of ‘lean principles’( e.g., eliminating waste, JIT, ) is important and open the door for more innovative ways to achieve that goal. The article objective is not to provide an ideal solutions or so, but it rather to just give some examples of such different ways of thinking that we need to consider. Thanks,

Re: Investment Commitment?? by Robert Merrill

Those of us who do agile in project houses (I started in 2003) have been forced to address the early-estimate problem, maybe more so than IT shops. We combined rapid-requirements techniques (facilitated workshops), function point analysis, and the principle of diminishing returns, to get to a "good enough" estimate, as fast as possible.



Tom Hodkinson has it right:
when you start a scrum project your estimates can be no better or worse than a traditional waterfall one.

The difference is that Agile has re-planning built in, and the feedback on estimate accuracy is of much higher quality, and arrives earlier. Because of the Lean principles, especially minimizing WIP, agile methods are inherently resilient in the face of inevitable estimation error.



The techniques for early estimation are out there, in both agile books like Mike Cohn's Agile Estimating and Planning and traditional ones like Steve McConnell's Software Estimation: Demystifying the Black Art. They just don't get a lot of emphasis in Agile circles.

Re: Investment Commitment?? by Benjamin Booth

On day one, estimates with agile are no better than any other unless it's a new project with a seasoned, in-tact agile team. Either way, with agile, you can do some damn good forecasting with only two sprints under your wing. To do this...

At project outset, at least, begin with Story Point estimation. Having an experienced Scrum Master is critical. Also estimate Ideal Team Hours for each task associated with the stories you bite off in Sprints one and two.

Then, it only takes four to six weeks to establish a solid ratio of Story Points to Ideal Team Hours/Developer Day (Just divide Stories accomplished by the total number of Ideal Team Hours it took to accomplish them). For example, if you accomplished 27 Story Points and it took 171 Ideal Team Hours to do it, you'd have a ratio of 6.3 Ideal Team Hours/Story Point.

At this point it's also useful to know the number of Ideal Team Hours/Developer Day. This calculation is easy - add up the total number of developer days you had available, and divide by the total number Ideal Team Hours burned in previous sprint (or use an average for n past sprints).

For example, a team of four people, working for four weeks, might have used 20 x 4, or 80 developer days. Divide 171(ITH)/80(DD) and you get 2.1 Ideal Team Hours/Developer Day.

Then, at four weeks after kick-off, have the team do another estimate of total Story Points for any arbitrary 'complete' set of Stories you want to put into production. If you're an established business, perhaps you've let customer input on an existing product drive what 'complete' means. If you're a start-up, 'complete' may be some minimal set of innovation-driven features that gain you a competitive foothold in your market. Suppose, after this round of long-range Story Point estimating, you end up with 212 Story Points.

Using our previous calculation of 6.3 Ideal Team Hours/Story Point, it follows that 212 x 6.3 will require 1,336 Ideal Team Hours to get accomplished.

So, how many Actual Days will 1,336 Ideal Team Hours take?

Recall that we said we averaged 2.1 Ideal Team Hours/Developer Day in our first four weeks. So, 1,336/2.1 = 636 Developer Days (DD). If we have a team of four, divide 636 DD/4 Developers to get 159 Days or, assuming 20 day months, about eight months. I'd use this information to turn around and tell the business unit it would take nine months (to accommodate some estimation accuracy inaccuracy).

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

15 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT