Pulling Power: A New Software Lifespan

IT projects are conceived for many reasons, only some of which result in software that truly makes a difference. Software development is expensive, and as budgets tighten, it becomes more important to put the money where it matters most.

In this article we look at how Kanban metaphor from Lean manufacturing and the Feature Injection template play into Behaviour Driven Development, working together to help us identify the most important software, reduce unnecessary artifacts at each stage of development, and produce the minimum necessary to achieve a vision.

In the diagram below, is a description of a software lifespan that is described throughout this article. The artifacts signal each role to create further artifacts until the vision is implemented and the business value is earned.

The most valuable software hasn't been written yet

Dan North compares software to metals. "Generating reports," he says, "is like lead. It's easy to work with, and it's cheap. It's also heavy, worthless, and drags you down. Developers sink a lot of time into writing report software. Why? Buy it, install it, hack it so it works. Use Excel. Don't spend money writing report software, unless you're the kind of company that sells them to other companies.

"Then there's software which is harder to work with; which needs configuration and conversations with third parties, but which you can still buy. This is copper." He talks about supporting subdomains, a term from Eric Evans' "Domain Driven Design", and references the example of a company whose core domain is finding oil. "They need GPS systems. Not everyone needs those, but they do; and so do a lot of other people. You can buy GPS systems, then plug them into yours."

"So," I suggest, "if you have a need like that which isn't differentiating, but which nobody sells yet, you might want to find someone to help make a product out of it."

"Absolutely." He quarters a card and writes down:

"This," Dan says, "is plumbing. I talk to CIOs and ask them how much they're spending on plumbing. Too much!"

"So what's above that?" I ask.

"Up here [top left] are your quick wins. We have software that's easy to work with, and which touches your core domain. This is ideal ground for Agile pilots; visible, high-value projects that no one's done before. By their nature they're differentiating. These are valuable. These are gold.

"Then up here, in this [top right] quadrant, we have the platinum projects. These are hard to do. They're core, risky and may require interaction with third parties, difficult conversations, and they're also valuable differentiators."

David Anderson defines four focuses for business: commodities, differentiators, cost savers and spoilers. "It's inevitable, then," I say, "that someone will spoil your differentiators - your gold and your platinum projects. They'll take your idea - bypassing all the risk that you've taken, because they can see that it works - and implement it themselves, much more cheaply. That's why we need to be Agile here; because at some point in the future, we'll need to change, and to find new ways of differentiating ourselves."

Dan grins. "Chris Matts reckons there should only be three reasons for writing software: making money, saving money and protecting money," he says, "and that third one makes it interesting. LastMinute.com is a great example. They sewed up all the markets - theatres, leisure, anything they could think of that was perishable and that people wrote off before its perishable date - by drawing up exclusive contracts with all their suppliers. LastMinute was an obvious idea in retrospect, but by the time anyone came to spoil it, there were so few players left that the diminishing return wasn't worth it.

"Of course, platinum projects have a high cost of entry anyway, because of their complexity. So in that way they're worth more than gold."

"OK," I say. "So, ideally we'd release new software every week or two, and we'd respond to business needs immediately. We know, though, that someone's going to spoil what we've done, so, to protect their money, the business has a minimum feature set that's going to make the cost of copying prohibitively expensive. Also, necessary complexity is actually a good thing. These let us keep our differentiators… at least for a while. Creating a differentiator, and protecting it, gives us a scope to work with."

Now we know how to identify an important vision and its scope, we can look at what we need to do to implement it.

The vision pulls stakeholders to create feature sets

Once the vision has been identified, the original stakeholders work out who the other, incidental stakeholders for the project are by thinking of all the people who need to be involved. The stakeholders then consider what they might need for the vision to be achieved. We used to use this template:

As a <stakeholder>
I want <something>
So that <I can achieve some purpose>

I remember the Guardian editors using this, and coming to the BAs with what they called "begging letters for functionality". Chris Matts recognizes this problem. "You've put the stakeholder first," he says, "so stakeholders can think of things they want then justify them. Instead, we can put the vision first. That stops people from making things up that don't contribute to the vision, and helps reduce scope creep. It should be this…" I scribble down what he suggests:

In order to <achieve the vision>
As a <stakeholder>
I want <something>

The dependency of the vision on the feature sets is comparable to the way in which developers inject dependencies in code; because they're needed, rather than because they're a good idea - hence his term, "Feature Injection".

We sometimes find it useful to break down the vision into smaller goals - means to an end - which can always be tracked back to that bigger vision.

Additionally, the stakeholder who wants the story isn't always the one who has to use it. Captchas - automated Turing Tests using hard-to-scan graphics of letters - are a great example of this. "As a user, I want a captcha, so that… wait. Hold on! I don't want captchas; they're a waste of my time." We extend the template to read:

In order to <achieve some outcome which contributes to the vision>
As a <stakeholder>
I want <some other stakeholder> <to do, use or be restricted by something>

For instance:

In order to stop bots spamming my site
As the forum moderator
I want users to have to fill in a captcha to comment.

Some features may not even be software! Perhaps our software stories will sit alongside things like training, logistics, network administration, legal work, etc. Stakeholders often want outcomes that can't be met by software, or are better met without it, like LastMinute.com's exclusive contracts. We need to consider these, too, when we think of the vision.

These stories are usually too large for the development team to handle or estimate. Mike Cohn calls them "themes", and Feature Driven teams call them "feature sets". We'll call them that, too.

The outcome of each feature set - for example, that bots don't spam the site - may not be achieved by one feature set alone, or even by one stakeholder. We may find as we progress that other feature sets can achieve or contribute to the same outcome. When that happens, we can look again at our feature sets.

Feature sets pull BAs to create stories

Chris Matts has a theory. "You know what people want?"

"What?" I ask.

"What they ask for. And you know what they don't want?"

"What?"

"What you want them to have. You know what else?"

"What?"

"Once they get what they ask for, they want something else."

As we know from attempting to use waterfall methodologies for years, we rarely know everything up-front, and our attempts to get it right at first are imperfect. The stakeholders want something different. When we do eventually get it right, the stakeholders want something more.

Usually, the context in which we decided on our stories and feature sets changes over time, and especially during the lifetime of the project. The only way to know if our understanding of the project is accurate is to get feedback early. The best way to get feedback is to give the stakeholders something that they can play with, so they can see the journey which a user or other consumer might take and refine their initial ideas.

To do that we can strip out everything that isn't essential to the vision, even if it's part of the minimum feature set needed to deliver a commodity or protect the differentiator. For instance, we could have fixed delivery charges for a shopping cart. Features for changing delivery options or discounting large orders could be part of a later story.

Think about almost any online store. A feature set which says:

In order to sell more products
As the head of Web Marketing
I want consumers to order goods online and have them delivered

Can be split into stories like this:

In order to sell more products
As the head of Web Marketing
I want consumers to be able to order some goods

In order to sell more products
As the head of Web Marketing
I want consumers to be able to put goods in a basket and order the basket

In order to sell more products
As the head of Web Marketing
I want consumers to have a choice of delivery options

In order to sell more products
As the head of Web Marketing
I want to offer free shipping when orders are over a certain minimum value

As soon as it is implemented, that first story allows us to get feedback from our stakeholder on aesthetics, etc., and lets them think about other stories such as placing adverts for related products, which might make a difference to the success of the vision. The other stories can each be added later, and we can get feedback separately on those.

It doesn't have to be a BA who refines these stories, though they're usually good at it.

Stories pull QAs to create scenarios

When a QA goes to test a new feature, they'll usually do three things:

• Set up some state, data etc. which the application can work with
• Perform some steps in the context they've set up
• Look for an outcome.

Then they'll set up a different context, and look for a different outcome.

We can use Behaviour Driven Development, and the language of scenarios, Given, When and Then, to define these. This helps when it comes to automating the scenarios.

Eric Evans, author of "Domain Driven Design", once asked what the difference was between a scenario and a use case. "You can't ask a business person for a use case," I replied, "unless they're already technical enough to know what a use case is. But you can ask them for an example. You can say, ‘Give me a scenario in which that happens.'" The language of BDD, like the ubiquitous language of DDD, is designed to enable conversations between the business and the developers.

"Given a shopping basket which is over the free shipping limit, when I go to checkout the basket, what should I see on the screen?"

"Oh… Shipping is FREE with this order!"

Through conversations, we find out new information. We can use that information to write an example:

Given a shopping basket with contents worth £150
When I go to the checkout screen
Then the screen should say "Shipping is FREE with this order!"

A QA can use this as an acceptance test. A QA or developer can automate this.

It doesn't have to be a QA who defines the scenarios. However, in my experience, they're very good at it.

All of this helps us ensure that our code stands the highest possible chance of being valuable when we start to implement it.

It doesn't have to be a QA who draws out these scenarios, though they're usually good at it.

Scenarios pull UI Experts to design the UI

I once facilitated a retrospective in which the developers had identified the UI experts as the biggest hold-up to their stories. "It takes three days from the first time we look at the story to actually being able to start work," one of them said.

"That's how long it takes to define what the page should look like," our UI expert replied.

"Hmm," I mused. "Is there a way of getting that time down? Maybe providing just enough for the developers to get on with?"

"Maybe. What do you need?"

One of them grabbed a card and drew a quick picture on it. "Something like this," he said, "just showing what's on the site. As long as we've got the content, we can write the code so that the format is easy to change."

"That's good," the UI expert smiled, "because we rarely get it right first time."

So, while the UI expert is busy refining the interface, the developers can do something simple which works. The look and feel of the interface can even become a separate story. That content, though - what should be on the page or screen, and which buttons a user should have to click to make it work - defines the first piece of production code that a developer can write.

A UI may not be graphical. For instance, an application may be used by another system. No human ever sees its output directly. The other system becomes the User, and the UI expert will know what kind of messages the consuming system needs.

It doesn't have to be a UI expert who designs the interface, though they're usually good at it.

The UI pulls developers to write code

A graduate developer, Jerry, asked me, "How will I know that my code works?"

"How will you know that it doesn't work?" I asked.

"Someone will report a bug," he replied, looking at the QAs. "It might be them, or maybe it will be the users."

"How will they know there's a bug?"

"They'll use it, and it won't do what they need it to do."

"Right. So, the only way in which we know that your code works is through the user interface. Whatever is underneath that is only there to support the behaviour of the UI, and to allow us to easily change that behaviour if we need to."

The UI is the only mechanism through which a user can experience the value of the software. Creating an interface through which code can be used is essential for the code to have any meaning.

By creating the UI first, and stubbing out or mocking any classes it needs, we can quickly get feedback on whether the UI matches the business's expectations. We can have conversations about changes we might need to make, and we stand a better chance of writing software that will be valuable. If the UI is used by another system, we can check that the two systems can communicate.

Code pulls developers to write more code

Jerry coded the interface. He kept it very slim, and stubbed out the other classes that the interface needed. Then he thought about what to write next. "So, I'll need a database table that has three columns…"

"Hold on," I suggested. "Do we need a database table yet? Do we have anything which can use a database table?"

"Well, we'll need to create a user, with a name, email address and user id."

"What for?"

"So our users can register on the site through the registration screen."

"What's the registration screen going to use, to help it register a new user? Is it going to do all the work itself?"

Jerry shook his head. "That would mean putting a lot of code in one place. It would make it very hard to change and maintain, and testing UI classes is difficult so we try to keep them small. I know we should use another class. Design patterns that usually work well call it a controller, or a presenter."

We know we're going to need to change the code at some point; either because, as Chris said, "People want something else", or because the differentiators for the business have changed. "So the UI is going to assign the responsibility of doing the actual work to something else. Do we know if our controller is going to need a database table yet?"

"No…"

"All right," I said. "Let's think about what the controller should do."

We separate our code into different classes, or modules. An appropriate way to do this is to think of the Single Responsibility Principle. We can ask ourselves, "What should this code do? What should it not do? What can be made the responsibility of another class? How would we use that class?" If we do this for all the code, it makes all the code easy to change.

At a unit level, the language of BDD is useful again. We can use "should" to help us pull out those responsibilities.

"So we know now how the controller should behave, and how it should use its collaborators - the user repository and the user information," Jerry said. "We also know how it should respond to events from the UI."

I suggested, "We can write an example of a fake UI class using our controller, describing the benefit we're looking for from the controller when we use it. This will help us to write the minimum code which is valuable to the UI. As a useful side-effect, it also provides us with unit testing and documentation of our code."

We write our example, or unit test, first. We use our IDE to create missing classes and methods, because it's quick. Domain Driven Design keeps the design and language of the code aligned with the business, and BDD helps us focus on the behaviour, the value we're looking for and examples of how to use our classes. This helps us have useful conversations with the business whenever we find out something new, strange or missing, and because the design of our code is aligned with the business, changes to the code will be proportionally as easy or difficult as the changes to the business process that cause them.

When we've finished coding each class, we move on to that class's collaborators, until no stubbed classes are left without a real implementation. By doing this, we ensure that we only code what's needed to deliver value to the UI. We don't code methods that we think might be needed later or columns in a database because it makes sense at the time. This also helps us simplify our design.

A good design lets us change our code easily as we find out more about the business processes, giving us multiple options for providing the value that the UI needs. As Chris says, "Options in the financial world have a value, and an expiry date. Real Options - options in the real world - have value, too, and a date at which the decision is no longer optional. Sometimes it's worth paying to delay decisions, and keep those options open." Good design takes longer, but is worth paying for because it lets us delay decisions or change our minds when we have better understanding.

This is why we use good design practices and automated feedback loops like unit or system tests; to keep things easy to change.

It doesn't have to be a developer who writes the code, but if you're writing it, you become a developer by definition.

We do enough to support the next most important thing

Let's try an example.

We have a simple vision: a noughts and crosses game. Let's pretend, for the purposes of the exercise, that no one has ever played this before, and no one has watched War Games. We have a big feature set, titled "Rules of the game".

Now we can draw a story from that feature set.

In order to enjoy playing Noughts and Crosses,
As a player,
I want one of us to win when we get 3 in a row.

Here's a scenario from the game, "Noughts and Crosses", automated with JBehave, and the UI which matches it:

Given a grid that looks like
OO.
XX.
X..
When the player clicks a3
Then the grid should look like
OOO
XX.
X..
Then the message should read "O wins!"


And here's an example of what the GameModel class should do:

@Test

public void shouldNotifyObserverWhenTheCurrentPlayerWins() {

    // Given a game which X is about to win

    GameModel game = new GameModel();

    game.playerActsAt(0, 0);

    game.playerActsAt(1, 0);

    game.playerActsAt(0, 1);

    game.playerActsAt(2, 0);


    MyGameObserver observer = new MyGameObserver();

    game.addObserver(observer);


    // When X wins the game

    game.playerActsAt(0, 2);


    // Then we should see X is the current player

    // and that Player X has won

    ensureThat(observer.game.currentPlayer(), is(Player.X));

    ensureThat(observer.game.message(), is(“Player X wins!”);

}

Once we've written this, we can write the code - the GameModel class - to help us make this example work.

In this example, we've mocked out the GUI's role as an observer with our own class created just for the example - MyGameObserver. By mocking out any collaborators that each unit needs, we can keep the examples decoupled from those collaborators, including those that don't yet exist. This helps keep things easy to change.

We have no score. We don't know whether X always gets to go first. We don't know what happens after the game has been won - we might have to restart the application. We've done just enough to support that code, for that scenario, for that story, for that feature set, for the vision. We've kept it easy to change, so we can always add the code for more scenarios and stories later.

When we're done, we release!

When we run out of collaborators that don't have real implementations, we're done with the code for that scenario.

When all the scenarios for a story run, we're done with that story.

When all the stories that contribute to a theme or feature set are done, we're done with that feature set.

When the stakeholders confirm that all the feature sets for a vision are completed, the vision is ready for release.

At every stage we try to get feedback as quickly as possible

By automating our examples and scenarios, we find out if our code doesn't work.

By automating integration with other systems and deployment to production or a similar environment, we find out if there are any technical issues we didn't think of, or perhaps any other stakeholders we should have considered.

By showcasing the application to the stakeholders as soon as we can, we find out if we can achieve the vision.

There are many other feedback loops in Agile and Lean methodologies. These mean that we can start working with the assumption that we might have got something wrong. We don't need to have everything perfect to start with - we can start with "good enough".

At every stage we try to minimize waste

Lean manufacturing tries to minimise inventory - parts of cars that haven't yet been added to a working car. In the same way, we try to minimise the backlog of stories, scenarios, examples and code which haven't yet been added to a working feature and thence to the vision. In Kanban systems, signals are passed to each phase so that each station knows to produce something specific for the next station to use. Each part is provided because a station needs it to fulfill customer demand. This is a pull system.

In some production systems, production lines have boxes of spare parts lying around in case they're needed. People need to work around them. The parts also need maintenance, security, space, etc., and if the business needs change enough, the parts have to be scrapped.

In some software projects, we mirror this. We create feature sets because we have budget to spend. We produce detailed analysis of stories we believe we'll need in three months. We write screens for features we thought were interesting. We create database columns and tables to store information that nobody uses, and more screens to gather the information. All of these artifacts require maintenance, have to be worked around, and scrapped if business needs change.

By only producing the artifacts that are needed, just in time for them to be used, Lean production lines can avoid this waste. Similarly, we can avoid the waste in our projects by creating artifacts only as they're needed.

And this is our pull software lifespan

The most valuable software hasn't been written yet.
The vision pulls stakeholders to create feature sets.
Feature sets pull BAs to create stories.
Stories pull QAs to create scenarios.
Scenarios pull UI experts to design the UI.
The UI pulls developers to write code.
Code pulls developers to write more code.
We do enough to support the next most important thing.
When we're done, we release!
At every stage we try to get feedback as quickly as possible.
At every stage we try to minimize waste.

The roles filled by each player in the lifespan don't have to be full-time - a developer could also play the role of a BA or QA, for instance. In that case, he would be wearing a BA hat. It's almost impossible for someone to pull the artifacts that he'll need for the next stage - eg, the stories - without thinking about what he's going to pull from that stage - eg, the code - so it may be worth getting different people to play those roles.

As with any methodology, there are other ways of achieving the same goal, ways of adapting this, and other pull systems that may work for you.

When Dan North introduced BDD, he said, "Programmers want to know where to start, what to test and what not to test, how much to test in one go, what to call their tests, and how to understand why a test fails."

This outline should help you know where to start, what to work on and what not to work on, how much to do in one go, what to call the things you're working on, and how to get feedback at every level of the software lifespan; so that whatever software you write, it's software that matters.