BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles No Projects - Beyond Projects

No Projects - Beyond Projects

Bookmarks

The Problem with Projects and some ideas on what to do about it

Applying the project lifecycle model to software development complicates both and makes developing good software harder. Using the language, terminology and mindset of projects to discuss something that is inherently not a project creates conflict, confusion, short-termism and sets inconsistent goals. Controlling these conflicts requires manager intervention. The two models conflict and increasingly create problems.

When project thinking is to the fore the emphasis on project delivery has the unfortunate effect of obscuring the true goals - so called goal displacement. Teams and managers should focus on benefit delivery and value generation but under the project model instead focus on time, budget and requirement goals.

These chronic problems have existed for years but modern trends in software development are making project thinking more dangerous still. The large batch size nature of projects is increasingly in conflict with continuous delivery, hypothesis testing and small batch releases. This conflict increases risk and reduces benefits.

Show me the money

Project success is commonly judged on three criteria: time, budget and features (or sometimes quality). Success is not judged on the benefit or value delivered to the customer.

Focusing on these proxy criteria leads to what psychologists call goal displacement. Rather than aiming to delivering benefit, and the actual project goal, energy is directed at time, budget and features goals.

Underlying these criteria and the project model are a number of assumptions:

  • The benefit is known at the start.
  • Benefit is correctly assessed.
  • Meeting time, budget and features will deliver the benefits.

All these assumptions are questionable. Decades of IT project experience tell us clearly that it is difficult to hit pre-defined time, budget and feature criteria, and even when we do the anticipated benefit may not materialise.

This problem alone would be bad enough but in software there is another problem: quality.

Some literature fails distinguishes between features (i.e. functionality) and quality. More features do not make for more quality. Features and functionality can be traded off against time and money but...

Too many believe that quality can be traded for time - and therefore money. These people believe that lowering quality (e.g. accepting more bugs) will hasten completion. That might be true elsewhere but in software the reverse is true.

Cutting quality in software development - accepting bugs, foregoing tests, not refactoring or designing - slows development in both short and long term.

What is a project?

Before we continue it is worth considering the question: what is a project?

There are many definitions of projects. Common to all is the idea that projects have a start and an end. However what the start and end actually denote in a software project is debateable.

Projects are not born fully formed with requirements documents, project plans, staffing lists and so on. There is no Requirements Fairy who magics up a lengthy document on the day of approval. Work occurs before the project.

Formal or informal pre-project phases are common but is this work part of the project or not? Should it be factored into the cost benefit assessment? Since dual-track teams will incorporate much pre-project work into the project they may end up looking more expensive.

Then there is the end: when is the a software project ever done?

The nature of software is that is always more work that could be done. In fact, the more the software is used, the more useful and valuable it proves the greater the requests which will be made on it. Denying these changes is to leave benefit unrealised.

Only dead software doesn't change. If software is not executed, if it provides no service to anyone there will be no need to change it.

When software is used changes will be needed. Users will ask for changes, competitors will introduce new software with must be matched, world events impose changes and technology change will all create the need to change the software.

Software which doesn't change won't be used. It will die.

Look at SourceForge. There are tens of thousands of open source projects which never change and are never downloaded. If the downloads ceased the changes will cease, if the changes cease the downloads will cease.

Living software must change to stay useful. Finishing software doesn't make sense.

The end of a software project may mark the death of software, or a hiatus in changes before the next project. Applying the project model to successful software results in one project after another on the same code base, or rather the same product.

Software product companies have long known this - although they may still use the project model. As non-software companies increasingly find themselves dependent on software for their business operations and models they too will find the project model is doesn’t fit software development. Software development is for life.

The Temporary Problem

The next problem is that projects are inherently temporary while the software they create is not. This creates two immediate problems: quality and team.

Because a project is managed as a temporary endeavour the aim is to finish. This creates short-term thinking and approaches aimed at getting across a finish line with little sense of what comes afterwards.

Lists of outstanding known bugs are a visible manifestation of this problem. It is seen as acceptable to deliver software with a list of known bugs rather than fix the bugs. These bugs have been classified as “low priority” or “not critical”. In fact, as the deadline nears the pressure to down grade bugs increases.

Throwing the software over the project finish line and leaving the Bug Fixing Fairy to fix the problems is rarely a viable strategy.

Leaving aside the arguments this creates over what is a bug, a feature, a change request and which actually need fixing, the bug list is the most visible part of a bigger problem. The bugs are themselves symptomatic of problems which will lurk beneath the surface: code that is crying out for refactoring, code which was never tested, technical debt and other problems.

When meeting the deadline is the only thing that matters then perhaps it is acceptable to ignore these problems but in reality successful software will be worked on again. These problems will be paid for by those who come next. Software projects run up hidden debts which are rarely quantified or explained.

The debt is compounded when the temporary organization is dissolved. It is crazy to dissolve development teams just because the project has finished: the software has only just begun.

It takes time to build a team - storming, norming and forming before the productive performing. And all the while the team is learning, generating knowledge, knowledge that only truly resides in the heads of the team. Documents, videos and wiki’s only scratch the surface of the team’s knowledge and can be badly misleading. Such methods capture explicit knowledge, far more tacit knowledge remains in the team’s heads.

If a company has paid good time and money to build a team, and paid good money for the team to learn to work as a team, and paid for the team to learn and generate knowledge then why should they dispense with this team? Why throw away that investment? Why throw the knowledge to the four winds?

Those who take new roles in the same company will find their future disrupted by requests from the past. Worse still, much of the knowledge will walk out the door.

I call this approach Corporate Psychopathy: killing performing teams.

True, dissolving teams at the end of a project probably happens far less often in real life than it does in project management textbooks, but the idea that the team will be dispensed with is damaging. It creates conflicts while the team is working and especially when nearing the end.

Big things

The second characteristic of projects is the idea that projects are big. The project model considers big pieces of work. This work was decided in advance and a team tasked to deliver these items.

While I have heard of a company which classed every bug fix as a project in its own right this is the exception. Using a fully fledged project approach for small pieces of work is inefficient -  the time it take to initiate a project may be far longer than the time to do the work. Unfortunately project initiation and governance are applied equally to small and big projects.

The cost of setting up and executing project management structures and processes only makes sense for large pieces of work. Almost by definition projects are non-trivial, projects are a collection of desired functionality.

Requirements may be presented as a list of “must have” line items, or series of Use Cases, or a pile of User Stories. Judging success on how many of these are delivered by some future date may have made sense 10 or 20 years ago but not any more.

In a world where teams are constantly seeking out the highest value does it make sense to call a project a failure if some items are left undone?

When teams value responding to change over following a plan it follows that some items will be added and some foregone.

When teams are delivering multiple times a day, gradually adding to that value of the system they are building, is an end date anything other than a line in the sand?

And when teams have rich feedback from customers using the software, does it make sense to ignore the feedback in preference for a predefined requirements list?

When a team is delivering frequently, each release sound and usable, they are repeatedly offering stakeholders the opportunity to stop right there. Each release is a potential end. Why continue to the bitter end just to meet project success criteria?   More valuable work may be waiting in another project, why delay it?

When this happens sunk costs are no longer a consideration - not that they ever should have been. The question should not be “When will the project end?” but “What should the team be working on next to maximise return?”.

The further a team moves from the single, big-bang, delivery model the more the project model breaks down. And since the multiple small deliveries reduce risk and increase the value delivered why wouldn’t a team try to get as far from the project model as possible?

An alternative model

What is needed is an alternative model in which to think about work. The project model provides a mental template for considering work. What team members and managers alike need is a new model which allows them to reason about work.

Let me sketch the alternative model. I’ve been known to call this the Beyond Projects model although more descriptive names might be Continuous Work, or Steady State Work Model, or Work Streams - take your pick.

Steady elements

Software exists, sometimes new software is created but most development work concerns existing, living, software. It makes sense to organise around what is stable rather than what changes.

What is required is a means of undertaking lots of small pieces or work on this existing product efficiently. Call these small pieces features, functions, user stories, use cases, epics, backlog items, jobs to be done, user requests, bugs,  whatever you like.

These small pieces of work can come from, almost, anywhere. One can argue over where they *should* come from (users, customers, clients, ...) and one might attach different weigh to different sources but lets just accept: requests to do work can come from anywhere.

Lets also accept that these pieces of work can come at any time. Today, tomorrow, next year; before the team starts work, after the first deliver, six months after the last delivery. Unlike in the project model ideas will not be penalised for arriving late.

This is not to exclude some over arching aim, objective or Big Hairy Audacious Goal. Such an goal is a good thing, it provides direction and helps assess the work arriving. This goal may be set in advance or it might emerge as work progresses.

What the goal should not be is a list of things to do: it should be singular and general not specific and multiple. With a goal in place - one which will deliver business benefit - the team can work back to decide what is needed to achieve this goal.

Means of production

In a software development organization - and many others - the means of production is the team.

Teams need to be stable and enduring. A team may grow or shrink over time. Once in a while a team may be dissolved and occasionally a new one be created. Like successful sports teams the nucleus of the team stays together.

The teams should be staffed to deliver business value. The team needs all the skills to do the work they are responsible for - including identifying new work and measuring benefit delivered too. Dependencies on other teams should be minimised.

Teams need clear sight of how their work brings benefits to customers, users and the business. Team are organised along business lines rather than functionally. There is no database or user interface team. Teams deliver a product or service to the business. A saleable thing in its own right, or tools to support a line of business.

Businesses increasing deliver products and services which are inherently software dependent or which could not be delivered without a software as a result all business increasingly resemble software companies. The business is software. Business folks need to learn more about IT and to work with IT people.

Think of teams like amoeba, they are the cells that make up the bigger organization. Indeed Kyocera has pioneered and approach called Amoeba Management - described in a book of that name by founder Kazuo Inamori.

Designed for another knowledge based field: specialists ceramics research, development and production; there is a lot the software profession can learn from in Amoeba management for.

Another way of thinking about these teams is as Object Oriented teams. They contain the data (knowledge) and functions (capabilities) to do the work needed. Rather than constitute teams along functional or hierarchical lines - akin to layers of procedural code - teams are objects. Each object delivers an service.

Finally, the more a team can be cross functional, the more it can contain all the skills it needs, and the less it needs to call out to other teams then the more effective work will flow through the team. Each time a team stalls because of a call out then it is a sign of where the team could be better.

Sausage machine

The Team - requirements go in, software comes out

A team is responsible for multiple products. Work requests come in and the team actions them against one or more of the products for which it is responsible. At the end of the day the team produces working software which is used and results in business benefit - preferably hard cash but business benefits may be in other forms too.

As a result of doing work and the end product being used, the customers and users will have feedback to give. Some may be direct requests for work and some may, upon analysis, result in more work items. Wherever, whenever, work arises it is all possible, its just a question of selecting the most beneficial to do within available capacity.

I like to think of this model as a sausage machine. The team are the sausage machine. Meat goes in, sausages come out. If you put pork meat in you get pork sausages out, if you put chicken meat in you get chicken sausages out and if you put horse meat, well you could get anything!

While the sausages which come out of the machine are worth more than effort - and raw meat - the machine runs. When the cost of producing sausages is more than they sell for the team stops. The team is a business unit, change requests are not a source of problems (as in the project model), they are the source of value.

Management challenge

The scarcest commodity in business is senior managers time. Senior managers, even Jeff Immelt, CEO at GE, and Jamie Dimon, CEO at JP Morgan, have exactly the same number of hours in a day as you and me, 24.
As one goes up a management hierarchy the size of the decisions managers make (the amount they sign off) increases. The project model plays to this hierarch. Senior managers can make big decisions about big development efforts - which is why projects are big.

The problem is that if we start to optimise team for small work items there is a mismatch. Senior managers have little incentive to be involved in the decision making process: Big men make big decisions.

Orientating delivery teams around large pieces of work - projects - is efficient from a management stand point because it allows senior manager to “sign off” on big things.

The project model fits with the hierarchical control model, but  when teams work in the small need to make lots of small decisions about small pieces of work. Consequently the project model is in conflict with the needs of the team.

There needs to be a new management model. One which is optimised not for occasional, big, decisions but for many small and frequent decisions. The project model makes efficient use of scarce capital; software development needs a model which makes efficient use of scarce time.

The new model comes into two parts. The first part takes us back to the team. The team is the unit of delivery, the team needs the authority to make those decisions - and the skills to make them well - and the team needs to judged by their performance in delivering benefits - not features.

The second part of the answer is to revisit the governance model. Traditionally project governance has been conducted against time, budget and features. In a steady state, beyond projects, model governance needs to be conducted by looking at actual benefit delivered.

Governance through portfolio review

If each team is making sausages, and each team is claiming success - more sausages! - then how is the wider business to direct the effects? How is the business to know which teams are really successful and which just look successful? Just because each team is optimising its own efforts does not mean the organization as a whole it optimised.

And how is the business to change direction? Enter new fields? Withdraw from others?

Fortunately there is a ready made solution: portfolio review. It just needs dragging into the twenty first century (the post Agile world!)

Once an organization moves beyond one team it needs a portfolio review process that happens on a regular basis, say every eight weeks. Teams are reviewed by reference to:

  • Their effectiveness: are they actually delivering or have they become dysfunctional?
  • The business benefit actually delivered: preferably how much money they have made for the company but somethings may not have monetary benefit. Either way the team should be able to demonstrate the benefit they have created.
  • The business benefit they foresee creating: not a list of features they intend to deliver - although that might have its place - but rather how they see they can generate benefit in the next period and beyond.
  • Teams own requests - to grow, shrink, stop or change.

Teams which are not effective might still make grand promises of what is to come but if they repeatedly fail to deliver then the promises are worthless. On the other hand, teams which are effective but are restricted in the benefits they can deliver might be better off repurposed into another area.

These portfolio reviews need to happen as regular as clockwork:  reviews are not delayed because teams “are not ready”. These are not like stage-gates of old which can be postponed until a team is sure it can pass. And unlike stage-gates these reviews do not review proxy artefacts, they review results.

These reviews are more akin to TV’s Dragons’Den (Sharks’Tank in some countries). Each team comes to its investors says “Look what we have done, if you invest with us this is what we will return.”

The review doesn’t just look at each team in isolation - although it will do that too - they look across the teams. Are the teams going in the right strategic direction? Which teams are worthy of increased investment? And which teams need to be stopped?  Teams might be stopped, reduced or repurposed not because they are not effective or delivering value but because they are no longer fit with company direction, because they do not match the risk profile, and because investment and risk need to be balanced across the portfolio.

Governance through portfolio review is where the big decisions are made. Managers give a (limited) licence to develop until the next review.

This model should sound familiar because it is: this is an internal version of the venture capital model which has proved so successful in Silicon Valley. Investors, venture capitalists, review many start-ups. The lucky ones obtain investment by demonstrating both their ability to deliver and a market opportunity. Once the money runs out they must seek more funding from investors.

Most start-ups will fail. A few will become profitable. And a very few will repay the investors enough to cover the cost of all the failures and leave a healthy profit. VCs have a healthy appetite for failure and so should internal reviews. Initiatives need to fail fast and fail cheap, after they fail it might be possible to salvage some learning, some knowledge.

False Projects

In many development environments the language of projects is used, and people have a mental model of a project with all the trappings, but on closer inspection work is not following the project model.

I call this “False Projects” Typical symptoms include:

  • Teams may, or may not, have a requirements document in the background but much of their work is driven by fresh requests.
  • Teams may have an nominal end date, perhaps one that aligns with the financial year, but the date is continually extended.
  • Effort estimates are frequently updated, and Gantt or Burn down charts refreshed, but the end date is far into the future.

In a False Project the term project is worse than misleading, it creates conflict. Conflict which permeates upwards into portfolio review making rational governance impossible.

To managers the “project” is some abomination of the PMI and APM ideal which should be fixed. They may even exert effort in an attempt to bring the work closer to the true project model, but in doing so are likely to destroying what actually works.

When a False Project survives, and when it delivers benefit to user, customers and those paying for it then it has value. Judged on business benefit delivered it is a success.

Conclusion

The conflict between the software life-cycle and the project life-cycle has always created tensions. As successful software teams get even better - specifically as they adopt continuous delivery - then tension and conflict between the two models will become acute. It is time for a new model of work management.

Modern approaches to software create meaningful choices, options. When teams deliver regular releases they increase the options available to the business. Options have value. The trick is to capture this value.

The traditional project model is poorly suited to capturing this value.  The project model aims to create value by controlling uncertainty, software development is evolving new days of capturing value by exploiting uncertainty.

Let me suggest that, in the world of software development, the project model is not fit for purpose. The mental model created by the project model does more damage than good.

The project model belongs to another age. State of the art software development teams are now delivering - to live - many times a day. Businesses need to update their approach to managing the work and how they requests work.

The first step is to recognise which of your projects are genuine projects and which are false projects. Once the false projects are identified you can start by wrapping them in a more appropriate management framework and applying more rational governance.

The challenge comes with the genuine projects: those that are scheduled to end, those that are pursing the false goals of time, cost and features. Such work has been set up to fail from the start.

Today, delivering on schedule, on budget and on features is a sign of failure, not success.

About the Author

Allan Kelly has held during his carieer just about every job in the software world, from system admin to development manager by way of programmer and product manager.  Today he works for Software Strategy where he helps teams adopt and deepen Agile practices, advises companies on development in general and writes far too much.  He specialises in working with software product companies and aligning products and processes with company strategy. He is the author of three books: "Xanpan - team centric Agile Software Development", "Business Patterns for Software Developers" and “Changing Software Development: Learning to be Agile”; the originator of Retrospective Dialogue Sheets, a regular conference speaker and frequent contributor to journals. He can be found on Twitter as @allankellynet.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Sub-contractors, how to deal with ?

    by Erik Gollot,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Hi,
    first, thanks for this thought. How do you think we can do when a company uses lot of sub-contractors ? What does "being team" oriented mean ?
    How do you manage contracts ?
    How do you decide to shrink or grow a team ?

  • The most relevant article

    by Greger Hagström,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I struggle to find word to describe how relevant I think the message of this article is.
    Impressive insights and I wished I could describe to current state of projects as succinctly.
    Allan, I think you would enjoy a book by Patrick Hoverstadt Called "the fractal organization".

    Thank you!

    One thing I could add is that in my own experience the maintenance budget of most organisations i've been in is 2 times larger than the project budget. There are two separate organizations basically where we should only need one. -correctly designed.

  • How to transfer this into a real life corporate/customer setting?

    by Rolf Idar Isaksen,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Some food for thoughts this. Especially for me as a project manager. We need more PMs on board here or those that initiate projects. If we're going to take these suggestions seriously, what does this mean in practice? Traditionally you get funding for a project because you can demonstrate a business case an specific deliveries / dates. The alternative suggested here is VC model with portfolio reviews. How do you suggest this could be implemented in a corporate setting and with a customer contract? A budget is needed and some assurance what we get for this budget.
    In PRINCE2 and other models you get funding from phase to phase proving project live up to business case at end of each phase or else project is terminated. In SCRUM you have Sprint reviews demonstrating completed /runable modules and decisions what goes into next Sprint - progressing towards a complete "scope" defined at the outset.
    All these models are there so planners/sponsors can have some predictability on what they put into budgets. We need a well-explained alternative model to transfer from this traditional setting.

  • Interesting Article...

    by Steve McGinness,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    The general argument that underpins the article is sound, that the way we apply the principles of project management to software development actually hinders good software development rather than makes it better. But that is no the fault of projects per se, but rather of the application of project management to software development by those that do not understand projects, have not been trained, and lack the self awareness to understand their limitations. Giving up projects will not remove the underlying issues that developing software faces. We will still, inter alia have deadlines, we will still be asked to say when we will complete, we will still be challenged to deliver early, we will still be asked to cut corners, we will still be asked to quantify benefits, we will still be challenged to plan. None of these are pre-requisites for a project which after all...is a unique endeavour to meet a specific objective within a pre-determined time frame...but they define the level of formality that should be applied to your project.

    So next time you brush your teeth in the morning, just remember that you managing and delivering an activity to maintain and improve your oral health...or run a project.

  • quality

    by paul smith,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    is there ever any proof reading of the articles done before publishing them?

    English is not my native tongue but there are so many spelling errors in here that I stopped reading the article half way through which really is a shame.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT