Liz Keogh: 10 years of Agile - the Prophecy of Failure, and the Failure of Prophecy
This article is part of the Agile Manifesto 10th Anniversary series that is being published on InfoQ.
For the first seven years of my career, I worked on projects which still retained much of their Waterfall habits. For seven years I sat at computer screens and tapped out code - in basements, late into the evening, sometimes at weekends - and not one thing I wrote in those seven years made it to production. Projects had endless reams of features added, were restarted, were shelved, and in one case made it as far as litigation - but no one used a thing that I wrote in those seven years.
So when I came across Agile for the first time, I was, like a lot of newcomers to the movement, enthusiastic. "You're doing it wrong! That won't work!" I promised, looking at companies who had failed to embrace the new, incremental, collaborative way of creating software. It was the prevalent attitude of the community at the time. A prophecy had been written: that companies who failed to change would fail to survive.
Ten years have passed since the manifesto was written, and still many companies are failing to deliver effectively. Some of those companies are "doing Agile". Others are still surviving with their year-long projects and up-front analytical methods intact. But they're surviving.
The Agile manifesto starts, "We are uncovering better ways of delivering software by doing it and helping others do it." For the contexts in which the authors were applying Agile methods, they worked. Mostly these were small, co-located teams with the right mix of skills to succeed. Over the last few years, increasingly we've applied Agile to larger projects and organisations. These Enterprise behemoths change more slowly, if at all. Richard Durnall wrote a post on Agile adoption patterns, showing how each aspect of the business breaks and is forced to change to support the Agile adoption - the people, tools, governance, customer, financial controls and organisational structure. More often, though, we're seeing the structure of companies hold firm against the change process. People retain Waterfall habits and thinking, creating deadlines that are often arbitrary and holding development teams to them. Tools are used instead of conversation, rather than to record and support it. Governance bodies still ask for certainty that's impossible to achieve over the lifetime of a project. Customers sit in one office with developers in another, unable to properly communicate or give feedback. Financial controllers insist that analysis is done up-front, theoretically ensuring a good return on the investment of the budget.
Part of the problem we face is that traditionally, Waterfall has involved "bicycle thinking". It starts from the premise that a software project can be decomposed into small pieces of analysis, which can then be implemented, and those implementations reassembled into a system.
Unfortunately, a "software project" isn't really made up of the analysis or its resulting implementation. It's made up of the people involved in the project, their ideas and interpretations, in a context which has been created by other people around them. Its properties are not those of the people, though. As Jurgen Apello points out in his book, "Management 3.0", the properties of a system emerge as a result of the interactions of the elements within the system. Agile can't succeed as long as the environment stifles interaction or rewards simpler interactions like those up and down a hierarchy.
If complexity is going to be happen anyway, we have to allow those patterns to emerge from the interaction of people on projects, and from the interaction of those projects themselves. We're guilty, as a community, of signing up to "individuals and interactions over processes and tools", then mandating processes to control the interactions, while supporting the processes - and not the interactions - with tools. In future, the practices we teach will be those which enable interactions, rather than controlling them. We've seen this already with the rise of metaprocesses like Kanban. Models for understanding complexity, and particularly the complexity of people, are also being taught in Agile conferences worldwide - Systems Thinking, Complexity Thinking, psychology and sociology.
These are also the kind of practices we need to change behaviour at higher levels in the organisation; to make the impact of anti-patterns apparent. We haven't seen as much change as we prophesied at the beginning. Maybe over the next ten years, we'll see a different manifesto emerge - one which starts, "We are uncovering better ways of enabling change by doing it and helping others do it."
About the Author
Liz Keogh is an experienced Lean and Agile coach, trainer, blogger and well-known international speaker. Coming from a strong technical background, her work covers a wide variety of topics, from software development and architecture to psychology and systems thinking. She is best known for her involvement in the BDD community, and was awarded the Gordon Pask award in 2010 for deepening existing ideas in the space and "coming up with some pretty crazy ones of her own".
Interactions don't scale well
I hope that most of the companies who are doing "waterfall" are in fact doing RUP. RUP is an incremental method just like the agile ones, only it's built on promises of outputs rather than individuals and promises of prophets.
Human systems don't scale without processes.
Our society has laws. It has bureaucracy. Basically anything containing more than 10 000 people will have a lot of bureaucracy involved for good reasons. They do so since ancient times. They'd even do it if we'd throw 10 000 newborns in a tropical island without adult contact. We invented writing to write contracts actually.
(Oh, and by the way, most part of IT is about actually supporting that bureaucracy: accounting programs, governmental information services, ERP, etc.)
I don't believe that a large system cannot be run well. It might have mistakes of course, it will surely have downsides, but pretending that the world just has to turn around just because something works with at most 100 people - that's just doesn't help.
Maybe, just maybe, we should accept that we're humans. We should accept, that like insects, humans also tend to create these hierarchies themselves, in order to better understand the world.
I don't say you shouldn't step up. I don't say you should shut up and deal with only the role you're playing. Not at all, not in the furthest sense.
Just accept that you won't be able to deal with 100 000 human beings individually, you have to simplify sometimes.
Yes, we have to be careful with those simplifications, and those who are part of it should step up if something is wrong with the model.
Yet 'agilizing' everything, trying to do what works best with 100 people scale up to 100 000, this is just nonsense.
Waterfall is reversion to the mean
My current project is part of a programme for a large company which started as an ernest attempt to adopt agile practices 18 months ago, and has since then been an inexorable retreat back to the usual (distinctly uncomfortable) comfort zones of fixed price + fixed scope contracts, ever more up-front planning, obsessive grooming of the project plan and documentation in lieu of conversation etc. While I'm sure much of this is due to an influx of management with no software (let alone agile) experience as the project has grown, the main driver seems to be an institutional need to arse-cover and apportion blame above all else.
Since these traits seem to be extremely common in large organisations (Anglo-Saxon ones, at least) I suspect that agile adoption at that end of the scale is sadly going to continue to falter.
Re: Waterfall is reversion to the mean
Let's see a company, which "successfully adopted agile". I can't name it for NDA reasons, it's a famous company, with shockingly falling stock prices ever since it did it.
A daily life: about half of your week is on different agile meetings, standups, scrum of scrums, retro, etc. Nothing ever gets written down, at least, not in digital form.
Administration is on post-it boards. No joke. Only problem that there's noone responsible to make sure they reflect reality, so they don't.
When you want to refer to a meeting - sorry, there were no meeting notes. When you want to solve an urgent issue on a meeting where everyone was finally collected - sorry, this doesn't fit the current meeting "type".
Responsibility: since there's group responsibility, everyone is trying to do their bests, and they claim they do. It's only that somehow the product going out is as close to unusable as possible. However,it doesn't fail its automated acceptance tests (even if goes out sometimes with bugs which makes it unusable for weeks). It doesn't bother anyone too much: the product itself is soo much beyond repair that noone uses it ever in the company. The only problem that we give it to the customers. When asked, it's perfect, it doesn't fail any tests and doesn't have bugs in it.
Interactions: any time an idiot from the design department can come to you and request some stupid, crazy, baseless idea. Worst: they can handle photoshop better than you. If you show it with UML or with any other method, that this is just complete nonsense, the only problem: the photoshop version shines better. So you ought to do it. Most programmers don't get bothered by this as they don't have to suffer the product itself - they use the rival products for years.
The product itself is a consumer product. You didn't buy it because you heard it's sh.t, yet billions use it every day. Those are non-geeks.
Before Agile was introduced, even geeks liked their products: it was designed by waterfall, yet it worked mostly flawlessly, and your grandmother could use it just as well as yourself.
Agile is not an exponential function, but a gaussian one. The other end is sh.t as well.
Re: Waterfall is reversion to the mean