Laurent Bossavit: Agile Ten Years On
This article is part of the Agile Manifesto 10th Anniversary series that is being published on InfoQ.com.
What's all the fuss about this "ten-year anniversary of Agile"? Is it just that we like nice round numbers, or is there something more to it?
The field of computing is still young enough that ten years are a significant time for anything to have been kicking around. Ten years, in our business, is just barely enough for something to have a "history".
How much do software professionals know or care about history? Nowhere near enough. I have only been in this profession for twenty years or so; you may not want to take my word for it.
"After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history." Those are the opening words of an essay by Jerry Weinberg, who was around when IBM (founded a century ago) put him in charge of Operating Systems development for Project Mercury (over fifty years ago).
Everyone knows the saying, "those who do not learn from history are condemned to repeat it". (But do we take it to heart?) Weinberg's essay is an updated version of one he wrote twenty years ago, right after the notion of Structured Programming hit its "peak of excitement". Chillingly, his remarks about the shortcomings of the Structured Programming movement remain relevant under merely substituting "Agile" for "Structured Programming".
History matters, if only to avoid being taken in by distortions.
As an example of distortion, consider the "creation myth" we have invented around the origin of Agile's arch-nemesis the big, bad Waterfall. The Waterfall methodology, the story goes, was widely adopted after people tragically misinterpreted a 1970 article by Winston Royce. Page 1 of the article contained the now-famous diagram of a software project modeled as a series of sequential phases. But this was only the first in a series of increasingly complicated diagrams which only slowly revealed what Royce was actually trying to say; and readers - you know how busy people are - tended to not go past page 1 and overlooked Royce's warnings on page 2: "I believe in this concept, but the implementation described above is risky and invites failure".
The problem with this (admittedly seductive) story is that Royce did not originate the sequential-phases model of software development. The careful reader of historical documents, for instance the proceedings of the 1968 NATO conference which founded the discipline of software engineering, will find ample evidence that this was already the prevailing model.
An accurate understanding of history, and a healthy skepticism of appealing myths, are important components in making actual progress.
Today I am often dismayed at how little attention is given even within the Agile community to recording the history of the various ideas that make up its "body of knowledge".
For instance, those who have studied Agile a little bit would know that "Agile is ten years old" is itself a distortion. The roots of Agile go back well beyond the February, 2001 meeting in Snowbird. Both Scrum and Extreme Programming had been around for the best part of a decade already at that time.
The most dangerous distortion of all is the notion, sadly prevalent today, that Agile consists of a handful of competing "brands" of processes: Scrum versus Extreme Programming versus Lean or Kanban. In the interest of pushing this or that brand, the interesting details of our young history are papered over.
Thus, standard presentations of Scrum tacitly encourage the student or reader to believe that "Sprint Retrospectives" are now, always have been, and forever will be a part of Scrum. In fact Retrospectives are a relatively recent addition to Scrum canon, which they entered by way of Extreme Programming, where they became popular thanks to a small number of individuals who became enthusiastic about Retrospectives following the publication of Norm Kerth's book, Project Retrospectives, in 2001. The "normalization" of Retrospectives as a canonical Agile practice can be dated back roughly to 2006.
An accurate history of Agile is hard to reconstruct because of the bottom-up nature of the movement: its expert practitioners tend to act with creditable disregard for canons or central authorities. They pick up ideas because they find them interesting and fruitful; they test those ideas in the laboratories of their own projects, keep what works and discard what doesn't. They are often unconcerned with priority or paternity.
Agile in 2011 is no longer identical with Scrum, or with Extreme Programming, or with Lean Software Development or with any of the "brands" that visibly claim to be the vanguard of Agile. (That, to, me, is the significance of the 2001 meeting, only clear in retrospect: it marks, not the start of the Agile movement, but the beginning of the downfall of these brands.)
However, it is still hard to know exactly what to make of Agile-of-2011. The nice, round ten-year number tempts us into making up stories that this anniversary marks some kind of turning point; that "Agile has arrived", for some value of "arrived".
I am among those who see this as a form of whistling in the dark.
The value in an accurate understanding of history is that we can assess how much has actually been done, and how much still remains to be done.
How much: the Agile movement has succeeded in overturning many previous orthodoxies. It cannot entirely be credited with the demise of Waterfall; this was already well under way at the time of Boehm's Spiral model, introduced in the late 1980's. I am thinking more of the renewed respectability of "developer testing" - back in the 1970's, and I think largely due to Glenford Myers' books on software testing, the notion that "a developer should never test their own code" became entrenched, received wisdom. The practice of test-driven development, originally formulated in the context of Extreme Programming, but applicable to the vast majority of Agile projects, showed that to be a counterproductive overgeneralization.
Similarly, Agile has recast in a new light many long-standing questions about software design, project scheduling, or the management of programmers. This is not the place to go into the details of these upsets; but I do want to suggest that Agile is a different discipline from either of software engineering or project management, not reducible nor fully belonging to either. There is great promise in this discipline, when it finally reaches its full development.
However we must also be honest about how little has really been accomplished. As Weinberg remarks in his essay, most teams today claiming to "do Agile" are only half-heartedly executing a diluted and poorly understood subset of its practices.
It is easy to rail against "half-assed Agile" and "ScrumButs" and give in to the temptation to blame the victims - to hold it against the students that they failed to understand something that was poorly explained, rather than hold the master responsible.
At this point the Agile movement is still just that - a movement, a community. We are doing, I think, fantastically well at putting up conferences and unconferences and workshops and kicking up new ideas and cross-fertilizing each others' thinking about software. But much less so, yet, at settling down and consolidating, at being accomodating to "newbies", and making even a small dent in the huge task of changing what is still, right now, being taught to software engineering students in universities worldwide.
My hope lies in the Agile community's ever surprising capacity to innovate. Specifically, I think that what is needed is a new generation of Agile institutions - not to replace but to work alongside the Agile Alliance and Scrum Alliance and the many smaller but no less necessary groups putting in the hard work of organizing a conference, an unconference or a Coding Dojo. For instance, institutions taking up the challenge of working more effectively with the research and education communites.
Yes, ten is a nice round number, but Agile at ten is still far from rounded out. Here's to the work that lies ahead - to the next ten years of Agile.
About the Author
Laurent Bossavit was an "early adopter" of Agile, stumbling onto Extreme Programming the year before the Snowbird meeting. He was a recipient of the 2006 Gordon Pask award for contributions to Agile practice. He now heads Institut Agile, a privately funded, independent entity whose missions include growing the Agile business ecosystem, creating stronger links between the business and research communities interested in Agile approaches, and providing stronger empirical evidence on the benefits and limitations of Agile practices.
Paradigm shift - Agile and Don Reinertsen's work
Thanks for this thoughtful article.
... and making even a small dent in the huge task of changing what is still, right now, being taught to software engineering students in universities worldwide.
This is a very important point.
The Agile movement got a lot of things right. In fact, as you say, the essential elements had already been around for much longer than ten years.
The Smalltalk group at Tektronix had already been developing software in a very "agile" style in the early nineties, probably even earlier.
In 1986, Takeuchi and Nonaka had already shown that what they called the rugby approach (and what soon came to be called the Scrum approach) is exceptionally successful in product development. In 1993/94, Jeff Sutherland, John Scumniotales (who is not mentioned in the new Scrum Guide) and Jeff McKenna (who is mentioned; he was also a link to the Tektronix folks) at Easel applied the Scrum approach to software development. (This was the first deliberate attempt to do this. A similar style of software development had been described by Jim Coplien in a paper that led to the introduction of the Daily Scrum meeting at Easel.)
Much of Scrum as we know it was fully formed in 1994, and results were outstanding. Recently, I did some research on the early history of Scrum; a brief summary can be found here:
So "Agile" works. What the Agile movement has not been that great at is explaining why it works, in a way that is fit for an academic context. However, I have high hopes for the work of Don Reinertsen. The Principles of Product Development Flow is not light reading, but this is the strongest attempt to date to attack the waterfall paradigm in product and software development. This is a book that is definitely be suitable for business schools and engineering departments.
From the beginning of the first chapter:
I believe that the dominant paradigm for managing product development is fundamentally wrong. Not just a little wrong, but wrong to its very core. It is as wrong as we were in manufacturing, before the Japanese unlocked the secret of lean manufacturing. I believe that a new paradigm is emerging, one that challenges the current orthodoxy of product development. I want to help accelerate the adoption of this new approach. I believe I can do this by helping people understand it.
.. and making even a small dent in the huge task of changing what is still, right now, being taught to software engineering students in universities worldwide.
Sw is one area where most of the teachers have no significant real world practice. Imagine learning to drive with a teacher that just had a lot of car catalogs from all brands and considered using hands and eyes as "implementation details". Everywhere where developers have been liberated from preconcieved ideas about how sw development should proceed (aka "The Method") agile practices appear as natural techniques. If we believe what others say this has been so since the fifties.
Don't get me started on rotten Academia ... At last, with Agile, open source ("... but it moves") and developer coordination in general, we are starting to reverse their negative effect.
To this day even de-facto sw working categories reflects this pyramidal model. To those of us that have lived something, documents pale in comparison. Let the academics disscuss the scriptures. Personally, I put some of my time to read historical documents about software as part of my ongoing education, but not to prove any point. I do not have a tool or methodology to sell.
I respect those that want to make Agile more palatable to Academia but I feel this is like trying to convince Creationists that Evolution has a point ...
Structured programming wasn't a fad
However, if you substitute agile back to structure, you see... Hey, but SP guys were right! Nowadays, we don't write non-structured programs, full of goto statements. i don't claim GOTO doesn't exist anymore, of course it's needed in kernel levels, but in general, we pushed non-structural optimizations behind the reach of the average programmer (optimizing compilers and os kernels).
On one part, SP solved the problem it tried to achieve, on the other part it didn't. SP wanted to bring comprehensibility to projects beyond the capabilities of a single "human skull", as Dijsktra said.
Nowadays our software systems are magnitudes larger which would have been impossible without SP.
But when you get a codebase for maintenance, doesn't matter if it was written just one year ago, or it's decades old: it's still incomprhensibly chaotic most of the time.
Personally I don't think that we have the right to say that Agile did anything with software design. In practice, it throwed such out completely. We are yet to know how good idea that was.
As for education, I believe some people had more horrible teachers than I had, and perhaps their scheme of education was wrong. My university always applied a pattern language-like style of education, continously reiterating the problem-context solution trinity until they were able to show us why the thing in question (notable examples being rup, xp, uml, opengl, .net and winapi) were conceived the way they were.
They also showed the wrong spots with this but it made us able to recognize and apply working patterns to given situations; some patterns are forgotten nowadays, as the "academia and se is complete shit" locomotive of agile is in full swing, but the context where they should have been applied didn't get lost. Perhaps in the next cycle...
NATO Software Engineering Conference 1968
The 1968 NATO Software Engineering Conference is also interesting, because it also did not represent a categorically pro-waterfall view either. Instead, it reflected the number of different approaches that were around at the time. Take, for example, the following quote from Alan Perlis in the conference proceedings:
"A software system can best be designed if the testing is interlaced with the designing instead of being used after the design. [...] Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself."
Glenn Vanderburg has highlighted this particular passage in some of his talks (including www.infoq.com/presentations/Craft-and-Software-...).
"An accurate history of Agile is hard to reconstruct because of the bottom-up nature of the movement: its expert practitioners tend to act with creditable disregard for canons or central authorities. They pick up ideas because they find them interesting and fruitful; they test those ideas in the laboratories of their own projects, keep what works and discard what doesn't. They are often unconcerned with priority or paternity."
Does the word "agile," especially when capitalized, add value? Does it mean anything at all?
Deployment Automation 101IBM