Large-Scale Agile Design & Architecture: Ways of Working
During my 2011 QCon London keynote on "Scaling Lean & Agile: Large, Multisite or Offshore Delivery", I mentioned — as an aside — that, "Architecture is a bad metaphor. We don't construct our software like a building, we grow it like a garden." This prompted many a tweet, and some people were interested in clarification or elaboration.
Weʼve included a free PDF of our book chapter "Design & Architecture" as a download to accompany this article.
The first thing Iʼd like to emphasize is that this is a very old point and metaphor. I started work programming in the 1970s (as an APL programmer), when Harlan Mills was an important thought leader. Many programmers back then, including me, were influenced by his writings, which emphasized top-down programming; for example, the paper, "Top- Down Programming in Large Systems" [Mills 1971]. Mills talked about incrementally growing systems from thinner to thicker vertical slices of functionality with step-wise programming refinement. Thatʼs how I learned to design and evolve large software systems, starting in the insurance industry.
Then, in 1986 (often attributed to 1987, due to a re-publishing in IEEE Computer), Fred Brooks published the famous article, "No Silver Bullet: Essence and Accidents of Software Engineering" [Brooks 1986]. As with many other programmers in the 1980s, this also had an impact on me. Some quotes from No Silver Bullet are noteworthy, with respect to the growing/gardening metaphor:
Incremental development—grow, not build, software. ... the building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be accurately specified in advance, and too complex to be built faultlessly, then we must take a radically different approach.
Let us turn to nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and self-renewing. The secret is that it is grown, not built.
So it must be with our software systems. Some years ago Harlan Mills proposed that any software system should be grown by incremental development.
So, I grew up with the metaphor of growing rather than building software systems, and when I first heard the term "architect" in software development (sometime in the mid 1980s) I still remember my feeling of cognitive dissonance. It struck me then as an odd name and metaphor, not consistent with my and my colleagueʼs experience of developing systems, large and small. And in the 1970s we were very familiar with the dysfunction of the kind of "architects" that today are called "PowerPoint architects" who are not hands-on master programmers, but who were far away from the reality of the code.
Consequently, to say in a keynote in 2011, "Architecture is a bad metaphor. We don't construct our software like a building, we grow it like a garden," is to share the default view that I and many colleagues from the 1970s had and have — it certainly wasnʼt meant to suggest some novel ʻagileʼ design insight or fringe idea.
And this is an oft-told message; for example, in The Pragmatic Programmer: From Journeyman to Master [HT 1999] — a personal favorite — Andy Hunt and Dave Thomas repeat, "Rather than construction, programming is more like gardening." They are once again emphasizing the growing, living dynamics of software development, very different than the building, "engineering" metaphors that have confused our discipline.
Well, "So what?", you ask; itʼs just a metaphor. (And by the way, "gardening" is just a metaphor with strengths and weaknesses too.) Well, itʼs a very interesting time for me to be writing this short article, because just two days ago, I was sent an email from someone asking about how to do accounting in the context of iterative and incremental concurrent-engineering development (e.g., in Scrum). For example, what could be allocated to operating expense? What could be allocated to capital expense (which has different depreciation and tax implications).
You see, a word such as "architecture" is not just a metaphor in its impact; it has concrete influence on policy and people. To pick up the accounting opex/capex story again, it turns out that accountants in the USA have looked at the "phases" and activities of software development through their naive view, to answer their question of activity- based accounting. These accountants learned (read, heard) that there is an "architecture phase" done by "architects", followed by a "construction phase" called "programming." Just like a concrete building, they have reasoned by analogy inspired by the words they are hearing (since they donʼt really know whatʼs going on), that the creative design step is the "architecting" done by the "architects" (and itʼs opex), and the "programming phase" is a non-creative manufacturing activity like making doughnuts on an assembly line, from the "architectural software blueprint." And itʼs capex.
Of course, this is utter nonsense. Any really experienced software developer knows that.
But the point is that words can have strong influence in shaping mental models, and then policies and behaviors. A non-software-expert with policy power (e.g., a CEO, a VP of HR or Finance, an accountant, a lawyer, ...) hears the word "architecture", and makes naive assumptions that impact policy, process, interactions, career paths, and much more. Itʼs not just a metaphor after that.
Therefore, in our latest book, Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum, [LV 2010] Bas Vodde and I wrote the "Design & Architecture" chapter that starts with the suggestion, Try...Think ʻgardeningʼ over ʻarchitectingʼ—Create a culture of living, growing design, which we elaborate in seven pages of more detailed explanation. And thatʼs followed by about twenty other suggestions for ways of working to support large-scale agile design and architecture.
You see, we believe that agile architecture comes from the behavior of agile architecting. Itʼs primarily about mindset and actions, not the use of a particular design pattern. And part of that mindset is thinking ʻgrowingʼ or ʻgardeningʼ over ʻarchitectingʼ.
For those interested in further exploring "software gardening" and our other tips for the behavior of large-scale agile architecting, weʼve included a free PDF of our book chapter "Design & Architecture" as a download to accompany this article. Other good readings on "software gardening" include the books and on-line writings of the Pragmatic Programmers: Andy Hunt and Dave Thomas.
[Brooks 1986] Brooks, Jr., F.P., "No Silver Bullet—Essence and Accidents of Software Engineering," Information Processing 86. H.J. Kugler, ed. Elsevier Science Publishers B.V. (North Holland)
[HT 1999] Hund, A., Thomas, D. The Pragmatic Programmer: From Journeyman to Master. Addison-Wesley.
[LV 2010] Larman, C., Vodde, B. Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum. Addison- Wesley.
[Mills 1971] H.D. Mills, "Top-Down Programming in Large Systems," in Debugging Techniques in Large Systems, R. Ruskin, ed., Prentice-Hall.
Craig Larman is a management and organizational-design consultant for enterprise adoptions and large-scale product delivery with agile principles and Scrum. He is the co- author (with Bas Vodde) of Scaling Lean & Agile Development: Thinking & Organizational Tools, and Practices for Scaling Lean & Agile Development: Large, Multisite, & Offshore Product Development with Large-Scale Scrum, and author of Agile & Iterative Development: A Managerʼs Guide. Craig has coached management and delivery groups at, for example, Alcatel-Lucent Networks, Xerox, Nokia Siemens Networks, Computer Associates, and UBS. He has also introduced agile delivery in the USA defense industry. Craig has served as chief scientist at Valtech, a consulting and outsourcing company with a division in Bangalore, where he and colleagues created "agile offshore development."
Bas Vodde is a consultant/coach with focus on organizational and technical adoptions of agile concepts. He is the co-author (with Craig Larman) of Scaling Lean & Agile Development: Thinking & Organizational Tools, and Practices for Scaling Lean & Agile Development: Large, Multisite, & Offshore Product Development with Large-Scale Scrum. Bas spent several years leading the Agile change program at Nokia Siemens Networks in Finland but now works for an agile coaching company he founded in Singapore with offices in China and Japan.
Craig and Bas are also the authors of the free, online Lean Primer.
Alternative metaphor: urban planning
Urban planning is also architecture, but: we need architecture
I agree there's a misunderstanding here: the misunderstanding lies in the notion of architecture. We perceive architecture as "build once, stay forever", which is not true: buildings are pretty much evolving, living things. See castles and churches a few hundred years ago!
(Nowadays our castles are frozen to be a museum, and building exteriors are required to be kept by city councils, that's why we don't see the extreme changes which come in every few decades; given the lifetime difference between a building and a software, it's like a monthly refactoring.)
Such evolving living nature comes from the basic notion of engineering: that is, we build technical support for a community of people (wether it be houses, waterpipes, or software). As communities of people are living and evolving things as well, so are the corresponding technological artifacts.
What's strange to me, that just by reading the chapter - thanks for the author for the permission to get such a sneak-a-peek - uses the exact same practices I used as a (software) architect, the only difference is, that in some cases, I created from the whiteboard a cleaned-up version in a word processor + UML editor Given my handwriting, I found it reasonable..
So: if you have a few days long design session at the beginning of a new project, that's an architectural session.
It doesn't matter if you use the word or not.
It doesn't matter if you use Word, or a Wiki to describe the architecture.
Where we put the emphasis is a question of culture: for some people, it's crucial to have a cleaned-up logical perspective on what should be done, for others, the freedom of creativity is more important, the vagueness which comes from not cleaning up the design drafts.
I find Craig's book chapter a pretty good architectural handbook on how does software (and also building!) architecture work in practice, only the notion of trying to avoid it as a result of meeting with some bad practices long time ago is found to be a bit disturbing for me :)
Architecture is a good word, we just have to know what it means exactly.
Re: Urban planning is also architecture, but: we need architecture
I do grok SW professionals view, when arguing that "architecture" is a fine metaphor, one just needs to know what it means. That is perfectly fine, while SW professionals do not use this metaphor while talking to non SW professionals.
We use metaphor do denote things that are otherwise cumbersome to describe. So instead of saying "that thing, ... (50 line definition follows)" we say architecture, as something that people are familiar with.
However, by nature, metaphors simplify our thinking about the thing that they denote. As Craig pointed out, this simplification can have dreadful implications.
"A non-software-expert with policy power (e.g., a CEO, a VP of HR or Finance, an accountant, a lawyer, ...) hears the word "architecture", and makes naive assumptions that impact policy, process, interactions, career paths, and much more. Itʼs not just a metaphor after that."
I cannot stress the importance of the statement above, as I've witnessed its implications myself in large organisations.
"Urban planning is also architecture, but: we need architecture as a metaphor somewhere." (Adam Nemeth)
I am not sure, why we NEED architecture the metaphor. I am afraid the rest of your post does not make it clear.
I was rather wondering: if architecture is not as solid as a building, what good is it as a metaphor? It is not very useful for me, as I always have to keep reminding myself that is not quite what it suggests.
Now maybe we, conscious SW professionals, can happily live with the burden, but policy makers outside SW land are just too prone to jumping to the wrong conclusions.
I'm happy to hear other people's view on this and hope I haven't just unleashed a flame war. ;)
Re: Urban planning is also architecture, but: we need architecture
First off, the basis of my standpoint is that software creation should be based on engineering; also an important notion is, that while in the 80s, the fashion was trying to over-engineer things, in the 2010s, the current fashion is a bit under-engineering instead, and while some cleanup of terms would be helpful, I don't really want to see some world-wide mess :)
(Let's say it's a bell-curve: sometimes we under-, sometimes we over-engineer, the best is middle :)
So, I'm trying to act as a counter-balance to such trends.
Therefore, we need the architecture metaphor to emphasize the connection between software construction and post-modern architectural thinking, with such persons as Christopher Alexander (his most famous book is called Design Patterns, I guess it rings a bell, it comes from him!).
Those people who think about the architecture as a linear process don't really understand architecture, it's not about a bad metaphor as a connection, it's a misunderstanding on what architecture really is, bad mental model of architecture in general.
On why I should keep this metaphor up inside customer interaction, it's easy: if you go to a real-life customer, he'll ask you to implement his own solutions to his perceived problems. While for some people, it's an ideal situation, for most of the customers, and including most of the engineers it's far from it; but: you wouldn't tell an architect what bricks to use. The respect of the expertise of the architect, the understanding that there are things not to be seen for a layman's are gives both of you some security,some relationship basis. While both of you could do it wrong, I prefer working like this.
Of course, some people with a software engineering degree don't know too much above layman's terms, and they can't really think in a way settled out by Alexander, which makes the situation hard; but that's exactly why we should emphasize the term, to show those people where to aim.