Modeling in the Agile Age: What to Keep Next to Code to Scale Agile Teams
Now that Agile methods have become mainstream in software development, working code (and automated tests) are being considered as the most important team artifacts.
Is modeling not needed any more? Is UML dead? I think that’s not true.
In this article, I’ll explore the spaces where modeling fits and plays an important role in this Agile age, especially when development scales to multiple teams and a shared understanding of the system’s “Big Picture” becomes essential.
Where is “Design” in Agile
While code is the truth, it is not the whole truth - Grady Booch
To begin with, I’ll describe a minimal process of an Agile team using Scrum. Figure. 1 shows an intentionally simplified process with the only essential artifacts left explicit.
- The “User Requirements” are listed as a “Product Backlog”.
- The development team picks items from the list and implements them within a short iteration (or “Sprint”).
- After each Sprint, the team produces “Working Software” (or “Increments”) as “Product Code” and “Test Code”.
Figure. 1 Simple Scrum framework
In this minimum framework, the input to the team is “User Requirements” as “Product Backlog” and the output is “Working Software” as code (“Product Code” and “Test Code”). No other design artifacts between them are explicitly described here. All the design intentions generated during the Sprint will hopefully have been deployed into the working codebase as assets of the team, but there will be information which cannot be expressed directly as code . Scrum is a process-framework and intentionally mentions nothing about design, but we still have design and design activities in our team.
As Grady Booch says “The code is the truth, but it is not the whole truth.” So if there is information which cannot be expressed or communicated by the code, where can we keep those information assets? That is the question this article attempts to answer.
Documentation is not Agile?
Model to have a conversation. - Craig Larman and Bas Vodde
“In our minds!” would be one answer to the previous question. Daily meetings, pair programming, design workshops and all these social practices act as synchronization and continuous integration of the minds of the team members. But when the team gets bigger, geographically distributed, and people leave the team, “models in minds” will quickly evaporate. We need to maintain the shared understanding of the system somewhere as documentation to share information which is not well communicated and retained only by the code.
One point that Agile makes clear is the value shift from documentation to conversation, so writing heavy design documentation (which often duplicates information from the code) is not the right approach. Documentation that makes conversations effective is the approach that we should take, and it should be the simplest possible set of models which works complementary to the code.
One aspect where models have an advantage over code is visual expression. In other words, text is a poor medium of communication in a certain context. Figure. 2 shows how textual communication fails miserably(thanks to Jeff Patton for introducing the book to me).
Figure. 2 How Textual Communication Fails
I guess this “wreck” is caused by a message left to an answering machine of a cake shop, but if he/she had been able to used a simple picture (and text), he/she could have easily avoided the wreck. Sometimes, “a picture is worth 1,000 words”.
So what models can be effectively used for which purposes in Agile teams?
Agile Modeling and Two Categories of Models
Let's keep the modeling baby but throw out the bureaucracy bathwater - Scott Ambler
“Agile Modeling” is a set of practices you can use in your Agile teams for effective modeling and documentation. This method aligns with the Agile values and principles and still helps you to benefit from the power of modeling. The emphasis is on models for conversation, not for handovers.
We have been using the practices and principles of Agile Modeling with our software development team and found that the most important role of models is to visually communicate the “Big Picture” or the “Bird’s-eye View” of the system design, which is difficult to accomplish via code alone. Without models, the team would be “four blind men touching the elephant . Each man feels only the part he is touching, and it takes a long time to unify the parts into a meaningful whole… the elephant.
My recommendation of the “Big Picture” models to keep and maintain consists of:
- “Architecture” of the system for the team to get a rough idea of the whole system structure.
- “Domain Model” to help the team to understand the concepts used in the problem domain.
- “Key Use Cases ” to understand the typical users of the system, and how they benefit from the system.
These are all essential to establish understanding of the system as a whole. Without models, how would you accomplish this understanding? If you have a large codebase, and make a “Big Picture” assumption based on a small incomplete view of it, you will make some poor choices on how to maintain that codebase. The “Big Picture” not only makes the team’s mental model of the system but also supports the vocabulary they use in the conversation and in the code that they program, i.e. the structures of the code as well as detailed naming of the programming constructs such as classes, methods, variables, fields, data, configuration and etc. In other words, these models are not only important for the team to make shared understanding of the system as a while, but also for the team to keep the codebase consistent and maintainable.
On the other hand, there are also temporary models which will be thrown away once the information is programmed into the code. Casual class diagrams with few classes and sequence diagrams describing the interaction of them usually drawn on whiteboards fall into this category. Those models encourage conversation and to burn the information to the codebase, and then thrown away.
So the core of the idea is to categorize the models into two types -- models to keep and maintain as assets and models to draw temporarily to have effective conversation. We call the former “keep models” or “Keeps”, and the latter “temp models” or “Temps” (Figure. 3) . Please note that the “Keeps” are not meant to be “frozen”, but are meant to be “kept maintained” and it grows over time. In the next section, I’ll propose three “Keeps” for Agile teams.
Figure. 3 Agile Modeling with “Keeps” and “Temps”
Models to Keep
Depending on the context (the number of people, criticality of the system, stability of requirements, enterprise system or embedded), the keep models will vary. Based upon my experience, here are some good candidates of them:
- Architecture As Class/Package Diagrams
- Domain Model As Class Diagram/ER Diagrams
- Key Use Cases As Use Case Diagrams + Sequence/Communication Diagrams
We mainly use UML, but you don’t have to stick to the strict UML specification. We use it because it includes sufficient standardized diagrams and there are a lot of educational materials published. ERD (Entity-Relation Diagram) and DFD (Data Flow Diagram) for data and processes are also used sometimes for the same reasons.
Figure. 4 illustrates the roles of the three keep models as a picture. In a nutshell, “Architecture” shows the structure, “Domain Model” shows the core concepts of the problem space, and “Key Use Cases” show examples of the usage of the system.
Figure. 4 Architecture, Domain Model and Key Use Cases
I’ll explain the three one by one with concrete examples and pictures from one of my teams.
1. Architectures As Class/Package diagrams
The architecture is a structural presentation of the whole system. It is often described by class or package diagrams typically to show global tiers (layers). For example, in an application with UI and database, tiers are usually set horizontally from UI to database, and one Use Case walks through them to accomplish its goal.
Other architecture patterns like “MVC” (Model-View-Controller) can also be chosen as a global architecture. Figure. 5 is an example of an architecture drawn as a package diagram, and apparently it is based on an MVC architecture.
Everyone in the team should understand the roles and meanings of the component of the architecture so that the team member can write code which fits in the right place in the architecture.
“Dependencies” are often expressed in this diagram between packages to avoid unwanted couplings or circular dependencies. Because from the architecture view point, inter-package circular dependency is the evilest problem that results in harder testing and a longer build time.
(Click on the image to enlarge it)
Figure. 5 Architecture As Class/Package Diagram(Example)
2. Domain Models As Class Diagrams or ER Diagrams
A Domain Model describes the concept taxonomy of the problem space in which the application works. In the human communication level, the vocabulary this of domain model is to become the “Ubiquitous Language” used in the whole stakeholder community including the users, domain experts, business analysts, testers and developers.
In the programming level, the Domain Model is also essential for selecting names of programming constructs such as classes, data, methods, and other conventions. A large part of the concept taxonomy (often called “entities”) is mapped into a persistent data structure in the database and often has a longer life than the application itself. And typically, the domain model (or entities) resides in the “M” package in the logical architecture if you choose an “MVC” architecture for your application. In a RubyOnRails type of applications, an ER diagram is more suitable for expressing an domain model because it is tied more directly to relational databases.
Also note that this domain model grows over time. Because domain is in the heart of the understanding and communication of the problem, maintaining the growing domain models in the team (or broader, in the community) is one big topic that is fully discussed in Eric Evans’s DDD (Domain-Driven Development).
Figure. 6 is an example of Domain Model expressed as a class diagram which presents the domain in one picture.
(Click on the image to enlarge it)
Figure. 6 Domain Model as Class Diagram(Example)
Key Use Cases as Use Case Diagrams and Sequence/Communication Diagrams
Key Use Cases are typical usages of the system, from the users viewpoint. There are two reasons why we include them into the Keeps. The first is that the developers often go into the solution and forget who the users of the system are and what they want to accomplish with the system. Use Cases help them to get back the users view and are a good way to have conversations with users (other documents are usually Greek to users).
The other reason is that Key Use Cases and their mechanics described as sequence or communication diagrams work as educational examples for developers. They describe how several objects in the system in different tiers in the architecture work together to accomplish the user goal. Draw a concrete example of a vertical slice from UI to database and illustrate how you implement the Use Case in the architecture.
The Key Use Cases don’t have to be complete to cover all situations. Just pick the typical ones and keep them simple.
(Click on the image to enlarge it)
Figure. 7 Key Use Cases as Use Case diagram (Example)
(Click on the image to enlarge it)
Figure. 8 Key Use Case Mechanics as Communication Diagram (Example)
Figure. 7 is an example Use Case diagram which makes the users of the system and typical usage of the system explicit. It doesn’t need to be comprehensive, but should capture the context of the system. The yellow Use Case (“Create Class Diagram”) is chosen as an example Use Case, and the design breakdown is diagrammed as a communication diagram in Figure. 8. With this example, the team can share an understanding of how the Architecture and Domain Model (diagrammed in Figure. 5 and Figure. 6) actually work to accomplish the features described as Key Use Cases. See Figure. 4, again, for the relationship of Architecture, Domain Model, and Key Use Cases.
You can use tools to draw these diagrams to make maintenance easier, and to have them printed on a large piece of paper, and posted on the wall. The wall will then become a discussion place of modeling workshops (as I will soon discuss in a later section).
Rather than divide and conquer, an XP team conquers and divides. - Kent Beck
With one small team of less than 10 developers, you may not have to maintain any models next to the codebase. As the development scales to multiple teams, you will need to get more benefits from modeling.
But remember, don’t invest too much time to prepare heavy documentation (with ZERO implementation) just to hand it over to someone you don’t know. Even when the team gets bigger, you need to try to create a thin vertical slice of your implementation to accomplish the Key Use Cases first to make an architecture seed, and then share the knowledge with sub teams using the working code and “Keeps” models. In other words, don’t’ try to “divide and conquer” (Divide the problem on the desk and throw the specification over the wall to make the sub teams conquer). Instead, “Conquer THEN divide” (More on this in large-scale agile design & architecture by Craig Larman and Bas Vodde).
Here, I describe how multiple teams communicate the “Big Picture” using the “Keeps”. The first “conquer” should be tried by less than 10 people in one team called “Tiger Team” at one location. After the first conquer, all the “Keeps” described above can be used as good documentation to communicate the understanding of the system. In Sprint 1, the tiger team conquers the Key Use Cases first to establish the first “seed” architecture and makes the version 1.0 of the Keeps as SAD (Software Architecture Document). Think of these models not as a specification, but as common ground to create understanding. And again, do not just hand the document over to the sub teams.
Figure. 9 Tiger team and Sub Teams
The best way to communicate design intention and making a shared understanding is to facilitate a Modeling Workshop with the sub teams (Figure. 9).
(Click on the image to enlarge it)
Figure. 10 Modeling Workshop using Keeps models on the wall
In the modeling workshops, one member of the tiger team (“Ken” in Figure. 9) first explains the SAD and walks through the models. With casual Q & A’s, he communicates the core ideas and the structure of the system. He can use the Key Use Cases to explain how the system components works together to accomplish a user goal. And he jointly designs one or two of their Use Cases with them, using Temps models and maybe pair-programming.
Don’t try to make the SAD complete. Use workshops as a way to establish a common understanding without handovers and having a rich conversation.
An important part of the sub team workshop, is feedback. In Figure. 9 “Ken” in Sub Team 1 and “Tom” in Sub Team 2 bring the feedback back to the tiger team and together with other members discuss how to make the keep models better. Figure. 11 is a printed diagram with a lot of notes during the workshop. It includes notes for understanding and also notes for feedback.
(Click on the image to enlarge it)
Figure. 11 Architectures with comments after Workshop
Hold the workshop repeatedly. And grow the understanding by “modeling”, not by the “models”. Remember, use “model” as a verb and “Model to have a conversation”.
People As the Conveyor of Knowledge
A lot of design information lives in tribal memory – Grady Booch
Even if the “Keeps” models and codebase can cover most of the knowledge needed to grow and maintain the system, there still is tacit knowledge stored in the brains of the team members. Grady Booch called it tribal memory.
There is a shrine called “Ise Jingu” in Japan. They have a shrine building and two sites of identical size next to each other. And once every 20 years, they move the building from one to the other. It involves the reconstruction of the building as well as the renewal of the sacred apparel and the treasures. This ceremony transmits the knowledge of building the shrine to the next generation. There is design documentation, but techniques, tools, and practices are passed from one generation to another by doing the reconstruction together. Remember “Experience is the best teacher” and working together conveys the richest knowledge of design.
Ise Jingu (Picture Source: Wikipedia)
With the ideas and experience that I have described, here are some final tips for you to use in daily modeling sessions, or modeling workshops.
- “Reverse and Model” A lot of UML tools support “reverse engineering” features that visualize the codebase in a just-in-time manner. Some of them have a nice “Drag&Drop” from source code and even from Github repository URLs. With the packages and classes which you reverse-engineered from the codebase as a foundation, you can start casual modeling not only with the “keep” models, but also with models made directly from the codebase.
- “Print and Draw” As described earlier, good interactive modeling workshops are facilitated with big paper posters on the wall (or on the table) and by doing the conversation with them. Draw notes and comments directly on the printed poster (Figure. 11).
- “Overhead projector and whiteboard overlay” Another way of sharing models in workshops is using an overhead projector and a whiteboard together, to simulate “Print and Draw”. Use a overhead projector to project the “Keeps” on a whiteboard and draw comments or put sticky notes on it.
- "Retrospectives" I proposed my version of the simplest set of "Keeps", but it can vary in your context. So start from my proposal or what you think works as the first "Keeps", and do retrospectives on models you chose after each Sprint, discussing which models worked well, which didn't and what to add. Find your "Keeps".
- "Mindmap Modeling" UML and other software engineering diagrams don't work well in discussions with users, planning, or other casual but imporant activities. Use mindmapping for those situations. See "Agile Modeling with Mind Map and UML" for the detail. Here's an example of "user story exploration" mind map I created with a user.
(Click on the image to enlarge it)
In this article, I explained how modeling fits into an Agile development framework like Scrum, and proposed the models you could keep throughout the lifecycle of the product. And I recommended facilitating a modeling workshop to communicate design intentions and to establish a shared understanding of the system. These practices become more important when the team scales into many sub teams.
I’d like to thank Hiroki Kondo, Alex Papadimoulis and Scott Reece for comments and Ben Linders for reviewing and editing my article. Special thanks to Craig Larman who first described the importance of modeling workshops (or “model” as a verb) and spent his time on an airplane to give me fundamental suggestions on this article.
The discussion of design in the Agile context is old. See Martin Fowler’s and Jack Reeves’s classics.
- Martin Fowler, 2004 , “Is Design Dead?”
- Martin Fowler, 1997 , “The Almighty Thud”
- Jack Reeves, 1992 , “What is Software Design?”
The concept of Agile Modeling is first discussed in the book “Agile Modeling”. And 3rd edition of “Applying UML and Patterns” covers the topics again.
- Scott Ambler, 2002, “Agile Modeling”, John Wiley & Sons Ltd.
- Craig Larman, 2007,“Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development”
Idea of Modeling Workshop is inspired largely from Craig Larman and Bas Vodde’s book.
- Craig Larman and Bas Vodde, 2010, “Practices for Scaling Lean & Agile Development”
- You can download Chapter 8 from "Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum"
The “Keeps” and “Temps” idea came from this picture.
Another InfoQ article that addresses the same problem and context. Still waiting for the part two.
Wider topic on Agile and Architecture.
I attended the workshop, discussed “Architecture” in “Agile”. The two don’t conflict.
About the Author
Kenji Hiranabe is an Agile software development practitioner, book writer/translator and CEO of Change Vision, Inc.He thinks of software development as a form of collaborative game, and have been searching for better ways to make it more productive, collaborative, and fun. 2008 Gordon Pask Award Recipient for contributions to Agile practice.
Thank you for writing this article
Diagrams in your architecture model?
A question for you; how many diagrams typically make up the architecture model that you keep? The example package diagram shows something about the overall structure, but I'm left with a number of questions related to technology (it is a web application, rich desktop app, etc?), the runtime environments, etc. Do you have a list of diagrams/views that make up your architecture model?
[disclaimer: I wrote an article for InfoQ about software architecture sketching and I'm curious to see how your approach compares]
Use KPT to improve model
Re: Thank you for writing this article
>My pain is that sometimes that agile is just interpreted as "Go Fast" and we will worry about it later,
Exactly! This is my pain and the motivation of the article. And I'm finding that as you picked Activity Diagrams, models that work best vary in the context. To find YOUR ENUF (ENough Design Up Front), simplest set of diagrams is essential, I believe.
Re: Diagrams in your architecture model?
The Sample models are from our product Astah, which is a desktop UML editor application(with which the diagrams are drawn, too) with Swing UI and in-memory database.
And Sorry, I forgot to includes your article to the further reading. "NoUML" is a nice name! I think we share the same issue to address.
Design - A missing piece in Agile (Scrum)
An effective engineering practice need to focus not only on test (i.e inspection), but also on design (i.e. prevention) for quality management and improvement. Yes, I agree with you that it makes sense to have a "lightweight and enough design" up front in Agile process in order to deliver faster, cheaper and better.
[Disclaimer: Recently I posted an article in InfoQ about the process of developing IT architecture including software architecture. ADMIT could be a helpful tool to think about the system's big picture architecture. www.infoq.com/articles/admit-architecture-frame... ]
Re: Use KPT to improve model
Here, I use "Keeps" as "keep models" working with "Temps". And yes, retrospective works great to find your own "keeps."
KPT(Keep/Problem/Try -- a retrospective framework introduced by Alistair and widely used in Japan only) is one of the ways to find and improve your "Keeps".