Software architecture documentation is an important part of the enterprise application development process. The key to understanding a project's architectural documentation needs is to understand the role that architectural documentation plays in a project's life cycle. The fundamental reasons why a project produces architectural documentation are communication, analysis and record keeping (e.g. keeping track of decisions over time so they don't get lost). The amount and type of architectural documentation produced by a project should reflect the communication and analysis needs for the product being produced by that project.
Architectural documentation is used to communicate from a project upwards to its management, from an architect or lead designer and to the developers, and from a project across time to future maintainers and developers. Just this simple statement leads to three of the questions whose answers help determine the amount and type of documentation. How much oversight is being placed on the project, how many developers and of what skill level are developing the product, and how long lived is the product going to be? The more oversight, the more the need exists for documentation to communicate to management. The more developers and the lower their skill level, the more the need exists for developer guidance. The longer the product is envisioned to live, the more the need for documentation to communicate to future developers on the product.
The communication part of the architecture documentation includes communicating to management, developers and communicating across time (project life cycle). And the analysis needs can come from internal sources to determine the quality of the product with respect to performance, security, reliability, and so forth or from external sources to determine compliance with regulatory or standards based requirements.
In this virtual panel, InfoQ wants to find out from leading software architecture experts about the significance of software architecture documentation and how to document the architectures especially in Agile Software Development environments.
The panelists who have answered our questions are:
Len Bass, Senior Member of the Technical Staff at Software Engineering Institute (SEI), co-author of the books Software Architecture in Practice and the upcoming 2nd edition of Documenting Software Architectures: Views and Beyond.
Grady Booch, IBM Fellow, author of Handbook of Software Architecture web site.
Paulo Merson, Senior Member of the Technical Staff at Software Engineering Institute (SEI) and co-author of Documenting Software Architectures: Views and Beyond book.
Eoin Woods, Head of the Application Architecture group at Barclays Global Investors, co-author of "Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives" book.
What is the role of software architecture documentation in development teams that are using Agile and Lean software development methodologies like SCRUM, eXtreme Programming (XP) and Kanban?
Len: The purposes of architecture documentation do not vary depending on the development methodologies. Documentation is not the only method of communication. In agile methodologies such as SCRUM or XP face to face contact replaces some documentation needs. It does not replace the documentation needs for detailed communication of design decisions. These are difficult to convey and to get correct through face to face meetings. Communication across time or to management is not dealt within the agile methodologies. In agile methodologies, the need for documentation is reduced to the extent that face to face communication removes some portion of the need to communicate down but not much more than that.
Grady: The same role as for teams using even higher-ceremony methods. In short, documenting the architecture of a software-intensive system serves several purposes: to permit teams to reason about significant design decisions, to codify tribal memory, providing an essential artifact for intentional governance across each iteration. For these more light-weight methods, architecture documents are more often used as a reflection of the as-built system, rather than as an to-be-designed systems.
Paulo: The role of software architecture in general is to be a bridge between the often imprecise and dynamic requirements, and the runnable implementation. Architecture documentation captures the design decisions. It's an artifact that guides implementation (foremost), can be used early on to evaluate if the envisioned solution is in the right direction, and can help with work assignment and tracking.
But what about agile? Saying that agile development is antithetical to architecture documentation is simply not true. In agile, the goal is not to avoid designing, but to avoid "big design up front" (BDUF). Broad and far-reaching architectural strategies are worked out up front, but many other design decisions are deferred until needed. For example, the data model is often discussed more thoroughly up front to avoid major changes later on. On the other hand, sequence diagrams and interface definitions for the features in the next sprint can be documented days before implementation.
Agile developers are expected to have design skills, and agile architects are expected to have coding skills. So the communication of design decisions is shorter and more focused. What is that in practice?
- More diagrams and less prose.
- Design information just enough to move on to code.
- Focus on diagrams showing the reference architecture (or the way patterns are applied). Then the design of similar features or flows is lightly documented because the solution can be inferred from the reference design.
- Sketches rather than formal or comprehensive diagrams.
- Design diagrams are crossed off rather than updated. This is better than leaving the design out of date. (I usually put a visible mark on diagrams like the one below.)
Eoin: It may sound obvious, but the answer to this question really depends on the team, the project and the context. An architectural description, like any deliverable, needs to be created for a purpose and an audience, otherwise creating it is a waste of time. Sometimes the audience is just the person creating it (to help with analysis or just as an aide memoire) but there is usually a wider group of people who are interested in any significant deliverable.
Personally, I've never found there to be a great tension between agile development and software architecture, although I know that's not everyone's experience - perhaps I've just been lucky! My experience is that a clear, concise architectural description, developed incrementally as the project progresses, provides a useful context for software development teams, agile or otherwise.
A good architectural description captures those aspects of the design not obvious from the code (such as deployment design, external interfaces or the system monitoring approach) and explains how the system meets its non-functional requirements (typically including HA, security and performance needs). If no one is interested in these subjects then there's no need for an architectural description, but I've found that this is rarely the case.
It's also worth mentioning that an architectural description can take many forms, a thick document often being the least effective one when you're working with an agile team. It's the knowledge you collectively gain and the analysis you perform while creating an architectural description that are important, not the document template that you use. An architectural description must be an effective communication medium and so wikis, models, databases, spreadsheets and sets of short focused documents can all be more appropriate ways of capturing and communicating architectural design information than a weighty tome.
Are there any special architecture documentation patterns one should follow in documenting the architecture of applications that are based on Domain Specific Languages (DSL)?
Eoin: I've not yet been able to work with DSLs on a production system, but I haven't found that changing implementation technology affects my general approach to architectural documentation very much. You still need to define the system context, the functional structure, the deployment environment, the information structures, the operational environment and so on, and also explain how the nonfunctional requirements are going to be met. The basic elements that you use to describe your designs in each section (or "view") of the description may vary according to the technologies in use, but you still need to describe the same things.
Len: To the extent that the DSL languages, profiles and stereotypes, additional modeling languages, and metamodels have defined notations (some do and some do not) then they would help in making documentation more concise. This would reduce costs if both the readers and the writers of the documentation have an understanding of the additional notations.
Grady: Not really. The essence of any approach to software architecture documentation is to identify the stakeholders and then the views that attend to their concerns. Deciding on the views that one needs is dependent on the domain and the development culture.
Paulo: Architecture documentation consists of multiple views showing the multiple structures of the system. The fact that you're using a DSL may not affect some of these views, such as the deployment view or the runtime (aka component & connector or process) view. The use of DSLs will more likely transpire in the module (aka code) view of the architecture. This view shows the implementation units, how are they organized in modules and submodules and their various relations and dependencies. In any view, it's very important to identify the types of the various elements being depicted. A DSL imposes a well-defined set of types of elements and relations. Especially when a DSL is mixed with other languages in the whole solution, it's important to identify the element types in your architecture documentation. In practice, a diagram key is all you need in most cases. So you use different shapes or colors for different types of elements but also indicate in the documentation what a rectangle means as opposed to an oval. And don't forget to differentiate the lines and arrows too. DSLs often come with automated tools, libraries or code generators. So it's also important to show the scope of your implementation and what's provided by the tools. Context diagrams can help. Whenever code generators are used, make sure you understand what kinds of runtime elements are generated and their properties. Finding out if it's a single process/thread or not, what communication protocols are used, what data storage mechanisms are used, and so on is relevant to assess quality requirements, such as security, performance and interoperability.
I once attended a demo of Microsoft Visual Studio being used to define a modeling DSL. It was quite impressive. But one can use a general modeling language like UML to document the design even if you're using DSLs. UML profiles are often used to define domain-specific symbols derived from general UML symbols.
Where do the new dynamic Java module specifications such as OSGi fit in Software Architecture documentation efforts?
Paulo: It's common for software architects to focus on the module view (aka code view) in their architecture documentation. They spend time explaining how implementation units are organized in packages and subpackages, how modules depend on each other, how are they organized hierarchically or in layers, and the flow of messages for particular transactions. Creating this code view of the architecture is very important, but often architects overlook runtime and deployment aspects. Documenting these other views require good understanding of the runtime environment, the platform infrastructure, the component model. The runtime view shows what your implementation units translate to in terms of threads, processes, Servlets, DLLs, and other components. It also shows data repositories, infrastructure services, external services, communication mechanisms (e.g., JMS, SOAP, http, local or remote method call). This runtime view allows you to reason about runtime properties such as performance, security, availability, which are hard to assess looking at package and class diagrams. Deployment views show the hardware infrastructure but also explain how deployment artifacts are packaged together and deployed. Such views help to reason about serviceability, performance, security and other properties.
Now let me try to bring home the answer to your question. If you're architecting an OSGi application, you should understand what kinds of components exist in the OSGi runtime environment and their lifecycle. My only experience with OSGi was developing Eclipse plug-ins, which kind of hides the OSGi framework except for the manifest files you need to create. But let me ask a few questions to architects of OSGi applications:
- Does your architecture documentation show what java packages/classes go inside each deployable bundle?
- At runtime, does each bundle correspond to a thread? Or does each service within a bundle have a separate thread?
- Either way, does your documentation show these threads?
- Is a new thread instance created for each call to a service, like there are multiple thread instances of the same servlet for each http request addressed to it?
- If I want to deploy a library jar file that will be used by multiple bundles, does the documentation show where should I put it?
- Are the events sent via the OSGi pub-sub mechanism synchronous or asynchronous?
- Does your runtime view distinguish types of calls and protocols (e.g., http) being used?
- Does the documentation show the interaction with the service registry, which certainly affects availability, performance and security?
Thus, the first part of the answer is to create runtime and deployment views of the architecture that reflect the component types, communication mechanisms and infrastructure services of the runtime environment.
The second part of the answer is about variability. In our architecture document template we have a section only for that. In the variability guide section we describe the mechanisms that exist in the solution that make the system configurable. Some examples of variation mechanisms are:
- Replacement of components (that provide the same interface).
- Component replication.
- Optional inclusion of components (plug-ins and add-ons).
- Build-time assembly configurations.
Frameworks like OSGi give you these variation mechanisms out of the box. The runtime picture of the architecture can change in terms of what components are there. The architecture documentation should document in the variability guide what components can change, what are the available options (or configurations) and their effects, what conditions may trigger specific options, and the binding time of each option (can be run-time for OSGi, but could also be build or deploy time).
Eoin: Anything that makes the component structure of a system more visible at runtime (and for that matter design time) is a great thing, as we currently throw away a lot of this information when we move from design to code (hence the need for an architectural description - see earlier answer!) I've recently been learning about OSGi and it seems to address this very concern, helping to make the functional structure of the system more obvious. All the same, module technologies like OSGi only help to reveal one sort of architectural structure when, in reality, there are many that need to be considered. So they don't allow us to abandon our architectural descriptions, although they may well provide many other technical benefits that justify their use.
Len: One of the standard views for documenting software architectures is the module decomposition view. Each module in that view represents a scope of encapsulation. OSGi is a means for specifying a scope of encapsulation so that fits naturally into the module decomposition view.
Grady: OSGi primarily attends to what Kruchten would call the implementation view, namely, the concerns associated with the physical packaging of components that are then provisioned to the deployment view, which addresses the physical topology on which these components live. As such, from an architectural perspective, OSGi is just a specific packaging technology.
How do UML features like "Profiles" and "Stereotypes" help in documenting software architecture?
Grady: These are the primary mechanisms for UML extensibility...but in my experience, the base UML language is more than sufficient for documenting the essentials of the architecture of most software-intensive systems.
Paulo: UML was originally created to model OO systems. It's now considered a general purpose modeling language. That means the same symbol can be use to represent many different things. For example, a UML component named Dispatcher may well represent a Servlet in a Java EE solution or a thread in the OS kernel. You can define stereotypes to specialize UML symbols. So if you know some of your components will be servlets, stereotype them as <<servlet>>. If used correctly, stereotypes automatically make your UML diagrams more expressive. UML comes with several standard stereotypes, but we're free to create our own stereotypes.
A UML profile is just a group of UML stereotypes that make sense together. There is a profile for Java EE, a profile for .NET, a profile for SOA, a profile for systems engineering, and many others. It's a good idea to reuse or create your own stereotypes and profiles.
Eoin: In a word, I think they're lifesavers. UML2's extensibility is one its strong points and is one of the things that has allowed UML to be used for many different things. Very few people take advantage of it, but modern UML tools have finally caught up with the language's extension abilities and allow you to use stereotypes and profiles to tailor UML into a more specific language for the task in hand. With a modest amount of effort this allows you to create a tailored version of UML, specialised to express the sort of constructs that you want to use in your architecture. The result is a UML-based architectural description language with ready made tool support, so overcoming one of the main barriers to adopting many research-based ADLs (like xADL, Acme or Darwin).
There are new specifications like Service oriented architecture Modeling Language (SoaML) which describes a UML profile and metamodel for the design of services within a service-oriented architecture (SOA). How should the architecture documentation be done in applications using SOA and Cloud Computing architecture models?
Eoin: My view is that SOA and cloud computing are relatively independent technologies (although one can make the use of the other easier). SOA is an architectural style that guides you to structure your systems as collections of loosely coupled communicating services; cloud computing is a deployment model that virtualises the runtime platform in order to simplify the provisioning, operation and management of the machines that systems are hosted on. They both relate to location independence, but address different aspects of it. Like DSLs, I don't think either of these technologies fundamentally change what you need to capture in an architectural description or how most of it is captured. However, as we discussed above, specialisations of modelling languages (like SoaML) can certainly help to make the process of documenting some types of architectures more straightforward.
Paulo: SOA is an architectural style that can be realized using a plethora of technologies, such as SOAP Web Services, REST, messaging systems, BPEL, and ESB. I think in practice when you create an SOA-based architecture, soon enough you need to identify what technologies you are going to use. For example, I think it's more insightful for the reader if a service provider component in the SOA design is stereotyped as <<dll>> or <<servlet>> rather than as a generic <<participant>>.
Think about it. Software developers have to keep up with an increasing rate of new programming languages, scripting languages, technologies, development frameworks, design patterns, IDEs and plug-ins, modeling tools, and development techniques. I as software developer am willing to learn a UML profile if it is simple and contains stereotypes that correspond to solution elements I'm already familiar with.
I think it's difficult that a technology-agnostic UML profile such as SoaML will be appealing to a broad audience of architects. Getting acquainted with this profile is certainly important if the modeling tool you're using supports the profile though.
With or without UML profiles, the key to properly document applications using SOA, cloud computing, OSGi, Peer to peer, or other architectural styles is to include an architecture view that explicitly shows the element types and relation types that are specific to that style.
Grady: Really, it's a matter of deciding what the essential design decisions are. In systems that choose to use services, the design of the messages themselves (not just the point-to-point services) is essential (and often forgotten). In the case of cloud computing - which, by the way, is a phrase that's currently more marketing than technical - attends more to elements of the deployment view.
What are the best practices and "gotchas" that software architects should keep in mind when documenting their software architectures?
Len:
- The software architect should keep in mind the purpose of generating the documentation. Over-documenting and under-documenting are both manifestations of misjudging the purpose of the documentation.
- The project is not the best determiner of the use of documentation as communication over time since most project members will not be part of the maintenance and enhancement effort and do not have a stake in the existence of the documentation.
- Wikis have proven to be an effective media for keeping architectural documentation.
Grady: Three things come to mind: don't over-document, keep this document alive over each iteration, be certain to attend to the views and cross-cutting concerns that are not easily extracted from the code itself.
Paulo:
- Document the architecture in multiple views.
- Always have a notation key in your diagrams.
- If you're using a design or architectural pattern, let the reader know.
- Follow a template.
- Only spend time documenting something if it will be useful to some stakeholders.
- Record the rationale for important design decisions.
- Review what you create.
Eoin: The first thing that comes to mind is that the SEI book "Documenting Software Architectures" has a very nice list of seven "rules for sound documentation" which is also available in one of their publicly available technical reports (SEI-2000-SR4) available from their web site.
I think a full answer to this question would probably require a short book, but in my view some of the more important things to keep in mind when documenting a software architecture are:
- document with a definite purpose in mind;
- create an architectural description for a definite audience;
- be precise in your descriptions, even if you're being abstract;
- define the notations you use carefully, so that others understand them unambiguously;
- use a set of software architecture viewpoint definitions to act as an aide memoire and guide the content of the description;
- don't let the document become the focus, the process, the content and the communication you achieve are the important things; and
- develop architectural descriptions iteratively, constantly reviewing their correctness and usefulness.
How can software architecture documentation efforts in an enterprise help the company in the current economic and market conditions?
Grady: This is the essence of treating architecture as an artifact (and I've just written a multi-page article on that very topic for my column, On Architecture, for IEEE software.
Paulo: Steve McConnell, Barry Boehm and others have observed that defects originated in the requirements or architecture are much more expensive to fix. In other words, a sound architecture really pays off. Now, it doesn't matter if your software architect has lots of experience and knows everything about patterns, current technologies, modeling tools and languages, because if this guy creates an ingenious design but the developers don't understand it, the project won't reap the benefits of a good architecture. Software documentation can easily fail to communicate ideas. One reason is that software engineers usually are not trained to write technical documents. Good architecture documentation is an enabler of:
- architecture evaluation
- software reuse
- successful procurement plans and RFPs
- project management tasks (estimation, assignment, tracking)
In sum, good documentation at the right level of detail (and not more) increases the chances of success in a software project.
Eoin: Developed and used intelligently, architectural descriptions and ongoing architectural reviews can help an organisation by providing more visibility into the systems being built (and indeed those that already exist) to help to answer questions like:
- do our systems have the right functions? (individually and collectively)
- do we have duplication between systems?
- are our systems built in a consistent way?
- do we have excessive sophistication, flexibility, interoperability, security or other potentially expensive system qualities?
- have we build a system that is inflexible and will cost use a lot to evolve later?
- are we about to make any costly mistakes?
Len: Software architecture documentation serves the purposes described above. They help an enterprise to the extent that an enterprise wishes to exercise oversight over a project, wishes to enhance communication among developers, wishes to enhance communication over time, or wishes to have an analysis of the design.
What do you see as the future of software architecture documentation?
Eoin: I hope that in the future we'll need very little software architecture documentation because we'll be able to see the architecture in the code and the running system! One of the reasons we need much of our architecture documentation today is because there's no way of representing architectural structures directly using the technologies we have at our disposal. I'd love to see our architectural constructs as first class implementation structures and then architecture documentation can evolve to capture decisions, rationale and analysis, rather than just capturing structures. On the way to this nirvana, I hope that work going on in the areas of DSLs and ADLs (architecture description languages) point the more immediate way forward, as we improve our description languages, on the way to working out how to embed the information right in the running system.
Paulo: The software architecture discipline is fairly new. There is a long path ahead until we get to a point where an architect creates architecture documentation that is readily understood by a developer who has never worked with that architect. The way to get there is to let new architects learn software architecture at school rather than try-and-error in the battlefield. This education includes proper ways to represent the software architecture for other people's consumption. Important initiatives in the direction of good software architecture education are: the work of Grady Booch on the handbook of software architecture and the publications and curriculum developed at the SEI.
Grady: There is a lot of energy being applied today with regard to architectural frameworks and methods: TOGAF, NEA, DoDAF, MoDAF, FSAM, Zachman, and so on. The good news is that there is a vibrant dialog going on with regard to these frameworks and methods - but I expect there will be a shakeout/simplification over time.
Len: The ideal development environment is one for which the documentation is available for essentially free with the push of a button. This will require an integrated development, requirements management, and project management environment. Although this will be a long time coming, it provides a worthy goal to strive for.