We're sitting here at OOPSLA 2007 with Markus Voelter and we are going to discuss software architecture documentation. Markus, what do you think are the most common problems with documenting architectures today?

The first problem is the same problem as with documentation in general, people perceive it to be boring and in many cases, an overhead we don't want to do,. I know they don't say you should document, but it's often perceived that they say you shouldn't document at all. So first of all you have to convince people that documenting soft architectures is useful and necessary and that of course you have to distinguish between, let's say, the ad-hoc system where two people hack together something within three months, they don't have to document the architecture. They are sitting next to each other, what's the point? But if you built a long lift strategic reusable, maybe, platform or something like that, where you want to strategically build many applications onto the same kind of architecture on metaphors then you have to document the architecture. So the question is: which aspects of architecture do you have to document? I think there are several: one is that you have to first come up with a clear definition of what your architectural building blocks are.

What I am talking about is things like components, connectors, messages, message queues, pipes, filters. You already see that there are some architectural patterns going in there, so documenting what the building blocks of the architecture are all about then the next thing is how do you build a concrete application with that, how do you implement a component, how do you send a message? That is what I like to call the programming model, so it's basically a API for using the architecture for building applications on top of the architecture. The other thing you need to document, I think is to simply explain why the conceptual architecture, your building blocks are the way they are and what each building block does and how they go together. And something I found very useful there is to use the pattern form.

Everybody hopefully uses those patterns and you all know the GoF book; the idea is that instead of just using patterns by saying: "My architecture is a pipes and filters architecture", you can look up what pipes or filters mean. My suggestion is that if you build your own concept, your own building blocks, that you define them as a pattern, meaning that you first describe the context in which you defined it, why you did it, the problem you wanted to solve with this particular building block of the architecture, the solution which would be the building block and then tradeoffs, advantages, disadvantages and where this leads you overall and maybe then which other patterns building blocks you came up with to solve that resulting problem. So using a pattern language even to describe the rationals behind the architectural concepts is extremely useful.

When people think about documenting architectures and documenting things like that, they usually think about UML, modeling tools and stuff, what's your take on that?

I recently had to do an architecture review for a large German automotive system supply thing and they sent me the material that I should review. I think it was a really big enterprise architect UML model. And I sent it back and said: "Well that is no architecture documentation" especially since it was only classes and no comments, no descriptions, so completely pointless. First of all, diagrams are extremely useful and I am not talking about models, I am talking about diagrams which are visualization of complex things, are extremely useful for all the things I've talked about in the previous questions.

So if you want to explain concepts, if you want to show how they go together, then using diagrams is extremely useful, no question about that. Diagrams are not formal, they are not necessarily correct or complete, they're just useful for communicating things to other people and we all know that a picture is more than 1000 words, so that's the purpose of diagrams. Models should be formal, in the sense that tools can process them. So for example if I have an architectural model, then I want to be able to, for example, verify that there are no illegal dependencies in the system, by following the dependency graph and the model and saying something about it, with the tool. Or I want to generate the complete technology crap code, the adaptation code to whatever infrastructure, I want to hide that.

I want to make sure that the programming model we talked about before is clearly separated from the implementation decisions, which applications I or you use, whether I use Spring or EJB or whatever the technology of the day is. So one way of isolating that is describing architecture in a formal model and generating stuff and implementing the business logic and well defined slots. So modeling is clearly useful. The question is what's the role of UML is in that context and generally UML was not build to be a formal modeling language. If you talk to the people who built it back then they said it was clearly intended to describe systems to other people; this is the diagramming thing again. If you want to use UML for modeling in the sense of formal models you need to typically use UML profiles and defining a UML profile is basically defining your own language.

It happens that you use the UML to do that. I mean, if you define a profile formally, with UML's mechanisms, then you are formally defining a language. Now some people, including me, argue that defining a language using UML's profile mechanisms may be the worst way you can use to build the language. There are much better ways. So what I am trying to get at is that once you have your conceptual architecture well understood, you know that you have components, interfaces, whatever your building blocks are, describe them formally in a meta model, come up with a syntax and then you can go ahead and formally describe systems built on that architecture and you can generate all that technology rubbish. So in that sense I am an extremely big fan of using modeling for architectures, but it's not enough to document an architecture with only a bunch of diagrams or for that matter models. You always have to explain why and you have to use didactically valuable tools to communicate it. Giving people a UML model with 5000 of classes and bunch of sequence diagrams won't work.

What format is this model's in this case? Do you visualize diagrams for the model or do you create the model from some visual tool or how do you create models?

A model is basically an object graph in the memory of your tool which conforms, and that's important, to formally defined meta model and has a well defined meaning by that. So using what I just said you can communicate the model to a tool. However, of course you have to communicate the model to somebody who draws the model and I already used the term "drawing", so one way of making models accessible to users, to architects or application developers, is to use a graphical notation for describing the models. So you visualize the formal model as a diagram in whatever tool: UML tool is one, maybe not a great choice.

It's also surprisingly useful to use Visio. Visio has its own internal object oriented data structures and it's actually not that bad for this kind of stuff. However you can also describe your model using text. For example, just to be cynical for a change, if you look at the UML diagram and you look at the interface, it's basically a rectangle with a textual specification in it. So this raises the question, which aspects of the software architecture, or for that matter or anything, should be documented graphically, visually, and which should be documented, for example, using textual languages. And interfaces are good candidates for textual languages.

If you want to use graphical notations, they always make sense if you want to describe relationships between things; like in ER diagrams you have all those tables or entities and you have relationships or state machines, you have states and transitions. A visual representation in that case is way better than anything textual, but interfaces, sequences of things, algorithmic things are much better to be described textually. So it really depends on what you want to do, you just need to make sure that you have a tool that allows you to mix textual and graphical notations, so you can really have separate pictures or texts for your various view points of the architecture description. Maybe one last thing, textual descriptions have one really big advantage: they're textual. So you can put them in CVS, you can diff them, merge them just as your source code and that's a big advantage. It's not this strange thing that is maybe managed in this UML group where a repository tool that somehow doesn't go together with CVS where you store the code.

How do you suggest in that you need to have formal models and do model driven development to document an architecture?

No, not at all. You can use the other things I mentioned, diagrams, patterns, informed definitions or documentation things, and that might be good enough, however I'm a firm believer in the idea that developers shouldn't do repetitive work. So if I can automate things, that's great. Also if you have this big system, like this platform with several applications are built on top of it, the problem is how do you make sure they all use your architecture correctly, whatever correctly is, in the sense you've expected or you have intended it to be used. So you can either do code reviews and policing which is generally not that great, because policing always has this negative connotation and what you really want is you want to make it easy for people to do the right thing and to do the right thing it's always good if people can't make any mistake.

If you have a good understanding of your architectural concepts and that might take a year until you get there iteratively. I'm not talking about big design at front stuff. Once you are at that point it is useful to come up with a language that allows you to describe these things and then just press the button and the system does what it should. And don't require people to remember all the implementation patterns and all the design choices and of course if you want to do that then you have to be formal, because you cannot generate an ultimate if you are not formal. But of course if you are willing to do stuff manually you won't have to be formal, you obviously see my bias that it's not required that you do model driven.

What are the concepts in documenting architectures? What are the typical patterns that you will use?

I am a big patterns fan, so I've been in the patterns community for quite a while. I've written two books on middle web paradigms described as patterns, so I am big fan there. You can use architectural patterns in two ways: one I already mentioned before and that is that if you have something that you want to explain easily you can use the pattern format, context, problem, solution, tradeoffs, to communicate an idea, a concept in a way that people can understand it as opose to just writing down stuff. My experience is that if people use the pattern form, they become better writers because they're guided by certain structures, like in school, first you write the introduction, then you write the main part, then you write the end. I hated that all the time, but patterns in some senses is the same thing.

It gives you structure of how you should communicate information. The other thing is, and I briefly mentioned that before, that there are these huge amounts of architecture patterns around. So if you build a system that uses messaging you have to read a book so you have an understanding of how mass driven systems look like. And then if you describe your system in your architectural documentation you shouldn't reinvent the wheel, rather if you say things like these, the system is fundamentally message driven, let's not use that example, it's simpler; if you build data processing this is a pipes and filters architecture, so it uses filters to process data and pipes to transport the data between the filters, see here this pattern.

Then people who know the pipes and feel this architecture they will read these two first sentences and they know in which way they have to align their mind to understand the system. If you start describing your system by: "This is a system aimed at starting data processing and then", if you do this without mentioning these patterns, then people who know the patterns are lost. So you do your audience a big favor if you reference existing patterns. The other thing is and that goes back to architecting with patterns, how do I use patterns when I build systems.

And one metaphor there always uses like if software design architecture is this multidimensional space, for the beginning let's say three dimensions, then the documented patterns are in some sense fix points in that space. So you start thinking about your system, you wander around, you think about possible solution, you think: "It might be something like that" and you gravitate to one of those patterns, you are there now, you can see: "Is that the right thing, are the tradeoffs OK, is it useful? Maybe it's not. Ok. Let's go wandering again."

You wander around again and there is another pattern. So patterns give you fix points in this generally infinite design space and that is extremely useful, because they have documented advantages and disadvantages, so you know what you are doing. So using patterns for building architectures and for documenting them is extremely useful. If I say you wander around and you notice there is a pattern, this requires that you have to know those patterns. So I think if somebody calls himself/herself a software architect you have to read Martin Fowler's "Enterprise Applications Architecture" book, you have to read Gregory's book and if you're in the embedded world you have to read the pattern books there. You have to at least know that it exists, so if something triggers your thoughts in that direction you have to know that there is something written, you have to go there. You cannot be an architect if you don't do that.

For architects who want to document their own thing, they will still end up in situations where they have things that are not common patterns today, that are not described. How would you recommend to get started writing their own patterns for their own, maybe, specific domain?

Getting started in principle is easy. Just take one of the patterns books and look at the format, so typically it's something like context, or what's the context in which you describe your problem, then you describe the problem and the solution, maybe you describe the forces that guide the solution to the problem and then you describe the tradeoffs and maybe resulting context, where are we now, where do we go from here. So that's kind of the general framework. Picking that up is easy from the books, so the question is not so much how do you get started, while you just do it. The question is how do you get good at it and one thing that I found really useful, I've been at Europlop conferences, not this year, but I've been there the 6 years before, they have this concept of a writer's workshop.

A writer's workshop is basically a group session where an author presents his paper, not by power pointing, but rather basically by not doing anything because the others in the workshop already read the paper and they give him feedback and they give the author feedback on style, structure, didactics, communication and also on content. So if you want to really do this, if you want to get serious about really documenting your architectures well, using patterns, then what you should do is you should start writing two or three, maybe four patterns or things in pattern form; panels always require those three use cases.

So you do this and then you organize a little writer's workshop, where people read this stuff before and give you feedback on it and you as the author, you're quiet in that session, you just understand what people give you as feedback and incorporate that. And if you did that three or four times somebody else is being the author at some point, you rotate. This is a good way of giving people feedback, it's a mechanism that's used by writers, by poetry folks. D. Gabriel who has this, in addition to being a LISP person, has this poetry site to himself. He has a book on this and it's a really good way to go, extremely useful, way more useful than the typical review process in conference for example.

How would a pattern that's specific for a domain or for a project differ from patterns that you will find typically in books?

Technically it doesn't differ at all. I just talked about that; it's the same form, it's the same idea. It differs in various nontechnical respects: one is there might not be the three uses in which case you are formally not allowed to call it a pattern, but forget it. The other thing is it might be proprietary/ secret. I've worked with a transportation company, building a railway transportation stuff, and they documented all their architectural decisions with regards to security, safety and performance stuff. As patterns then of course they didn't want to make that public because it was their know-how.

So it was documented in the same way but it was never ever published. And that gets us to the point where the core difference, except for: "Do you want to make it public or not?" is the scope. If you look at GoF, for example, the context of the GoF patterns is object oriented programming in languages like C++. It's important to add that because the Smalltalk folks are not very impressed by GoF patterns because they can do it with language features. If you describe your own stuff as patterns then the context might be building insurance applications based on our nice insurance platform that we've built strategically and the patterns describe, for example, how to acquire resources, how resources are managed.

Typical cases are stateless, stateful, a trivial example, but why did we decide to build our system as a stateless system. And then you can explain what the context is, the context is you want to have a scalable system, fault tolerant, and then you can say the problem is how do we keep state and still make sure that if one of the notes dies the state isn't lost. You could either repplicate state, you have several stateful severs, but that means you would have to replicate the state over and over and that has n2 bla, bla, bla, and that's a performance problem, so you don't do this. The other alternative would be this, this and that.

So the solution we chose was to use a stateless architecture where the notes don't have state, well maybe they're cached to increase performance, but they don't have the "truth". The "truth" is stored in a database backend that is shared between the various servers. By the way, this is a repository pattern, so if you use this word it's useful. The consequence is of course you make an assumption. The assumption is the database never dies, otherwise this whole thing would never work. Now, is this a valid assumption? Maybe you go to your next pattern and explain how you do this, for example by saying the database is replicated and there is this Oracle highly efficient replication scheme that is way more efficient then if you would replicate the server so we don't have the same problems here.

The example I gave isn't really domain specific, it's a general architecture decision that it's not proprietary for an insurance system, but instead explains why you chose that and while I explained it I referenced certain other patterns and in that case was only one. So that is the general idea.

There are common ways to do architectural documentation is to write up and then the architects throw on to the developers and the developers are supposed to read it and they don't. How do you communicate your architecture to the developers?

That's an extremely important topic. I was once paid a full-day salary for being at a costumer and the only thing I did was I convinced them that it's important to document an architecture with examples. It was a embedded real time so people were relatively formal in their mindset and they thought: "Well, if we document the architecture as basically a sequence of specifications and decisions and rationales then we're ready, we're complete." And that doesn't work. If you download any tool from the internet which means in some sense there is some architecture, some API, you have to understand, you never look at reference documentation. You always look for the five-minute tutorial.

If it isn't there, throw it away; get the next tool; that's how I do it. So if you want to make your architecture extremely accessible to application developers who build application on this architectural platform always use tutorials, walkthroughs, examples. What I like to do is I like to have a 5-minute tutorial, a 20-minute tutorial and 2 hour-tutorial. And it explains in increasing level of detail. The first tutorial should be complete, but only built a useless Hello World thing.

So that is the general pattern I would say for documenting; always go to-down; explain something completely at a certain level of detail, then explain the same thing in an increasing level of detail, then explain it again and get another level of detail and then stop doing that and rather explain specific things. And do that maybe in a problem-solution fashion like FAQs. You basically understand how you build systems. Now what do I have to do to do this particular variation or thing or special case? Explain it, code sample and so on. So always take the developers by the hand and walk them through how to use the stuff you've created, extremely important. Otherwise you will not be able to communicate it.

Are there other ways of documenting architecture, more unusual ways maybe?

The stuff we talked about until now basically assumed you write a document: PDF, HTML or UML document; the model is also static in that sense. However with today's technology it's extremely easy to use other media. For example you could have a video where tow people sit next to each other, or even better, stand in front of the flipchart and discuss certain architectural concepts. And whenever people come into your project you could have them: "Watch these videos.", so they get an understanding of how things work. If you do this you have to make sure that the video isn't longer than 10-20 minutes because videos are unsearcheable.

Well with your technology they are, with these questions on the side, so maybe that's the way to go. The point is you have to somehow index what is going on. The other thing there is that if you want to communicate these examples, these tutorial things, the programming model we discussed just before, you might want to do a screen cast. So instead of explaining things and having 25 million screen shots in your documentation why not simply have somebody program this thing, record it from the screen and the guy who does it explains what he does as he goes. I've used that a couple of time, it's extremely useful.

And then of course what you can also do is the video you are currently looking at, most of the content is transported through the audio channel. The fact that my face is somewhere here isn't that relevant to the topic, so another way of doing things is of course audio, only you could have your architecture broadcast, where you have a episode every week that explains the backgrounds of some of the architectures, like some decisions, some concepts. Tutorials aren't that useful, because you need to see the screen, some screenshots, but if you explain concepts and backgrounds it's very useful.

If you've heard software engineering radio, then you know that if you do it with two people, one asks questions and the other explains it, then the questioner plays the dumb guy and asks again and asks for clarification, then you can communicate relatively complex things with audio only. And the nice thing is of course that somehow is has a different feel to it. People can put on their IPod as they go home and they can listen to this stuff and it's not perceived as work necessarily. It has this somewhat light attached to it. So if you can use these different channels written, audio, video, maybe a little blog of the architecture team that talks about the new stuff is very useful. Another thing that you should consider there is the back channel, so the feedback from the architecture users to the architecture team. You could have discussions going on on a wiki, you can have comments on your blog and of course it's always a good idea to sometime exchange people, so the architects have to eat their own dog food and some of the users can go to the ivory tower and do the cool stuff. So this kind of coordination is important.