A Personal Reflection on Agile Ten Years On
This article is part of the Agile Manifesto 10th Anniversary series that is being published on InfoQ.
I was astonished to be invited to what became the meeting that originated the Agile Manifesto because my work had always been based around building models. The Structured Development for Real-Time Systems trilogy with Paul Ward and the pair of Object-Oriented Analysis books with Sally Shlaer all focused on analysis and design, placing much less emphasis on the coding and testing favored by agile practitioners.
Moreover, I had recently read Kent Beck's Extreme Programming and I was horrified by the lack of emphasis on up-front thinking, abhorrence of models and the deprecation of documentation. However, it was clear that lightweight, extreme, or agile approaches were gaining traction, so I resolved to attend the meeting. The fact that it was in the Rockies in winter had nothing to do with it.
I introduced myself as a spy who hoped to derail their evil plans. While I found many of the positions taken regarding process to be sound (talking to the customer, for example, or timeboxing), the rejection of models was surely unjustified. Yes, there had been an overemphasis on “big design up front,” but surely models had some value. Yes, writing documents (and models) had come to be seen as an end in itself, but surely the correct response is not to throw the productivity gains from modeling out with the dirty documentation.
So just why are models so bad? Because they don’t execute, I was told. Yet my work over the preceding decade had been all about building executable models. Over the few days of the meeting I had the conversation shown in the figure below with almost all of the signatories, sometimes more than once.
This intellectual passing-in-the-night comes about because we had differing ideas about what the word “model” means. Some signatories viewed models (if they thought about them at all) as sketches, to be drawn on napkins for communication purposes then thrown away. What drew the most ire was the view of a model as a blueprint to be thrown over the wall to developers who did as they were told. I held neither of those views. I thought models ran.
Though we were executing models in 2001, it was with our own action language. I heard repeatedly at the meeting that you could not write a program to say “Hello World!” in the Unified Modeling Language (UML). And although you could, in fact, say “Hello World!” in UML, it certainly wasn’t easy. In fact, it was an advertisement for writing code. This had to be addressed if modeling was to become widely viewed as executable.
In 1997, when the UML was accepted as a standard, there were but seven actions you could carry out in a model, and one of those was “uninterpreted string.” (You can certainly forgive the signatories for thinking that models couldn’t execute!) I proposed a standard Action Language for UML. Some folk immediately suggested standardizing on Java. Or Smalltalk. Or …
But an existing programming language is insufficient because it at too low a level of abstraction. It therefore mixes (to some degree) application concepts with their implementation. For example, when we build a particular data structure (a list of Customers, say, with pointers to lists of Accounts), we have made a decision about implementation. In the UML, we would instead express this is as an association, but we would not specify its implementation.
Moreover, when we come to operate on this data in code, we have to take into consideration the implementation we have selected. For example to sum up all the account balances for a specific customer, we are required to traverse the linked list. It would be simpler to say (somehow) “sum all of the account balances for this customer.”
It’s not so much that we want to be able to execute models in UML—we could just add code to achieve that. It’s that we want the higher productivity that comes from operating at a higher level of abstraction away from the implementation. We also want to be able to change our minds about the implementation without changing the description of the problem. The UML model would have the same application meaning if we put Customers and Accounts into a database implementation.
Over the following twelve years, we (the members of the Object Management Group, the organization that standardized UML, and especially Ed Seidewitz from Model Driven Solutions) have produced a series of standards that define the kinds of actions permissible in UML; defined a subset of UML for execution; defined the semantics of that subset precisely; and (finally!) defined a standard Action Language for UML.
In that language, the example above would be written:
myCustomer.account.balance -> reduce '+'
where “reduce” is an operation that “reduces” the collection of balances produced by traversing from myCustomer to all their accounts by applying a binary operation, in this case ‘+’ repeatedly until only a single value remains. Of course, that can then be translated into an implementation using linked lists. Or a database. Or anything else. There are no lists in the specification, nor any traversal of them, just a very compact statement of what needs to be done. (By the way, one design requirement for the language was that it be easily accessible to Java programmers, so much of the syntax looks like Java. The example here was chosen not to obfuscate but to illustrate the power of moving away from the implementation.)
The other signatories were kind enough, back in 2001, to write the manifesto using the word “software” (which can include executable models), not “code” (which is more specific.) As such I felt able, in good conscience, to become a signatory to the Manifesto while continuing to promote executable modeling. Ten years on we have a standard action language for agile modeling.
We rarely see the words “agile” and “model” in the same sentence, but they are not at all in conflict. Rather, modelers can learn a lot from the agile folk (building tests for models early, for example) and those following an agile process can benefit from the increased productivity and eased customer communication that come from sharing executable diagrams with their pair-modeling customers. Surely that’s a win for everyone.
For more detailed information, including critical references, see this link.
About the Author
Stephen J Mellor is an independent teacher and consultant focused on methods for the construction of real-time and embedded systems. He is the author of Structured Development for Real-Time Systems (way back in 1985), Object Lifecycles, Executable UML, and MDA Distilled. He is also (perhaps surprisingly) a signatory to the Agile Manifesto. Until recently, he was Chief Scientist of the Embedded Software Division at Mentor Graphics, and founder and some-time president of Project Technology, Inc., before its acquisition. He participates in multiple UML/modeling related activities at the Object Management Group, and was a member of the OMG's Architecture Board, which is the final technical gateway for all OMG standards.
Mr Mellor was the Chairman of the Advisory Board to IEEE Software for ten years and a two-time Guest Editor of the magazine, most recently for an issue on Model-Driven Development. He is also adjunct professor at the Australian National University in Canberra, ACT, Australia.
Pedro J. Molina
Nothing is more agile than an executable model that needs no coding at all.
Executable model - a utopia?
But, agile approach also has a problem which is currently not very fashionable to talk about. The question is "can a coherent solution always emerge by adding individual features?"
Executable UML - A misunderstanding of boundaries?
However, I think executable UML itself is a serious misunderstanding in itself. Once you execute something, it is a programming language. That means, either through libraries, language constructs, or code conventions, you deal with the same problem: get your abstract thoughts to actual implementation.
Therefore, its model is just as complex as in any programming language: it's not the model anymore, but the whole system.
Simply the laws of complexity regarding to the Turing machine kick in...
How could this happen?
I think our communities have two misunderstood boundaries, and maybe they are related to each other.
The first misunderstood boundary is of the position of plans in engineering. Never ever any single building was built purely based on its plans I believe. A plan is a rough understanding on what we want to achieve.
The sole existence of Software Engineering is based on the assumption that you should think it over before you try to make it run; going straight to coding is just as nonsense in this regard than trying to run the plans. I think that on the above discussion, I would take a position which is opposite to both parties: you shouldn't run your models, but you should have a - disciplined - method to think through what you want to do first!
The more advanced a building gets, the less formal are the plans: I remember, when we built our house, structural plans were taken with rigor; but then, the size of the toilett was measured by the mason by sitting on a chair, instead of tools. Interior plans were nearly never taken seriously.
The more we go into details, the less elaborate the plans are, and, I believe, the less elaborate they should be. UML taught me what are the crucial things which won't change:what to look for at the first day, how to build something when we don't have too much information, which will still stand the time. Not considering implementation details when dealing with the details - like, not considering what kind of data structure should we use when we're dealing with a reduction - would be irresponsible in my point of view. We should use a model which express this: the model encoded in the programing language.
The second misunderstood boundary is: who's the mason of this building? Who is the mason of the electric dreamcastles we build? For years, we thought it's the low-level programmer. This is simply not true!
Every single computer language is an abstraction on the electronic runtime, of sparks and silences. We don't give a computer a castle: we give a computer a full blueprint of a castle: hey,computer, when they request you to build such a castle - by clicking program.exe - this is what you should do!
So, every single programming language is therefore a full modeling language on its own right, with a full range of abstractions.
Now I understand that Java has a lot of implementation details in it always, even if it hides some of them. To the contrary, python has less of them, for example.
I don't like the level of abstraction of Java personally, but I think every single programmer would agree with me - even Kent Becks, I'm sure, after reading Implementation Patterns from him - that even in code we should build different abstraction layers.
I don't think java should be the modeling language where we express rough understanding of use cases; but on the other side, I don't think UML or anything coming from the modeling/ system engineering community should be the language where we discuss the details, let alone try to execute it, horribile dictu, in production, not just in simulations.
As the fight we should have for modeling is not what kind of end-to-end language we should use: the fight is about how to think about unknown territories.
End to end feedback
Very early feedback on understanding need/business process model, workflow, functional UX (user experience), applicability of an algorithm
-> simulate solution on meta level (e.g. UML, BPA/BPM, ...)
Verify earned business value on technical solution in a few weeks including design feasibilty, architectural viability, non-functional behavior incl. performance aspects of UX, immediate Re-action on e2e feedback, shifting technological platforms
-> proof-of-concept / iterative e2e story development on target
So choose or do both (incl. set-based concurrent design for the latter)
Mixing Agile & Models
At OutSystems, the company I work for, we build the Agile Platform, which allows building enterprise web applications using models. To make sure our projects run smoothly, we use Agile methodologies to manage them - and we've been very successful with this approach.
The language we use is a DSL, not UML. It can be argued that this is merely an abstraction or a graphical representation of the code of the application, as Adam already mentioned in the comments. But that's fine!
The goal of these models and DSLs is to raise the level of abstraction so that developers can spend more time focusing on what the business needs, instead of struggling with the details of the system. The big challenge is for developers to trust the applications and code generated by these platforms as much as they trust the JVM.
Once they do, they can be way more productive, deliver software with higher quality, and quickly deliver and adapt applications as the customer needs... Or, to put it in another way, they can become more Agile! :)
Nothing wrong with DSLs
What I am opposed to is having general-purpose execution languages which claim to be better than state-of-the-art programming languages: at that moment, the "restricted view" definition of the model disappears for me.