BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Interviews Jim Coplien: Why DCI is the Right Architecture for Right Now

Jim Coplien: Why DCI is the Right Architecture for Right Now

Bookmarks
   

1. I'm Sadek Drobi, I'm here with Jim Coplien at QCon 2010. Jim, can you tell us a bit about yourself and what you've been up to lately?

It's wonderful to be here at QCon. I am a consultant these days, I'm living in Denmark and I've been doing a lot of work recently both in Agile and in architecture. I guess what I've been up to recently is this year actually enjoying life a little bit more, taking a lot of vacation. I just came back from a week in the Finnish woods, having a good time skiing and thinking that was fun. I've been working on a book and doing some training and traveling a little bit, just having fun, keeping connections up.

   

2. Jim, as a programmer, I've been looking around. I'm seeing a lot of languages around and I try to discover what there is on the JVM, Scala or whatever, Java, Clojure - a lot of languages. I'm seeing even on the .NET side of things there is F#. Looking at these languages, trying to discover them, I see some similarities between languages, some of them overlap, but they do different things in a different way. I see a lot of things, but I don't know how the think about them. Jim, you are the paradigm guy, so why don't you tell us about this kind of things - a lot of languages, a lot of paradigms, how to do about them?

Looking at this from the point of view of design paradigms, it's getting to be an interesting time because of languages like Scala and F#. There have always been functional languages around, Scheme has been around for a long time and CLOS (the Common Lisp Object System). They are very interesting languages, but they have not been culturally compatible with C, basically. The C language really dominated programming certainly through the 80s and the 90s. And you find people like Richard Gabriel, who is the CLOS and Scheme guy kind of bitterly saying that C really killed programming. C is the last programming language.

It may be called Java, but it's still C. I tend to agree with Dick. At the level that's important, which is a level where we think about things, that C and C++ and Java and C# are all the same programming language. They are all in the van Neumann paradigm. We tend to think about things imperatively. Maybe we package things into these things called objects and sometimes it makes things a little worse, sometimes a little better. But what's happening now, is it seems that, for whatever reason and I think the reason is multicore, is that these functional languages have become at least fashionable, probably for all the wrong reasons.

And I think the expectations on these languages are far too high in terms of being able to solve the problems and people foresee that the multicore and parallelism are going to provide. Nonetheless I think it's healthy, at least for the thinking people, because it's going to get the thinking people interested in broader areas of language and exploring alternatives to this rut that we've been in, which is the C language rut for so many years.

   

3. I've got for example in C# today, multi paradigms, because of the evolution or even in Scala the same example goes. I can do things with objects, with inheritance, with generics, which is something like templates in C++ and with functions. Here I have a problem and now I should find what to do with this, which one to choose, which one works for me better. What do you say about that?

This actually brings an interesting discussion to mind that I had many years ago. This was probably in the mid 1990s and it was between me and my good friend Tim Budd. Tim Budd is at the University of Oregon, in the US or maybe it's Oregon state, somewhere in the state of Oregon. I wrote, as you know, a book called Multi-Paradigm Design for C++ and Tim Bud also wrote a book and his was called Multiparadigm Programming in Leda. He created a programming language called Leda.

He and I discussed what do we mean by paradigms. To me, it means the things you just said, templates and procedural and objects, but that isn't what Tim means. To Tim, it's things like functional programming, rule-based programming, procedural programming and object programming. We can have some minor religious arguments, because I can say really that procedural programming and object programming are almost the same computational model, but what Tim means by paradigm is computational model.

So the rule-based paradigm is a fundamentally different way of thinking about things than procedures or objects, as is dataflow programming or functional programming. He was maybe a little ahead of his time, at least seeing the current hype about languages like F# and Scala and so forth. Scala has a lot of real elements of functional programming in it, and a little of applicative programming, C# really doesn't. C# is just kind of a very clean very well thought out Java in which you can do things right. And I think it avoids a lot of the mistakes that Java made, but Scala is something different.

I don't know where this is going to go in terms of really succeeding in the marketplace. I've been doing a lot of work with Trygve Reenskaug just on object orientation and getting it kind of cleaned up and figuring out what kind of real language support we need for objects over the past 3 years. I think we have a very good story to tell about that now. But it's very clear to us, crystal clear that over the past 10 or 15 years that people really haven't understood what objects are about. I don't know if I hold out much hope for people really grasping what functional programming, applicative programming are about, how they work with each other, what the relationship is between functional programming and objects to get applicative programming so you can work in a stateful or real time environment. There is a lot of maturity that the industry is going to have to find somewhere and I don't know where it's going to come from.

   

5. Is it the mistake of C++ or the class-based object-oriented programming? Is it their fault?

It's easy to lay it at the feet of a programming language, but even in parallel with C++, probably the most outspoken person about classes being fundamental is Bertrand Meyer, who's one of the most brilliant people on the face of the earth. Eiffel, in its own right, is a brilliant programming language and it's really thinking in terms more of design than of just programming language syntactic sugar. Even he didn't really fall into this vision that people like Alan Kay and Nygard and Dahl or Trygve Reenskaugabout what objects really were supposed to be.

He got caught up in this class oriented view of things. The Smalltalk people, when they put together Smalltalk originally the computational model was exactly right, in terms of thinking in terms of objects. And then they screwed it up with the language. The language is so class focused! Most languages that we're saddled with today had made this error. C++ was not the first language there. Smalltalk made this error before C++ did. I don't know exactly where Eiffel is in history, but it made about the same mistake, so did C#, so did Java. All of them have tripped over this same stone in the road.

I don't know if this has to do fundamentally with the Western way of thinking or maybe it's just whoever it was there first. Smalltalk set some expectations and everyone else just followed that way of thinking of things and we're stuck with that. I don't know. I don't know how we fix this going forward; I don't know how we avoid falling into another misinformed rut. It's a really serious problem. I've seen it be the most serious problem facing the industry in any area of progress or any endeavor in my career over the past 30 or 40 years. The same thing happened with patterns, with object oriented programming - you name it. The good drives out the perfect.

   

6. Let's start with object orientation. What is for you a correct object orientated computation model?

First of all, I don't know if that's the right question. I just came from this lovely talk on "What's all the buzz about functional programming" and the speaker very nicely said there is no single answer and even if functional programming is here because of multicore the expectations are way too high. I can tell you what objects are, but it's with a big footnote that says "So what?" We can define the word - "Object" is a word we use.

I've learned a lot in the past 3 years in fact working with Trygve Reenskaug who's the inventor of model-view-controller and he was there with Dahl and Nygard in the early days of Simula and I think has carried their vision forward and we're now taking that vision forward into this DCI paradigm. One of the things that most people miss is that object oriented programming is about objects. Most people are playing with classes and there have been some glimmers of hope and light from people like Dave Ungar, who have the Self language, which is more in the direction of thinking about objects.

Academia has never really caught up with this idea of thinking in objects. Thankfully, some of the programming languages are slowly starting to move in that direction in a really almost unconscious way, like Ruby and Python. It's a lot. If you choose to think in objects in Ruby and Python, you can think in objects. That is you are really programming objects and the class or the module is just something we need to put in there to get the objects. On the other hand, you can write perfectly good C++ code in Ruby or Python, as well, and still be thinking in classes.

This raises a broader question about really how much does language matter. I mean I can give you Ruby and, if you've learned C++, you're still going to write C++ in Ruby. So, it has to do with how you basically think about things and maybe that has more to do with our culture and here culture might mean our education in University or in the profession than it does with language. I really challenge the so-called Orphean hypothesis in the programming language world.

I really don't think that language constrains or shapes that much about what a great designer can express or does express. I think language is greatly overrated and we need to look beyond language for the solution. Language can help. Good languages can entice people and excite people and open minds, but it's not enough. It really comes to changing world views.

   

7. Exactly. In languages like Java you can do instances, but then there are patterns and everyone talks to the singleton pattern so that everything turns into classes. What is the problem? I don't know. Java enables you to do as many instances as you want and actually it was optimized for doing objects, but people still talk to the singleton pattern, the only one on the list of patterns for not doing instances. Is it a usability problem of the language or is it people that are not reading enough? Where is the problem?

If you bring up patterns, I think that's an interesting touchstone with respect to this problem. The great people who use patterns, never use the word pattern. They just do the right thing. Christopher Alexander, which is where we get this idea of patterns from in architecture , says that patterns are just training wheels. The words he uses are "Patterns are the gate through which we pass on our way to enlightenment." If you are really using patterns, you are not saying "pattern, pattern, pattern." You are just doing the right thing and you know when not to use patterns as well.

Very often I'll go in and I'll do an architecture review at a company. They'll say "Here we're using the visitor pattern. Here we're using the adopter pattern." These people are still programming in terms of training wheels. I kind of get scared. If we're really a profession, someone who needs to put in those kinds of comments should not be allowed to touch code that's going to ship. OK, because they are still in school, they are still in training mode; they still have their training wheels on.

This whole excitement about, for example, the singleton pattern and talking about that shows that even the level of dialog about the problem has not reason to the right level of maturity yet. The use or abuse of the singleton pattern has not been the single success or killer factor in any architecture I know of. Yet, people get religious about this. People get religious about details that don't matter and very few people are talking about the big picture.

Now, everyone is talking about functional programming as being the solution tp multicore. We just came from a talk that says "This is all about having a stateless world. We don't have to worry about side effects, more or less." But there are states in the real world. OK? There are well defined demarcations of time, there are points in schedules, there are real discrete events in the real world. And most software is based on events and messages. That doesn't have anything to do with the von Neumann paradigm, that has to do with real world happenings.

That doesn't fit the functional model at all, unless you have a very degenerate form, where each event creates a new instance of the software when you execute it. There has been very few people who thought about how we integrate the real world of events or the real world of state in object oriented programs with functional programs. We did a lot of this in Bell Labs. There was a guy named TomBurrowswho was my office mate and we put together a wonderful environment.

It's still the most wonderful environment I've ever programmed in that's called GIL (Graphical Interface Language). It was a symbolic reflective language; it had polymorphisms, so it had the kind of object-oriented flavor that we find in common object-oriented programming languages. But the computational model was basically a functional model with kind of a loop built into it where you could have states. Every time a state changed, you propagated that state change into the functional tree and did a reevaluation, updating your states. You had this huge clock cycle that ticked every time there was a state change in the real world.

This is a paradigm that we actually used to write real code for real systems. Some of this shipped in the control consoles of telecommunication systems and monitoring consoles and we develop a lot of understanding of this and it's great. I'm seeing the same ideas come up here 20 years later. My question is how do we get the average graduate of a University to understand these things when their professors haven't given them even either the challenge to understand them or the professors don't know? Or, if the student answers on the exam, something from this kind of paradigm "No, that's not right! You can't do that in Java!"

   

8. Do you think Java is taught too early in the University? Often, everyone is happy to learn Java in the University, which is a way too specific language to be taught at the University. Is it maybe one of the problems? Or we have way too many programmers because of simple programming languages? I'm searching for an answer. Maybe it's in the University.

We're all searching for answers, but I actually think what's important is to formulate the right questions. What should the industry look like? Should everyone be a programmer? Should my mother be able to program her stove? Is that a model of society in the future? Or should programming be left in the hands of the elite few who really know what they are doing? I think there are many answers and there is a broad spectrum here.

I don't want someone who has the academic credentials of a 2-year degree having learned Java writing the software for the flight control of the airplane I'm going to get onboard. There was a conference in Finland where there were actually some people who were writing software for airplanes and someone asked the audience "Would you get on board of the airplane if you knew your software were on there?"

And one guy raised his hand, the rest would not get on the airplane. Why did the one guy raise his hand? He said, "I would get on there because I knew that if it were our software, the plane wouldn't be able to take off." That's the level of professionalism we talk about. I wish that uncle Bob was here (we've been in front of the camera before) because to him professionalism is doing TDD (Test-driven development). Well, that isn't going to save us from this kind of problems. Professionalism to me is that on a much higher level of how I am going to be responsible or answerable to society for the code I built.

That has to do with a lot of thinking and a lot more than just technique can give me. This gets into morality. This gets into value systems. This gets into the systems thinking and for systems thinking you need to get beyond where we are with programming languages today. This is one of the reasons I'm excited about architecture these days. That starts to get us beyond the programming language limitations and into more systems thinking. There is an interface between those and we can use programming language to express architecture. And that's a great delivery vehicle for me. I risk getting people enamored with the programming language tricks and missing the big picture, as well. But the goal is to get them enamored with the big picture.

   

9. Often people miss the concept and go for syntax or for whatever sugar of the language offers. You've been working also on this DCI domain context interaction pattern and I've read the papers and articles you wrote about this and sometimes I feel it makes sense. I read that there is a problem about object orientation. Can you describe this? What is problem about object orientation that you are trying to solve with this pattern?

There are several ways of talking about it. DCI is data context and interactions. It was an idea that was the brainchild of Trygve Reenskaug. And Trygve and I have been working together on it now for about 3 years, and now there is a whole bunch of other people who joined the effort. One of the main guys has been a guy named Rickard Oberg who's from Stockholm. Sometimes he works for a company called Jayway and they've actually shipped product using the DCI paradigm.

Where do I start in terms of limitations of objects. A lot of the limitations have to do with the fact that people think about classes. When they express what they want to do in software, they need to express it somewhere. They express it in code and in these classful languages, like Java, C++, C# and so forth, we have classes, and that's where you express it. You need to express two kinds of things.

One is the basic form of the system. If I'm writing a program like a view graph builder, I'm going to have shapes like circles and rectangles and pictures and those are objects. The object paradigm is fine for that. And those objects have operations on them, like resize and recolor and move and they are very primitive. That belongs in the object, that's where it belongs because it doesn't do much with anything else. But more serious operations like I'm in a text editor and I have spell checking, which involves the thing I'm spell checking as well as a dictionary and a screen and then some interaction with the user to say "Should I change this word or not?" is not really a method on any object. It's just something that we have in our head and current programming languages and object methods have not given us a good home for things like spell checking. For similar reasons, they haven't given us a good home for concepts that come up in the real world, like phone call or bank account.

Bank account is not an object. I don't go into my local bank and go into the safe and there is a bag with my name on it on some shelf. I mean that's not an object; it's a collection of responsibilities that are computed dynamically on the basis of some transaction logs and audit trails and other data that the actuaries own. We've always had a hard time finding where to put these concepts in object orientation. That's the problem that we're solving - you take the use case part of the system that is what the system does and you express it in something that isn't a class.

What we're using to express it is a collection of roles because this is how people tend to think about the activities of the system. It is in terms of roles that objects play rather in the total interface of an object. It's just like right now you are Sadek Drobi, you are a person, you are an object. How many methods are there in your interface? There is eating, there is dancing, there is going to conferences, there is lots of things.

But right now, in your role as an interviewer, it's your role as an interviewer that I care about. I'm working with a slice of your interface and I have my interface and Diana has her interface sitting back there running the camera and we, as roles are interacting without having to put everything together in one interface. In our old style object-oriented programming there was one interface that mixed the relatively stable data part of the architecture together with this rapidly changing dynamic part of the architecture that supports the use case. We want to separate those out.

This is something that architects have known for years, even about buildings. In building they are called sheer layers, like the sheer layers between layers of earth in an earthquake. Rubchinsky?was an architecture critic who talks about this phenomenon in architecture. This is where you are bringing it into the object style of doing architecture in the real world in object oriented programming. Again, this is just for object oriented programming and this is just for object oriented programming and that's just one tool, so it's not the total answer. And Trygve and I are asking right now what does this look like in a concurrent environment. That's a whole other set of paradigms we're going to have to bring in here.

How does this integrate with model-view-controller? How does this work in an interactive environment? So, there is still a lot of work to be done, but I think we've really clarified a lot of what object orientation should have been and now we know how to do it. We are actually deploying systems using it and that feels good.

   

10. It feels like it's very hard to move people towards something or at least to let them try it. There is a new concept and if it's not a programming language, people are not very interested in concepts, I feel. It joins something that we told minutes ago, right?

Every idea goes through three stages, I read the other day. People will say it's irrelevant, then they say it's wrong, and then they say it's obvious. When Trygve and I published an article on Bill Venners' web site, artima.com, that was kind of the first big integrated article on what DCI is and then there was a blog where people could interact with us. There were a number of people that said "This is the stupidest article I've ever read.

This is totally wrong! You do not understand object oriented programming!" Here is someone telling Trygve Reenskaughe doesn't understand object oriented programming and Trygve and I saw this, we looked at each other and smiled and said "Congratulations! We've succeeded". We have an idea that's good enough that it's made people in the rut uncomfortable.

Maybe they don't even know they are in the rut or maybe they don't even know they are uncomfortable, but they can't even see the new vision. They're blind because of what academia and culture have been feeding them. We know we're onto something very important. We know it works. We even have some metrics now, we have some measures to show we're really on the right track. We have a deployed system that we know it works, but there is tremendous resistance.

I don't know when the clock should start ticking. From the time when objects first were really realized, about 1967 until they really became in use was about 17 years. 1984 or so is when you started seeing things come alive in the object community and it may be the same for DCI. So I don't know where the clock starts. But give it 13-15 years, then you'll start seeing people say "Of course, this is obvious." If I live that long, I'll look to see if people are doing it then.

   

11. If we go concretely into things, like I do object orientation and my problem is that often I search the code for where is the algorithm, where is the real thing that has been happening and I see it spread everywhere into polymorphism. If I don't have a very advanced IDE I don't get what's going on. Is ta problem that is solved with DCI?

That's exactly the problem we are trying to solve with the DCI. I can look at the algorithm in the source. The object oriented people were right in that objects should be kind of intelligent at runtime in doing their part and only their and to the degree that they know about other parts it's a bad thing. Because then the coupling and cohesion are wrong. Objects should be relatively isolated, but we don't code objects, we code classes.

There has been this confusion. With respect to the algorithm that you can't find, I should be able to find that algorithm because its parts do belong together and you're right - the object religion says that we have to put this algorithm into a barrel and blow it up with dynamite and then allocate the fragments to all of these classes. That's called allocation of responsibilities. But we're constrained in the way we allocate responsibilities by the domain partitioning of the classes.

Good classes follow Eric Evan's domain driven design, but that isn't the ideal partitioning for carving up the algorithm. Roles are the ideal partitioning for carving up the algorithm into parts or chunks that you can understand and then you need to dynamically map the roles onto objects at runtime. But we want an administrated entity where we can reason about the algorithm the same way I could reason about it in Fortran.

There, I said it! I said the "F" word - Fortran. Procedural programming still has its place for some definition of procedural programming. If it quacks like an algorithm and flies like an algorithm, and it walks like an algorithm, I should be able to deal with it as a programmer, like an algorithm. That's what DCI is about. It is restoring the identity of that algorithm while still honoring all of the principles of coupling and cohesion of the object paradigm.

   

12. To be able to use DCI, I need some support from my preferred programming language. Are there now languages that support the DCI paradigm?

"Support" is one of these fighting words. It is possible, in C++, using templates. We've known how to do traits and templates for 20 years. You can use these traits to inject these roles, which in C++ we represent as classes into the domain classes. Scala does it very directly. Scala has first class citizens called "traits" and we can use those to capture the roles. It's really beautiful in Scala.

There are stylistic things you can do in Ruby and Python that are very elegant, extremely elegant ways of communicating this. They are 90% there. There isn't something called a role in Python or Ruby or Scala or C++, but you kind of do it. So far, we have only found one language in which you can't do this. I said this 2 years ago at JAOO, that's Java and that's interesting in the sense that we had a lot of people try it and the solutions are all just unbelievably ugly.

My hope is that we can push the Java community into a direction where they evolve the language to be able to support this. There is no limitation in the JVM. This works in Scala and Scala can compile to the JVM. The Jayway people in Sweden have created an environment where they put some annotations into the Java code that allows it to do full boom DCI. This is the first real commercial application of DCI and it works.

The syntax is a little ugly because it's based on annotations, but if we can push the Java programming community into adding that to a forthcoming Java standard, then we're in good shape and that's my hope right now, that's my dream. I'm doing a lot of work with people like Rickard and other people at Jayway to push this forward.

   

13. Moving to Agile, I've been at a good keynote by Martin and he listed a lot of things I should be doing, like testing and things like practices I should be doing all the time. I can swear to you that all my colleagues do it, but still I'm pretty sure they produce bad code. What's the problem?

Bob and I always have an interesting dialog. Most of what Bob says by all means do it. I really like his book, Clean Code. There is no rocket science in there. It's just have good indentation, have very descriptive identifier names, keep your function short. All of these things are just hallmarks of someone who's competent. I don't know if it's enough to call them professional, but at least it's a sign of good housekeeping. However, the important things, the things that really matter, where the leverage is, is in architecture and in system thinking.

Bob knows architecture very well, too. We have slightly different perspectives on this, but we have some very strong shared perspectives on architecture. It's just that Bob has chosen not to make that the thing he talks most of the time so far. He's talking about kind of the training wheels because that's where the industry is and Bob knows that. What Bob is telling people is perfectly appropriate for his audience, Bob knows his audience. I have maybe a slightly different audience and I'm trying to raise a bar a little bit.

So I'm trying to say "Let's think in terms of architecture. Let's think in terms of domain knowledge. Let's think in terms of the mental models of the users of the system and the domain experts and do this at a level of systems thinking." That perfectly complements the things that Bob talks about most of the time. Once in a while, we'll have a very nice agreement. I don't know if I'm going to show this in my talk tomorrow, but I showed it in my tutorial where Bob Martin is talking about architecture and saying "There were the days and we thought that just throwing a few tests at the system and running a few tests are going to be enough to give you good architecture."

(Pardon me, but can I use a bad word on this tape? Never mind, you can find it on the web, it's called the Scatology of Software Architecture - given the word "scatology" you can guess the rest.) We have very important points of agreement and by all means, keep your house clean! Software and programming are a discipline and Agile does not mean laid back California guitar playing programming. It is a hard discipline that requires getting up and brushing your teeth in the morning and scrubbing your code and relentlessly refactoring it to make it beautiful.

I like how Bob is leading the charge of this in the industry. On the other hand, we also need the systems thinking and that's what I've been encouraging people to do. Think in terms of domain analysis. Think in terms of the big market picture. Think in terms of mental models of programmers and the mental models of your end users. Everything matters.

   

14. Thinking about Agile, looking at the Agile Manifesto, it doesn't have anything that talks about this ability of a software. It does work in software, but not usable software.

If I were to do a new edition of the Agile manifesto, I'd change "working software" to "usable software" and I'd change "customer" to "end user" and a few other changes in that regard. This is a bug in the Agile manifesto. I saw a posting in fact from Alistair Coburn, who as we know, was very instrumental in the manifesto and he said the same thing. He says "We blew it. We missed this issue of the usability of the software."

Someone asked me in another interview the other day "What's next? What's the next big thing?" and I said "I don't know what the next breakthrough will be, I can tell you what some of the new feds will probably be. If I had any wish of what the next breakthrough would be or the next thing people get excited about it's not going to be functional programming, it's user experience." It's getting into the psychology of computer programming, all the way from the kind of things that Jerry Wineberg talks about, into the psychology of the programmer, into making mental models of how people think about the programs they are using.

That's where I wish the next great products would come from. Again, people who come into programming, come through academia and they usually do not have programs in psychology or mental models or even in interaction design, or ergonomics or just the basics of the mechanics of design of good interfaces and it's all left up to a tool. "Use this, GUI builder and everything will be fine!" and nothing is further from the truth. Good interface design is really hard and there are very precious few people who are skilled in that in today's world and we need more of these people.

   

15. Are you suggesting that usability starts from code that a programmer writes?

I don't know if it starts at code but it stops at code. It involves code. Kent Beck used to say "You cannot hide bad code behind a good interface" and I totally agree with him. There is not much Kent and I agree on this level, but it's good to agree with Kent on something. This represents again another fundamental misunderstanding of what object oriented programming is about. Alan Kay's vision is a thing called The Dynabook, where at birth, every child is issued a computer, which is an extension of their mind and processing ability and of the memory.

You get technology upgrades and everything, but the idea is that the objects that are in your human mind are also in the mind of this computer and should have as seamless an integration between them as possible. That's why Trygve did model-view-controller, to make that link between the memory of the computer and the memory of the human being. Good objects are about capturing the end user mental model in the code, in the program and most object oriented programmers don't think this.

They think what object oriented programming is is encapsulation and coupling and cohesion and polymorphism and interfaces and classes and they completely miss the big picture, which is this about human mental models. Even if we can get that into the mind of the mind of the object oriented programmers into our curricula is being what objects are about and how you find the right objects and then reduce them to classes. That would be a start, but my fantasy would be that we do this at a much larger scale.

The main focus of programmers resides in understanding the mentality and psychology and thought processes of their markets and of the people in their markets, rather than looking for the golden chalice of coupling and cohesion. They have something to do with each other, but the focus should be on the people.

   

16. One of your papers was the inspiration for Scrum and Agile. Your paper was cited by Jeff Sutherland and it was the inspiration of Scrum.

There is a small part of Scrum called stand-up meetings that was inspired by some organizational patterns work we did and in particular some work with Borland and Quattro Pro for Windows. This notion of the Scrum, the daily Scrum at the stand-up meeting, the inspiration for that came from some work I did.

   

17. What do you think of the state of Scrum today? People seem to be getting more and more certifications. I'm afraid it's getting into a certification kind of business or what is it?

I am a Scrum trainer and so I am in the middle of this. In fact, I resisted being a Scrum trainer for about a year and a half or 2 years, just because of the problem that we're seeing now. Most people who are practicing Scrum think of it as a set of techniques. I go home and I do a product backlog and I do this and I do that and then I'll be Scrum and I'll pass the Nokia test (All hail the Nokia test) and I can say that I'm doing Scrum.

I think for the good people who are honestly trying to do Scrum, that's kind of the state-of-the-art as people have this checklist and they are saying "OK, am I doing all these things?" and that's probably worth something. Scrum comes from a little bit of my work, but most of Scrum comes from this paper called the New Product Development Game by Takeuchi and Nonaka and this goes back into a lot industrial practices in Japan and one of the key ideas that Jeff Sutherland has about Scrum and it reflects the Japanese roots is this notion of Kaizen.

I was having lunch in Finland at a conference a couple of years ago and there were some guys sitting around the table and it was all guys and you know what happens when you have guys at the table. It's this chest beating thing going on. So someone said: "We're Scrum because it gives us productivity" and another guy said "We use Scrum because it gives us high quality." Then there is this other guy who is kind of humble and he says "We use Scrum because it shows us where our problems are".

That's what Scrum is. Scrum is something that makes problems visible, it is about openness and honesty and visibility is about making problems visible and there is kind of a morality behind this. In contemporary software, if you found a bug, "We're going to fix you, we're going to reeducate you, we're going to have tools to find these bugs. We're going to make sure you won't do this again". Whereas in Scrum, if you find a bug, it's like "Oh, you found a bug! Thank you! That shows an error in our process.

Now we have information. We can improve our process. Thank you for finding that, for creating that bug. That's shows an error in our process." Scrum is about continuous improvement. Very few people who practice Scrum get that. There are a lot of people throwing around terms like Kaizen. I was in Japan about a month ago and one of the things I learnt there is that you can't do Kaizen without Hansei. Hansei is this very deep sense of sadness or regret that is going to drive me to make things better. I have to feel honestly sorry and regretful about what I'm doing.

Again, this is about feelings and it's about people, it's not about "OK, we will do the mechanical procedure to do process improvement here because we have found this bug." No, this is about people and changing people. Jeff Sutherland understands this and I don't think a lot of the Scrum community does. I'm not even sure a lot of Scrum trainers understand this. I think the state of Scrum is it's become sadly kind of another methodology.

If Jeff Sutherland were here, he would be very alarmed by this. Scrum is not a methodology, but I think that's how most people are adopting it - the set of checklist and practices that they can say "Well, no one can blame me, because I can show you that I pass the Nokia test." And you should be looking to be blamed all the time, because that's the only way you improve.

   

18. You just told that you are working on a book called Lean Architecture. Can you tell us a bit about this book?

This comes out of the work that I did with Trygve on DCI as well as earlier work on domain analysis and a lot of the work of my wife Gertrud Bjornvig on requirements and on what requirements formalism to use in Agile. Of course, there is a broad variety of these. The book isn't only about nerd stuff, like DCI and domain analysis and what the code looks like and what you should deliver, but it's also about the people. What are the roles and how do they interact.

The main theme that goes through the book comes from Lean, it comes from these Japanese practices that underlie Scrum and that's a single piece continuous flow, where the whole team works together on every element. The whole team does analysis, the whole team does design, the whole team does implementation. This is how Canon and Honda and all these other companies were working as described in this paper, by Takeuchi and Nonaka that inspired Scrum.

This is where the real quality and productivity (if you care about productivity) come from in Scrum. What we've done is we've brought that notion over into software architecture. People have a really hard time getting this. Earlier we talked about people have a hard time understanding programming language. People have a hard time understanding this single piece of continuous flow. In fact, a lot of people who have tried Scrum, have now given up.

Now, there is this new fad called Kanban, where instead of having single piece continuous flow, everything is done in silos, it breaks down team work, you're doing things in discrete steps and this to me shows a very deep fundamental misunderstanding of what good development and team work are about. But I think what it is it's a fallback position for the weak, who are unable to make teams work, but they can prove that they have a process and they show they have a process working.

Just like a Java programmer can say "OK, I'll show you that I'm programming in Java. Therefore I must object oriented". I'm using Kanban therefore and because that's such a fancy Japanese word, I must be very good." The level of systems understanding in the current industry is very poor and that's why I wanted to get out our vision, Gertrud's vision and my vision, about what system development should emphasize in this architectural perspective.

It's not just a vision. This is something we've seen work in dozens and dozens of companies, for many years. It's not rocket science, but it's just something that people are doing. As we say in Scrum, Scrum is all about common sense, but common sense is very uncommon and that's what a lot of it is.

   

19. Did you cross teams with more than 3-4 that really works well? Maybe it is about the team size. When we get a lot of programmers or the team a lot of members, it just doesn't' work.

You're certainly right, but words mean things. I know that Diana Larsen put up a definition of team. This was at a conference in Florida, about a year ago, where a team is a group of 5 ± 2 co-located people working towards a common goal. I remember when I was in AT&T we talked about the 5 ESS team, which at that time was 500 people. 500 people are not ateam. There is something, and maybe they have a common goal, but they're not a team.

Words mean things and this word "team" means a small group, so of course teams are effective, but to be effective they have to be small. The more experience we have with Agile, the more we're finding. In the earlier days it was seen of 7± 2 people, because it was this magic number that comes out of the short term memory analysis in human psychology, which is 7± 2, which people throw around a lot. We're finding that 3 is better than 5. It's small teams that can really perform.

Size has a lot to do with it. That's something we found in a research behind the organizational patterns book. In fact, there is a lot of empirical analysis in there where we look at team size and correlate that to things that matter to the business. Small teams are important.

   

20. I'm going to ask you to recommend 3 books to our audience.

Let me start with the first and foremost, Mythical Man-Month. If you haven't read Mythical Man-Month, read it. That would be number one. Number two, just for fun, Zen and the Art of Motorcycle Maintenance, which was a pop 60's book, but I think the insightful reader will find some seeds there of systems thinking and thinking about design in new ways. I guess a third book could be a toss up between read something by Christopher Alexander (it doesn't matter what it is) The Oregon Experience, or The Timeless Way of Building, or A Pattern Language. Or read the architecture book that's coming out from Wiley that Gertrud and I wrote. The reason I'm recommending that (I'm not doing this interview to sell books) is because it covers a lot ground.

It covers the human side and the technical side and integrates them together. I'm realizing here at the end of the interview, given our discussion, that this systems thinking is a very important thing we need to get people to. I think what a book can do is get people from their nerd-centric perspective on the code gently into the systems thinking and plug in all the little pieces in between that make the connections, that connect the dots. I'm sure there are other books that do that too, but I have a hard time thinking of them.

May 29, 2010

BT