Stuart Halloway on Clojure and Functional Programming

Recorded at:

1. I am Sadek Drobi. I am here at QCon with Stuart Halloway. Can you tell us a bit about yourself? What you have been up to?

Sure. I work at Relevance in North Carolina; we are a small Agile shop. We have been doing a lot of development primarily with Ruby over the last several years and in the last twelve- eighteen months have gotten very interested in Clojure programming. So I wrote the book "Programming Clojure" which has been out for several months now and we have been doing a lot of customer work in Clojure and had a lot of interesting early experiences with this new language.

2. Can you give us a brief introduction into Clojure for people that don't know what it is?

It's a challenging problem, I mean people always want the elevator pitch. And Clojure has new ideas or combinations of old ideas about how to do things on so many fronts that you really can't narrow it down to an elevator pitch, so I've narrowed it down to four elevator pitches. Pitch number one - it's about a great language for programming the JVM. We have this assembly language for the JVM called Java and then there is a lot of other languages that you can use on the JVM that have specialties in various areas, all of which are more expressive and are a lot easier to write code in than Java. So Clojure does that. Second big piece is that Clojure is a Lisp and so if you are someone who is concerned about problems of meta programming, then eventually you are going to love Lisp and you are going to like languages like Clojure.

The third piece is functional programming, so Clojure is a dynamic functional programming language. As such I think it's a gentle introduction to functional programming in a business setting, certainly has been for me. And then the fourth piece is an approach to concurrency and state that is built around what's called the unified update model. So the notion that most of your program, say ninety five percent plus, is in a pure functional subset and then the non-functional, the imperative parts of the code, are atomic updates of values with well defined state and concurrency semantics and very easily separated out from the rest of the code and sort of kept sane.

3. As you mentioned Clojure is a Lisp, isn't this a barrier for developers? I mean people get scared from Lisp. Do you have any prior Lisp experience or functional programming experience?

It is a barrier, it's a challenge, and the fact that it's all of these things at once is a great strength but it also raises the challenge. If you are, in 2009, a Java developer and you are looking for something more expressive, there are things that you can do that will have a much shallower learning curve and give you real value over working in Java. And so we have done work in JRuby and we've looked at Groovy and Scala, and all of those are, I think, easier curves but less up side and so we wanted more up side and, you know, pay the price and slogged through it.

I had the good luck of starting my career in the eighties doing Emacs development.I've had not a lot of other kinds of Lisp experience over my career, I wish I could say I'd done a ton of Lisp back in the day, but have used Emacs throughout my career and so was comfortable with Lisp syntax, if not wildly enthusiastic. It turns out that that is not as big a problem as people think. One of the things that needs to be done is, a case needs to be made for structural editing modes in IDEs. So when I am doing Clojure I use paredit mode in Emacs and what paredit does is it lets you look at your code not as characters, words and lines, but in terms of the structure of the language.

So it says "I know this is an atom or this is a list of code" and I can manipulate it at that level with a bunch of keyboard commands. So at one level it's almost like a tiny toolkit on which you could build re-factorings, which is useful by itself. Another piece of that which is really important is that all of the closing parentheses are ignored by the primary editing plans in paredit mode. So the thing that freaks people out when they first see Lisp code, it's not so bad all the open parentheses, like "Oh there is an open parenthesis, there's another one, there's another one, there's some indentation. holy crap, there is twelve close parentheses". When you are a Lisp programmer you don't see those and when you are using a tool like paredit mode your editor doesn't see them either.

If I am at the end of the word and if I hit kill to end of line and there is a bunch of parentheses those are just ignored by the kill, and they just get pulled back in and it all works. So I think that structured editing modes, in fact structured editing modes in other languages would be really cool, they are just a little bit harder to implement. One of the benefits of Lisp's homoiconic syntax is that the code is data and so having an editor that is smart about the structure of code is pretty simple, because you are in a list or you are in a vector or you are in a map and that's about it.

If you are in a language with a lot more syntax it's like "Maybe I am in a method or maybe I am in a constructor, or maybe I am in package imports or maybe I am inside of a for, or maybe I am inside of a while loop" everything looks different. But there is no reason that a paredit mode couldn't be made to work with those languages if you had the whole AST available, sort of translate back and forth in real time. Paredit mode actually works with other languages in Emacs. I am using the Emacs starter kit these days, which is Phil Hagelberg's contribution and when the Emacs starter kit is installed it seems to, at least on my install, turn on paredit mode for JavaScript as well, which is another fairly simple language structurally Paredit mode does not work as gracefully there, and that could be just me, I mean I haven't gone and tracked this down, but there is no reason why it couldn't. And so it would be interesting to see that idea take off, and I think that other languages would benefit from it and Lisp would look less scary at the same time.

4. So what are the application domains of Clojure?

It's interesting. I think that the irony of functional programming for me is that functional programming is all about composition of pure functions to build applications rather than OO or mutable code. At the same time it turns out that functional languages are really awesome at doing the non-functional parts because those parts are structured and those parts are isolated from the rest of your code and you can clearly say "Over here I have ninety five percent of my code base, it's composed of pure functions", all these are incredibly easy to test and document and build, and write optimization and automatically cache results or whatever and then there is this very small part that does mutation of state.

By separating out those things it's much easier to deal with that small bit of state that remains and one of the things that falls out of that is that functional languages are really good at doing things that deal only with out of process interesting state. So something that talks to a database or something that answers web requests, or something that talks to a no SQL data store of some kind. There is no interesting state in the application anyway, I am doing a little bit of stuff and then I am asking a question of some other process, and waiting for an answer and then handing that off to someone.

There is no interesting state and there is no benefit to carrying around a bunch of noise, which is sort of incidental complexity of state in the application. So this is a very broad sweet spot but I think that the sweet spot for Clojure and really for other functional languages as well, is applications that have state. So by pulling out the part that is not state-full it actually makes it much easier to work with the part that is. So there, I have claimed the entire world almost all applications do that.

The flip side, the place where I think that Clojure is not a sweet spot today is anything where there is a solution almost done in another language. So if you sit down in front of a development project and say "Ninety seven percent of the intellectual property in this development project is already done in open source library x and I am just going to knit that together and put a bow on that and everyone wins", then...

An example of that would be simple CRUD web apps with Ruby on Rails; that's a sweet spot for Rails. aTe convention over configuration means that there are hundreds of decisions that are important, but not critical, to ship in some small app that are kind of already made for you and those conventions don't exist yet when a language is new. And so certainly for certain kinds of web apps, that's not the sweet spot for Clojure right now, and I think that is true anytime you are making a language switch that is also a paradigm switch, that your conventions haven't sort of evolved yet.

So when at our company we made the switch to Ruby in 2005 we saw the same trade off except from the other side, the conventions weren't there in Ruby yet for things that we were used to doing in Java. So that doesn't feel like a scary thing but it's a decision matrix and at the day job less than half of it is Clojure at this point, so it's definitely a matter of integrating it in over time where it makes sense.

5. And when you are willing to integrate your already existing Java code for example in Clojure, do you think you have to wrap it or can you interact with it directly? I mean this is a gap between two paradigms, right? So would you wrap it or is it as easy as to call it directly?

It depends on the kind of Java code that you are going to be using; some Java libraries are very Clojure friendly by nature like Joda-Time, because it is immutable time objects. It's time and date as it should have been done in Java to begin with, that kind of stuff is a no brainer.

Java libraries that don't have in memory conversational state, so some Java library that communicates with something else and so the only purpose is to munge arguments into some sort of communication format or medium, it's really easy to call that stuff from Clojure or any other modern JVM language and I wouldn't use Java for that. I would use Clojure or maybe JRuby to do that kind of thing. Where it gets more interesting is if you had an existing Java application that was built around the paradigm of mutable state, a significant application, and you want to add to that with Clojure – it's not a very good fit.

Certainly Clojure has some antibodies against mutable Java code, so when from Clojure you look at a Java collection it actually gets reified as a Clojure sequence and becomes immutable on the Clojure side so you are protected. If you are looking at a Java bean you can call bean on it, and once you call bean it's converted into an immutable Clojure map so again you've said "OK, I am going to adopt the Clojure way of thinking".

That's not going to be a sensible thing to do in an existing application that is built around a paradigm of mutable states. So the stuff we've built we haven't done that and I think that if you were introducing Clojure in that kind of environment you would start by building a small piece in Clojure that was in a different process, or was within some sort of boundary within the application, and said "OK inside here we are going to adopt the Clojure world view, and we are going to do this the functional way". This is not nearly as much benefit to the language if you are not going to accept the paradigm. If you are not going to accept the paradigm stick with something that is expressive and not as paradigm bending; you'll do Groovy or JRuby or something like that.

6. Is it concurrency that made you interested in Clojure or is it more than concurrency?

Actually I wasn't interested in the concurrency at all at the beginning, I was interested in the Lisp aspect of it. I mean we were very happy, we felt that compared to the industry average we were using very sharp tools, but it was kind of a quest on how can we make them sharper. And I am an enthusiast of Paul Graham and Steve Yegge and other folks that have evangelized Lisp to the rest of the developer community over the years. Paul Graham in his writings has laid out very clearly what he thinks Lisp's advantages are.

I've always been immediately meta in every programming environment that I ever worked in. When I play Magic: the Gathering I always play blue, so I 'm always the meta guy in anything so I've always felt drawn to Lisp and obviously there is not that many business opportunities over the past ten years to do that. So when Clojure came out, I don't even remember when I first heard about it but it's "Oh, there's this elegant, tasteful Lisp that runs on the JVM, oh I'll go use that". I was sold, I didn't need the rest.

So for me Clojure is over determined. If it had its Lisp features and its elegance of interop with the JVM I would have switched just for that. I wasn't looking to be swayed over into functional programming. I had successfully ignored the siren song of Haskell and other things for quite a long time; not successful anymore, right now because of Clojure I am looking at FP from other angles as well. And the concurrency piece is important right now because of multi-core, or people say that it's important right now because of multi core and I think there is truth to that, but really it's just as important in terms of thinking about state.

And Rich Hickey, the creator of Clojure, has done a terrific job in some recent conference presentations that are recorded and up on the InfoQ website. He's done a terrific job in laying out the case for the way Clojure thinks about updates and the way Clojure thinks about state as a really different approach to what most of the industry is doing right now.

And so I've certainly been won over to that. One of the interesting things is that the approach to state is remarkably easy to use. I mean not only does it work but it doesn't melt your brain to try to figure it out, and so we've had no trouble with people coming in who don't have a background in concurrency and saying "Oh, you know what? If you want to update something you have to put it in a transaction and if you don't it will throw an exception and you'll see it in your unit tests, it won't work". That is pretty easy, people are used to the notion of transactions scoping updates and so it's been remarkable how sort of a non story the concurrency part has been from developer cognitive load.

7. Rich on several occasions told us that we got the whole state concept, we got it wrong in enterprise and industry. Can you elaborate more on that?

So I don't think I can do it any better than he has done it and it's a very different story. I mean when Rich tells the story he starts quoting famous philosophers from history and talking about physics and it's intimidating. But in short the way that we think about state in the objects world is that we have objects, and objects are a combination of identity and behavior in state and they are all thrown together and from a getting started perspective there are some advantages to that. So for example in OO at a first approximation the "How do I solve this problem" question is really easy.

If someone walks up to you and you are an object oriented programmer and they say "I have a blah blah blah, what do I need?" the answer is "Oh, there is an object in a class to do that" and that is the answer for every question. The problem with that is that it doesn't match the way the real world works and in particular you'll see this when you start to do concurrency.

In the Java world we have this double checked locking is broken problem and the interesting thing about that is that when I first learnt about that issue it was about JVM esoterica. There is a bunch of people who are trying to take this shortcut with how they approach read operations on data and that shortcut doesn't work because the JVM has certain behaviors which are there for a good reason and so it's wrong headed to do that, but you look at that through the light of the way Rich has approached things in Clojure and the people that were trying to avoid locks for reads were onto something.

Because in the real world, operations don't stop for observation. And so I can be involved in doing something and you can observe it and that doesn't affect me and in the real world we don't get to say "We are going to stop this because somebody wants to observe it", and Clojure is built on that model because all of our data in the Clojure world is immutable, and you can look at it, and it goes from one immutable state via transformation that creates an another immutable connection and then there is an atomic moment where the reference switches over and points to that new state, and observers are not impacted.

And so you can layer into a system one or ten or a thousand observers and they are not going to have any impact on the operations that are going on. At the same time it is performant because all of Clojure's data structures - all* of Clojure's data structures - are trees and so it has the API of a map or a set but the implementation is a very quickly fanning tree so that this old version of the data and the new version of the data can share ninety five, ninety nine percent of their structure so it's very efficient for all the observers to be able to look at this snapshot at the moment the observers were looking and for us to be building the next snapshot that happens to actually use most of the same data at that time. And this is not incredibly new, in some way if you come from a Haskell background or somewhere like that, but it is certainly outside the mainstream of the way most software is being written today.

8. Several programming language implement functional programming in a slightly different way like there is the Lisp way, there is the OCaml way, and there is for example the Haskell way. What do you think of functional programming, I mean which way do you like to see it and what is an interesting software[stuff?]? Because we are hearing more and more about functional programming.

So a) I am not qualified to answer. There is a ton of interesting stuff in the functional languages that I am exploring because I got excited about it because of Clojure. I think that we don't know much; in terms of the amount of industry effort that has been put into it, it's minuscule compared to the effort that's gone into OO and so what we think we know now is going to evolve very quickly over the next several years. One of the things that is interesting in Rich's analysis of this is that the static versus dynamic typing debate is actually kind of superficial.

That the difference in thinking about mutable state and references to mutable state is a lot more fundamental than whether you have a static or dynamic type system. I think that's true, at the same time I think the value of type systems is still not a sorted or solved question. And hand waving and saying "Well some are good for some and some are good for others", isn't satisfactory to me and I don't have a good answer.

Clojure is dynamically typed and I think that that means, for me at least, it provides an easier learning curve than Haskell. But I don't think that that makes it the only way to work and I don't think there is any reason why you couldn't have a very statically type language that had strong overlap with Clojure in terms of the concurrency and functional features and I think that would be a really good thing. One of the things I would be interested to see is a language that had good type inference and a really strong type model and a friendlier learning curve and an easier sort of - you don't have to get the types entirely right. Because one of the things as an Agile software developer, type discovery maybe goes on throughout the process.

It's not that you sit down and figure out what the types are and then write code and it implements and the types match and everything works. It's that your discovery of the types is evolving over time and dynamic typing I think - well, let me put it this way: I know how to do that with a dynamically typed language. I know how to sit down with stakeholders and model things and very iteratively make those models converge on what they need. I think that is doable with static typing, but it's not something that I'm good at.

I'd like to learn more about that and I'd be interested to see that happen. At this point it feels orthogonal to me to the whole issue of functional programming and my mantra for 2010 is everybody needs to be learning about functional programming. pPck your school, it doesn't matter that much to me but start somewhere, and then we can hash up some of the other stuff when we get there.

9. So Clojure chooses to use reference types for dealing with states, dealing more correctly with state. There are other languages like Erlang and Scala copied it they use actors. Can you explain the difference between these two dealing with state?

First off I think we need both. I think that Clojure has solved, to the extent that we know how to right now, what in process semantics should be about state. And so the whole notion that we can have a thousand observers watching things happen without having to stop the world, that's really important and Clojure gets that piece right.

I think what Erlang does is really about something that works out of process. So I am going to do something and it's going to trigger something that happens somewhere else, on another process, and there is copying involved. So we could be copying state from here to there, and it's much more around messaging and that's really an out of process model for thinking - not you couldn't, you could do it in process, you could have an API that looks like the way Erlang thinks and everything happens in a process, but it feels more suitable out of process. I mean you are going to pay the overhead of copying things around.

And Clojure is really about let's get the in process case, that doesn't require copying, to work. There is a scenario there that is doable and it's not done very much in programming languages. If you use these immutable objects and have references, then you can have some amount of coordination locally without having to do stop the world and locking. And a lot of things are going to have to scale out across processes and across networks and those things are going to need a model like Erlang's.

I would expect to see languages that have both models supported coming on. It's an open question how much language support is needed for the Erlang model. The Clojure model clearly needs language support; that needs to be at a fairly language level "Right, here's how we'r going to do things". The actor model can be added with more or less pain even on top of a mutable language if you want to do it. It will also be interesting to see in the JVM world, do we really need an actor library per JVM language. Part of the idea of all this interop is that somebody should win. Congratulations Scala yours is the best and the Clojure guys are going to call it too, and so will the JRuby guys. So it will be interesting to see if that actually happens, and it will strengthen the case for all of the JVM languages to see that kind of interop happening. So that would be exciting.

10. So is it possible to use Clojure's model outside of Clojure, do you think that it could be possible to do that?

It is possible, there is a, and I don't remember the name, but there is a JRuby gem that actually gives you access to Clojure's lazy immutable collections and to references, and so you could program against those things from Ruby. I love Ruby and at the same time I don't find that very interesting. If you buy Clojure's model, unless you are just deathly afraid of Lisp, then just use Clojure. It's not that "I love Clojure's model but I need some other language X because Clojure isn't very expressive", Clojure is incredibly expressive.

So I don't feel a lot of compulsion to do that kind of interop and in fact on the places where we have done polyglot programming on our projects, where we've had multiple JVM languages in play, we've ended up doing that with small processes that communicate with each other by passing JSON around instead of by having it all together in one process. So yes possible, but I'm not sure it's a great direction.

11. So imagine like a Java programmer will start programming with Clojure and then there are no objects, and even multi method dispatching is something completely new for him, don't you think that there is a gap, a considerable gap, for today's Java programmer regarding the paradigm?

If a programmer knows only Java then yes, it's a pretty big gap. There are a lot of ideas in Clojure that there's no analog to in Java, so trying to find hand holds and learn the language is going to be challenging. The flip side is, how many Java programmers really only know Java? To me the definition of a junior programmer is 'knows only one language well'.

And so Clojure is not a junior programmer's language; it's not necessarily a place to go to at that point. But somebody who has been doing Java for ten years I'd be astonished if they didn't know two or three other languages to some degree of familiarity and then you start to be able to build on something. So maybe you've done some work in JavaScript or Python and you are accustomed to using high order functions even if you are not accustomed to using immutable data structures, so there is more traction there.

The other thing is that if you are accomplished with the more popular design patterns in Java then, even if you've never used multi methods, you can quickly see "Oh, there is a design pattern I don't have to write code for anymore". I don't have to do the visitor pattern anymore, I get it, look multi methods, problem solved; and for me, from my Java background, every Clojure feature was like a light bulb "Oh yes, I see what that solves". There is also an advantage to having significant history with mainstream industry languages and then go look at a language like Clojure because you have an appreciation.

If you just learned Clojure you might have a casual attitude to the features that were in it and say "Oh, this is a bunch of features that were thrown together and maybe another language with a bunch of other features thrown together would be equally interesting". I actually think that the features that are brought together in Clojure are very tastefully collected and solve a whole bunch of problems that I was having on a daily basis writing code in other languages, and so it does kind of go both ways. At the same time I was primarily a Java programmer until 2005 and I spent three years doing Ruby before I made the switch to Clojure and so that certainly argues for a progression up Paul Graham's love scale of languages rather than just starting at the top.

12. For people that are interested in Clojure can you give us, from your experience, some motivating examples that gives motivation for them to switch to Clojure from your own experience about clients, about where you applied it?

A couple of things: just having functions be first class citizens. and just getting rid of mutable state around iteration tends to reduce lines of code by almost an order of magnitude. And you can see this with any language that has some sort of simple approach to functions and blocks, compared to the traditional OO pattern-heavy kind of way things are encoded. And if you don't think reducing your code base by a factor of ten is beneficial then you are probably not going to agree with me about almost anything else either. But most people are like "Oh, that's really interesting".

But where it really becomes compelling is where you find the occasional re-factoring or change of approach that operates at a higher level but has that kind of effect on changing the code base where - we were working on a project where we had to implement a protocol for data, and so there is a message that has a certain structure and you have to receive the message, you have to unpack the message, you have to apply some business rules,based upon the message and some stateful stuff, do some stateful things based on how this message interacts with past messages, and then you have to assemble a message in response. And a naïve approach to that, you solve this problem for one message end to end. Great.

Now you have a second message, now you do a third message, now we do a fourth message, and so forth, as you start to do that you realize that you are doing a lot of incredibly repetetive coding. Inheritance and the kinds of tools for re-use that are in common play would solve some of that, but by no means all. And what we were able to do with Clojure is basically say, "You know what? We are going to define with a macro what looks like a new language feature – def message". So you define def message, it lays out the structure of the message in data and then there is an interpreter that can read the structure created by def message and it sort of generates all the rest of the code for doing all the message processing, and then you have that very thin layer in the middle of doing the business logic, against now what has actually been converted into Clojure maps; just data.

And all of the messaging stuff is solved, with a set of def messages. Having pulled things out in that way then those def message objects could also be used to drive documentation. So the def messages almost look like they were copied out of the spec book. And in fact we tried to do that, we tried to literally take from a pdf document, here is the actual structure of a message, boom, we're going to make a def message. So macros were a big help in that piece and so thank you Lisp. The other piece that was really helpful was that with Clojure meta data we were able to decorate the various things that we created when we encountered a def message so that other tools could operate on them.

So you could ask questions like "Go and find me all of the things of kind X in the system via meta data and then apply some additional service to that". And Clojure meta data is just Clojure data. So it's extremely easy to work with meta data and add meta data. So that was an example that felt really nice. You could have done the things we did in a non Lisp but it wouldn't have been as easy and the result would not have felt as much as if it was part of the language, it really was an internal, it's not really a DSL, it felt more like adding language features, which is the Lisp thing where you use macros to add language features that attack things in your domain.

A really interesting piece of that is that the way that system works really was a very dumb interpreter that reads through the data and does all the data munging to convert between the formats that were needed on the wire and the formats that we wanted to work with. If that thing is too slow, we re-write the interpreter as a compiler that actually compiles Lisp code that does that and get an order of magnitude speed up and we don't have to touch anything else in the system. If we hadn't pulled out that notion and had this kind of data driven approach to that, that kind of optimization would have been impossible.

And I have seen plenty of systems where that copy and paste, "Oh, we'ren message seven hundred now, and we have done this repeated things seven hundred times. Oh, we need to make it faster." Well, there is seven hundred ad-hoc, bespoke places that you have to go to make that work.

13. Like today as programming language offerings we get some kind of pure languages like Clojure which respects some kind of paradigm. And there are languages that mix things like Scala. So you have Erlang, you have Clojure on the pure side, pure paradigm side, you have Scala that mixes things, you have C# that mixes things. So what do you think? Do you think a language that mixes features is a good thing or do you think there are tradeoffs and it will spoil the feature?

That is a terrific question because I don't think it yields a simple answer. Ruby feels like a lot of different kinds of ways to do things language, and I love Ruby. Clojure has a very strong perspective on how to do things and I like that as well. They both have their purpose. I think one important piece is tasteful selection of abstractions to solve real problems, and so languages that I like are languages that the designer of the language had a certain aesthetic that they applied when they were solving problems and you could feel that through the language.

I like languages that have benevolent dictators, not languages that are by committee, by and large, there are exceptions - but benevolent dictators can have a world view and within that world view there can be incredible variety. Like I said Ruby feels like a very multi paradigm, you can try different kinds of approaches, whereas Clojure is a much more "Here is a way of viewing the world". Although even that is very challenging because if our notions of paradigms are all screwed up then we don't have even intellectual traction to be saying that.

If Clojure is as important as I sometimes feel that it is, we may look back in five years now and say that Clojure really is a multi paradigm language of five paradigms we don't even know about yet, that are all within the "Doing things right" rubric of functional programming and a model for time that's built around a unified update model with software transactional memory. Maybe that is an urgently right way of doing things but within that there are half dozen different paradigms that we haven't discovered because we are all over here in object land.

I think that different JVM languages, the ones that people are talking about, the ones that have been talked about here at the conference today, all serve different needs. Scala has a ton of different things in it, from an aesthetic perspective that feels a little bit non ideal to me, never a feature unwanted. But the flip side to that is the beginner learning curve for Scala for a Java programmer is pretty simple, it's pretty Java like and then you can go off in this direction and do this or you can go off in that direction and do that. I think that Scala is in some ways like C++ was, that you can start from C and not be very different and then you can gradually do something that is a lot different. And that analogy argues for great success for Scala in the sense that C++ certainly had a lot of traction building on C.

14. It could be the end, as it was for the C++. Isn't it a problem also to have all of this stuff?

Well it's a challenge. I think Scala is a good and useful language but I have chosen Clojure, I like the perspective. I think that syntax is superficial and so I don't consider syntactic similarity to be an urgent reason to move forward with languages, but other people clearly don't feel that way. People look at, say, Groovy versus Ruby and say that Groovy's similarity to Java in syntax is the most important thing there.

I think there are far more important distinctions going on between Groovy and JRuby than the syntax and in fact I think that I would still feel that even if there syntaxes were reversed.; even if JRuby had more of Groovy's syntax and vice versa but that's not really the most important thing. But you know one of the nice things is that the JVM has created a platform on which we can have these debates, that the interop capabilities that are baked into the virtual machine allow us to have all these languages that can really talk to each other and while I think it would be gratuitous to have four or five of them compiled and running together in the same process, most of the time, at least where we are today, I think the possibility of writing code and deploying it in these different languages and not having to be relying on one is really exciting.

And I think that even if certain individual initiatives or efforts fail, that the trying a bunch of different things all at once in terms of - across the entire industry - is going to lead to good outcomes for all of us because we are going to find good pieces. I also think there is going to be healthy competition. It is my understanding that Scala's collection classes are picking up some of the good ideas from Rich's collections in Clojure, and I think we will continue to see that sort of thing happen over time. In particular I think that Clojure's strategy of using trees and having immutable collections is such a good one that there is no reason not to have those available in any JVM language and I think we'll see that happen.

15. So you moved from Ruby to Clojure. What is the next language do you think?

Well certainly the other languages that I am going to look at in my copious free time, right now I am interested in Haskell and Erlang, for obvious reasons around functional programming and concurrency and because they strike me as the most different and the most likely to take me out of my comfort zone and keep me learning. Looking at my career trajectory I would say that I don't know what my next language will be yet because it probably doesn't exist yet. I usually stay happy with one for three or four years and then go on to the next thing, so I can't wait to find out what it is.

16. So we see languages like Groovy which are built by mostly pragmatic choices and we see other languages like Clojure which is build around theoretical ideas and research papers. So where do we stand? I mean sometimes theory gets too far from practice and sometimes pragmatism brings not very beautiful results, so what do you think?

I am much more on the pragmatic side and I think probably I would rephrase the way you characterize Clojure in the question. I think Clojure is extremely pragmatic and informed by academic research. I mean Rich is a practicing software developer and before that a musician and he is not an ivory tower theory geek, although his theoretical chops from where I am sitting seem quite strong, and the things that he's done in the language seem to impress the ivory tower crowd reasonably well, but I think that his language choices have been motivated by work on real world systems and so I come down almost wholly on the pragmatic side.

I think it's great to have people like Rich who are practicing pragmatic practitioners who also somehow found the time to read seven hundred academic papers and say "Here is a practical problem I am having and here is the academic paper that would solve it" and pull those things together, I think that is really incredible and the JVM is a great environment for ideas like that to bloom,.Who knows what will happen next month. Maybe there will be another JVM language a year from now that blows our doors off the way Clojure is doing right now.

17. We heard a few days ago that closures, in the sense of the function that close values, will be added to Java. Do you think that it's a good thing? Some people want Java to die, some people want it to live more. Some people want it to die so people move to other languages. What do you think of that?

It doesn't matter very much to me. I want the VM to continue to get love and I want the VM to get faster and I want it to get better garbage collection and I want tail recursion to be easier to do and I think that all the different language communities want those things. Those things are kind of equally beneficial, maybe somewhat disproportionally beneficial to functional languages, but beneficial to all the JVM languages and so I want to see all that happen. And so for me asking what I want to see happen to the Java language, the Java language is irrelevant. It doesn't make any difference to the industry anymore, except in so far as a bunch of people don't believe that yet and so if something happening to the language freaks out a bunch of people, then I would care about that.

If Sun or Oracle did something to the language that terrified people and caused people to flee the VM then that's bad for Scala and Clojure and Ruby and all of these great languages. It's not a technical question, it's a mass psychology question. Don't do something that scares people is my advice on the language level. If freezing the language makes people happy then freeze the language, if adding features slowly makes people happy than add features slowly. I can't imagine features ever being added to Java that would bring back people that have gone on to other languages because we are not talking about a short distance.

For a while I thought it would be interesting to just say "Announcement: Groovy is Java 7" and Scala has picked up enough traction since I first glibly said that that might make sense too, although I think that the differences there are more significant. But I don't think the language matters at all, except its psychological effect, at this point.

18. Groovy has been implemented also for .NET. Do you have any opinion about what's going on on the .NET side, regarding programming languages?

I do not spend very much time on the .NET side of the world. I am a big open source supporter and I think that Microsoft, although they are slowly getting better at that, they are in their decade in the wilderness in terms of technology leadership. Regardless of what the bank accounts and spreadsheets say at the end of the day.

I know that there is Python and Ruby working in .NET and the Clojure port to .NET is tracking not very far behind Clojure on the JVM in real time, but I've never even fired it up because I would have to get a Windows box to do that. I think it's useful. I think that for languages like Clojure that want to have a small core and be able to host on different VMs, that the exercise of doing it on the .NET VM is useful, even if I don't use it on .NET myself. I think it's interesting and it's cool but it has not had - we build mostly server side and some mobile so what's happening in the .NET world just doesn't affect us very much.

19. So Clojure is portable from the JVM to .Net, but if you don't use any of the libraries of the JVM or the CLR, what do you think about that?

There is a remarkable amount of things that happen in Clojure that are not very library oriented. The whole sequence abstraction, the underlying concurrency abstractions, that stuff is all going to work and then you are going to do IO and other things and you'll hit a brick wall.

That is going to be a challenge and so Clojure itself - there is Clojure and then there is a separate repository also managed by Rich and under the same open source library called Clojure-contrib, and Clojure-contrib is all manner of third party stuff, some of which is incubator for eventually going to Clojure, but most of which is just useful utilities and add-ons and goujon and sA lot of those things depend on really being on a specific platform, so for example I have written a contrib called clojure.contrib.jmx which maps down to the management extensions to Java. That just won't work on .net and presumably there is some or multiple equivalents to that in the .net world and so if the .net side of things takes off we are going to have a lot of porting and cleaning to do on contrib and it's a problem that sorts itself.

Either there is a lot of interest in Clojure, in which case there is manpower to go and solve that problem, or there is not and we don't solve it and we've certainly done the Agile thing so far which we've completely ignored it. Actually when I first proposed contrib somebody bounced back on the mailing list and said "We have to decide how we are going to mark the things in contrib that depend on the JVM so that we can anticipate this day when this problem is going to happen".

Every once in a while I think I demonstrate wisdom and I said "That's exactly what I am not going to do, I am not going to solve that problem. I promise that as the guy who made this problem, that if you make that into a real problem I will clean it up when it happens but I want to ship the JMX support, which I am hours away from doing, not have a week long debate with fifteen other people about how we can deal with .net when, if, maybe it happens".

And so I think we are doing things roughly in the right order, although I think there is a little bit of a danger that Clojure can take off so fast that contrib could get a little bit out of control; that it could need more gardening and more specific ownership and stewardship than it has right now. I mean there is six, eight committers and thirty contributors, or something like that, to contrib right now but there is not a super strong unified style. And the nice thing is that Clojure is opinionated enough, I mean we get enough style questions answered but then there are some style questions that are not uniformly enforced and this could be little things, coming from Ruby I am accustomed to an optional argument at the end of methods which is a map.

And Clojure takes the more Lisp-y approach of un-rolling that map into a keyword and then a value and then a keyword and then a value at the end of the method. If half the methods that take optional arguments did that in one way and half did them the other way, or something like that - that's not a problem but I can imagine running into other problems like that if it grows too fast without oversight. Right now things are good but it's something to keep an eye on.

20. Stuart, what are your three favorite IT related books?

Three favorite IT related books: Neal Stephenson's "Cryptonomicon", which is not exactly an IT book and it's fiction, but I always have to put in a plug for that. I am a big fan of Peter Norvig's Lisp book which I am working through right now. One of the interesting things is that most Lisp books are really good. And so there are a lot of them out there but Norvig took a functional style with Lisp - not all Lisp was done that way. There is a lot of common Lisp code that's not functional, but his style it's paradigms of artificial intelligence programming, PAIP, takes a very functional style with Lisp and it's a beautiful written book. And another favourite IT book.

In my house I have hundreds of thousands of IT related books. And my shelving system is passionate or un-passionate. And then the un-passionate are all thrown in closets and then I have one tiny shelf that's got two rows of books on it that I feel these are really important books that I want to come back to again and again, so I am just thinking about what other books that are on that list have been on my mind. I think the other one that I will put out there is "Godel, Escher, Bach" and the thing I like about it is, I mean the first off it's a fantastic book, but secondly Hofstadter does something which more authors should do, and he has like a twenty page annotated bibliography at the end of Godel, Escher, Bach and so you have this experience when you read a great book, you read a great book and then you say "Wow I really wish I can have lunch with Doug Hofstadter or follow him around for a day and find out what he thinks about a trillion other things".

And then you get to the end of the book and it's like "Now I am going to give you a twenty page essay which is a guided tour of the favorite things in my library that I really am passionate about" and his annotated bibliography actually has in it a paragraph about why he likes each book and then a star beside the really special ones and then two stars beside the really, really special ones. One thing I've always wanted to do is read even 10% of the books that he has referred to at the end of Godel, Escher, BachOne other thing book related, Rich Hickey has a Clojure book list on Amazon.com and I like the fact that it has thirty six or thirty eight books in it and it's like "If you want to learn Clojure you might want to read these". And in your face to that whole "in 21 days for dummy" attitude to software development. Here is a language that you probably need to read ten or eleven books before you are going to be really good at it, and I like that challenge.