Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Interviews Dick Wall on Scala, DI with SubCut, Scala Compiler Plugins, Java's Future

Dick Wall on Scala, DI with SubCut, Scala Compiler Plugins, Java's Future


1. We’re here at Scala Days 2012 in London; I’m sitting here with Dick Wall. Dick, who are you?

Hi, I’m Dick Wall. I am a long, long career Java developer. I’ve been working in Java since ’94-’95 and then in the last three years, I’ve kind of switched over to doing everything in Scala, discovered Scala and I were soul mates back in that time.

So in addition to what I do for work, I created and I co-host, the Java Posse podcast. I founded and led for three years the Bay Area Scala Enthusiasts which I think was one of the first Scala user groups in the US. I don’t know if we were the first; but we certainly were very early.

And just recently I found somebody to take that over which is quite nice after three years of running that. And I’m just about to go to Scala consulting full-time; so it’s an exciting time in the Scala realm right now.


2. So you run the Java Posse Podcast?

I’m a co-host. We’re partners in the Podcast but...


3. But it seems you have defected to the Scala world, why is that?

I, much like a lot of people, I think, about three, four years ago, I’ve started to despair that Java had become a bit stuck in a rut, there weren’t timely updates anymore. I was really starting to miss some of the features that I saw in other languages, function literals and closures being high on the list and I was also just fed up with JavaBeans getters/setters and all that good stuff.

So I started looking around for alternatives and I messed with Groovy a little bit and I’m sure there will be people jumping up and down going "Don't say anything negative about Groovy" and I won't because I liked it but I’ve always liked statically typed languages; and so once Scala came along, it looked like the right combination of features and familiarity for what I wanted; and I started learning it, wrote some dreadful code for the first few months of using it, trying to basically write Java in Scala which I highly recommend, by the way, as a strategy for picking up and learning Scala to start with.

And then over time, school of hard knocks, I gradually got better at style and learned functional style and that sort of thing but I still keep doing the Java Posse podcast.

One of the ways we look at it is Java is more than the language, we’ve always said on the Podcast that there’s really three aspects to Java: there’s the language, more interestingly there’s the virtual machine, the JVM; the primary thing that Java has that I think all of the other developers ecosystems wish they had is the community that surrounds Java, that community is starting to diversify into other languages but they are all still mainly hovering around the virtual machine and the community still hangs together.

So you get people going to Clojure and to Groovy and to Scala and to JRuby and yet they all come back and talk with one another; there’s a lot of cooperation between the groups, you can see that at things like the JVM language summit which takes place every year. Bryan Goetz and some other people organize that just for that reason and so I think there’s a lot of life left in the Java Posse whether or not we actually talk about the Java program ming language.

I should also point out that there’s two other people still in the Java Posse, Joe Nuxoll recently decided that it was not him anymore because he’s gone on to do design work more than anything but both Carl and Tor did not share my obsession with Scala at least not yet; and they both work in Java although technically speaking, Tor works in Android which is not really Java, at least that’s being sorted out in the courts right now.


4. That's an interesting position there, that you mentioned Android, you mentioned that there are many different languages that compile the JVM; but in a way, there seems to be a rift with Android which isn’t really a JVM, it's not a JVM at all in case Google is listening. So if you program for Android, you program in Java the language, what do you think of this rift?

I don’t think it’s any secret that the way Dalvik works is to take class files and send them through another compilation stage into .dex format which is the Dalvik executable, it gains a lot of advantages for mobile devices doing that but at the end of the day, anything that you can compile to a class file can still be compiled into Dalvik, there are few thing that are missing, reflection is hard on this deliberately so because it’s a big security hole having reflection on that, but for the most part, you can take Scala or Mirah, and there’s a lot of people working on Mirah which is the statistically typed JRuby derivative; JRuby perhaps is the correct word [JRuby light], and that’s a lot of work going into making that work on Android.

People got Scala running on that although it brings some fairly heavy runtime considerations with it; and so it’s a valid target and it still feels very much part of the Java ecosystem whether or not you call it Java; it’s pretty clear that there’s a lot of alignment with the Java community and the Java programming languages.


5. It seems talking about Google, it seems Google is trying to replace Java with another four-letter language, Google Dart, obviously on the web in the browser maybe also on the server with serverside Dart and also might replace Java as an input language for GWT, for instance; so what do you think of Dart?

I got to admit I haven’t really looked at Dart in-depth yet; actually Scala is at the top of my list of languages to master first; so it’s on the list and I’ll get to it at some point.

I think from what I’ve so far, the Dart play feels likes it’s more aimed at Javascript as a kind of Javascript alternative long term than it is taking over from Java; and Google has some other language like Go is a very interesting one that they got; they also had, (I forget the name of it) but they were talking about one that had some kind of dependency injection at the very core of the language. I saw that at one of the JVM language summits, I’ll probably remember the name of it later; but this isn’ t the first time they’ve gone into the alternative languages game.

I think just like I would expect or, I would just like to freely admit and say about Scala, I don’t think anyone’s is going to unseat Java as a language anytime soon; it’s got way too much momentum. We’ll probably be around forever like Fortran and C and I don’t expect anything or nor necessarily do I want anything to come in and replace it; what I want is a healthy alternative language ecosystem and I think we’re really close to getting that now.


7. Obviously you’re now not using the Java language anymore; you’re using the JVM and the whole ecosystem, the library, so the two aspects with Scala, you get the stable JVM, GCs and the libraries.

Dick Wall: Yes, and I use all of those; I like the alternative library implementations but I also still use Java libraries in this; so I’m not sure if I’m really answering your question.

Werner Schuster: Are you writing Java?

Dick Wall: No, I’m not writing Java at all anymore.


8. Do you see people writing in Java the language?

Yes, I still think people will be writing in Java the language in five, ten, twenty, years’ time and it will probably still be a pretty major force; but I also think there will be a lot of alternatives and it doesn’t suit every purpose; so I do think we’ll see more people start migrating to other languages that suit their problem domains better; and I think that’s healthy.

It’s the old story, when all you got is a hammer, everything starts to look like a nail. We’ve been there for a long time; Java and object-oriented programming have suited us well but there’s other alternatives now and they can make us more effective in a lot of different domains.


9. So talking about Scala, you have a Scala project out called SubCut, what’s SubCut?

SubCut is, let’s start from first principles, I am a software engineer that works on large enterprise server systems, usually, big data and over the years, I hesitate to even call it a pattern because it really is a very simple concept but the idea of wiring up configuration so that components can be reconnected and reconfigured easily has been kind of ongoing project, kind of a problem domain that is attacked from all sorts of different ways.

So in the early days, you had JNDI, the Java native directory and interface, whatever it was the directory look-up mechanism, now it wasn’t Java native, was it? Java naming and directory interface, JNDI, and that was one way that worked for awhile but it was heavyweight and nobody liked that.

And then as time went on, Spring and others became more popular and those are dependency injection; so the idea is that you tell an object when it’s created, what its connections are to the rest of the world or to the rest of the system that it’s working in; and those frameworks have been refined over time; Guice was a big jump forward because it used annotations and made the whole thing a lot easier than a lot of XML configuration and I think what I'm trying to do with SubCut, actually the way SubCut started out was I had been using the Cake pattern, and I had used that successfully on a number of projects; but I found it started to become a little heavyweight in particular when trying to do a very in-depth functional testing in a number of ways where I just wanted to throw a couple of mocks into database somewhere down in the hierarchy and have everything else work the same so I could fire up a whole bunch of functional test of these things.

So I actually started out with the mind to kind of rewriting Guice in Scala and that was probably what the first SubCut looked like but I was doing, I was thinking, "Gosh, there’s easier ways to do a lot of this stuff in Scala," and it ended up that I just kept removing more and more stuff from SubCut and what I was left with at the end of the day is effectively a domain specific language for doing the binding which is an idea that I took from Guice, an idea that Guice does very well; we can do it a little bit nicer in Scala because we have type manifests and type parameters that just make things look a little bit neater in the language that you use.

And it has a binding module which is the same kind of idea as the Guice modules, and the other part of it is a compiler plug-in which makes a couple of thing a bit nicer; and at the end of the day, those are the three things that you really get out of it.

You mix a trait in that says, "This object's injectable, it carries down configuration actually through implicit parameters which is something the compiler does and the result is an incredibly lightweight mechanism that you almost don’t know you’re using: it’s a tiny little bit of magic it‘s apparent in the compiler plug-in but enables the implicit parameters to be passed down invisibly and other than that, it’s just a very lightweight DSL and binding mechanism.

So I think it’s around about 400-450 lines of code it’s all now and half of that is the compiler plug-in which was actually pretty complicated to write not much documentation on that; so it took me a little while to sort of reverse engineer it and figure it out.


10. So your experience with the compiler plug-in, for people who might not be familiar with them, what do you work with? Do you get the raw AST or what do you get?

You get a life cycle, you get the ability to plug into the compiler life cycle at anyone of the gaps between, I think, the 27 different phases that the Scala compiler has.

SubCut uses a pretty early phase right after the parser has constructed the abstract syntax tree. And what you get handed then is the abstract syntax tree and you get to pattern match and manipulate it however you want to, presumably creating a valid abstract syntax tree as a result because otherwise it won't compile through the rest of the system.

So the mechanisms are actually familiar to anyone who’s done any pattern matching in Scala because it’s all case classes and pattern matching to find the expressions that you’re looking for; and then actually constructing it is all done through case classes; so you then construct new abstract subtrees, you know, syntax subtrees and you kind of plug them into the existing tree and once that’s all in place, then you say, "Okay, hands off now; the compiler can go and do the rest of its work." and literally in the case of SubCut, it’s very simple, it looks for a trait called AutoInjectable and when it sees that it adds an implicit parameter for the binding module and a val holder field in the class that holds that value and then that’s it effectively, the step says "OK, I’m done enhancing this class now".

So what the upshot is is you can go through and add AutoInjectable to any class and it makes it dependency injectable at that point in time.


11. So how do, in your experience, how do compiler plug-ins interact with tooling, because tool designers usually don't like people fiddling with [the language]?

Right, and it could be. ScalaDays is a brilliant conference to come to just because of the people that are here and the conversations you get to have and I actually had this conversation with a couple of the IntelliJ team who I went to them and, they said, "You know, there’s a couple of thing that I’d like to see that would help out SubCut," and they weren’t what they’re expecting to hear, so they expected that there would be compiler errors when you start using SubCut; but actually that’s not the case.

The AutoInjectable trait already has the methods that you call on it; it’s simply missing an implicit which the compiler wants there when you actually do the compilation phase; but IDEA, at least, unless you turn on the auto typer that they’ve got, it doesn’t really care about that when you’re writing it; so the result is essentially a pretty smooth experience using SubCut in IDEA. You don’t see errors when you’re using it. The methods that you’ll expect to see are all there and then when you actually hit compile, the compiler plug-in kicks in anyway and it makes the code valid before it reaches any further.

I think if you turn on the auto typer which is something we don’t do a lot in Scala usage at my company yet because it isn’t quite ready in a number of other areas; it has differences of opinion with the Scala compiler.

So generally we leave it turned off, I think if we have that turned down, we might see more difficulty and maybe the IDE would need to be enhanced to take into account that; and I don’t think it would be too hard to do that although I have a layman’s knowledge of this.


12. I guess it'd need to support compiler macros?

Dick Wall: Yes.

Werner Schuster: Basically run the same code as the compiler.

Dick Wall: Yes, it would need to insert the compiler plug-in in the right phase and let it manipulate the syntax tree in the same way and then carry on, let the IDE compiler get its hands on the rest of the code after that.

But actually it works really smoothly, the things I was asking for turned out to be related to symbols and symbol usage which is something we use in SubCut a little bit for convenience and it would be nice to have simple code completion on those so when you start using a symbol, it could actually help you complete it because those are the names of the bindings that you’re using and you have to kind of remember them right now; there’s no help in auto completing those, some kind of simple autocompletion would be really nice in there.

So I think I got my case heard and may go see that in the future.


13. So we’ll have to thank you for that.

Yeah, SubCut won’t be the only library that benefits from that, symbols are somewhat well used in Scala and having just some simple autocompletion around symbols would be really nice and any IDE, I think it would benefit a whole bunch of projects.


14. Talking about tools, so you have a talk here about Scala and engineering.

Actually it’s "An engineer's guide to the Scala community" and it’s kind of an interesting meld of the community side of things and the technical side of things that I work as an engineer as my day job and community building or community interfacing, I guess, is a kind of a hobby, a sideline that comes off the Java Posse and just coming to events like this.

And I think that unfortunately, many of the engineers that I see at these events don’t realize that there's a lot of similarities between the problem solving that you do on a day-to-day basis with a programming language and the kind of problems that you see working with the community and actually applying some kind of rigorous engineering decision making can really help with the way you deal with people and the way you reach out and work with the community and it can be surprisingly effective when you do so.


15. But people don’t conform to Type Classes, do they?

They don’t but they do or they can behave in fairly predictable ways and if you know what some of those likely outcomes are, it can make you a more effective communicator and that can only help your engineering technical side, so tell somebody that they’re doing it wrong, they'll probably immediately put guards up, it’s human nature but demonstrate or offer to work with somebody, pair program with them, they'll start picking up ideas much, much quicker; these are effective ways to improve just about every aspect of your work and that’s sadly overlooked a lot of the time. So people skills matter it’s, I guess, what it comes down to.


16. Giving a plug to the sponsor of the conference, [SkillsMatter]

Oh yeah, skills do matter but people skills matter as well. I haven’t thought about that, yes.


17. So talking about community and Scala and your work in the enterprise, so there’s been criticism of Scala about breaking compatibility and things like that, have you seen these problems, what’s your take on this?

I’ve experienced compatibility breakages and problems related to that quite a lot because I work with Scala day-in, day-out and certainly the move from 2.8 to 2.9 was a rocky one, we used a lot of third party libraries. It was made worse by, I won’t mention any names, but there was reliance on libraries from a particular company that does a lot of libraries, very high quality ones but unfortunately they didn’t move to 2.9 for a very long time and we had to stop weaning ourselves off of those libraries and finding alternatives because we wanted to move to 2.9 and the binary compatibility was hurting us there.

I don’t really say that there is an easy solution to it like, I think, any of the engineers in the Scala community, I understand the problem which are mostly related to traits and the way the JVM doesn’t really handle traits, so you have to work around limitations and I would much much rather have traits than have them removed in order to get binary compatibility back; there’s no question that it's a positive decision to keep them.

It does mean that there’s some extra work and some extra difficulties when you go between versions but you save so much the rest of time, it’s an easy trade-off to make.

And if you got the source for the libraries most of the time, you can just recompile unless there’s some kind of other breaking change and the more libraries you use, the more likely you are to see something like that.

But I also believe that a binary migration tool that has been talked about for sometime that maybe coming sooner or later and I think that will alleviate just about the remainder of the troubles, that should be good.


18. So the Scala language can help offset the problems with these libraries?

The idea, I believe, with the binary tool is to take .jar files and rewrite them, swizzle them internally so that they conform to the new binary interface that’s required; so without needing source code you can take any of the dependencies that you’ve got and make them work with the new version of Scala and if that works, that would be terrific, that would be the removal of one of the last bugbears of changing versions.


19. So talking about changing versions, what’s your favorite feature that you want in Scala, if you had one feature to choose?

I think actually I’d like to say and it maybe possible to do some of this now. I need to think about it with the addition of the dynamic trait and Scala 2.10, but I’ve always thought Groovy did well with the builders feature; I like their way of building up XML documents or any other kind of structured document through the builder objects which are basically dynamic objects that you can call methods on that don’t exist and those methods when you call them cause, you know, side-effecting behavior on output or something like that. So you actually build up a XML document hierarchy or something like that.

And I’d like to see that because I’d like to have a consistent way of building up either XML or JSON or other data representations in Scala. Right now, the XML support's very good, JSON support is good depending on the library you’re using but different and there’s not a lot of consistency in this. I think that’s probably a feature I’d like to see. Another one, how many am I allowed?


20. I’ll give you another one if you answer one more question. So what are dynamic traits, are they like method_missing in Ruby?

Actually that’s a great definition; so a dynamic trait when used in Scala allows the compiler to know that when a method is called on that dynamic trait that doesn’t exist; it actually rewrites it to a call, to a general method handler with the name of the method that you called and any of the arguments that you gave and that lets you create completely dynamic objects so you could theoretically produce a façade for a database table that the compiler lets you access properties fields in the database table and just goes out and reads those fields and brings them back in, something like that.

You know, something like that could theoretically, I think be used for the kind of Groovy builder pattern but I don’t know yet whether it would be a complete replacement on it, I haven’t tried.


21. So what is the second biggest [Scala] feature?

The second one I’d like to see, it's actually a small feature that I suspect people will find, maybe not everybody would like to see this in there. I’d like to see the ability to have an initially mutable object that allows you to change values for set up then go into a frozen state where it no longer allows any kind of changes, so just to be able to have the convenience of building something up without needing to take lots of immutable copies while you’re in the kind of initialization or construction phase and then tell it, "Okay, from here on out, this is immutable, it really behaves as a completely immutable class, you can’t ever change anything in there again, and that would be a nice feature too.


22. So the Clojure space has something like this with transients.

Dick Wall: Yes.

Werner Schuster: So Scala people should look at that, maybe they have.

Dick Wall: I’m not a language designer; I know language designers and I would leave that in their capable hands but it is something I bumped up against a couple of times that I would love to be able to see. I’m very much a pragmatic user of Scala. I like what I see in there and I have written a compiler plug-in; so I have at least got my hands dirty one time in the compiler; but yeah, I think there’s a lot of considerations when adding a feature like that and I definitely want somebody who really have their head around the whole design of the language just coming in there willy-nilly is probably a bad idea with something like that.


23. To wrap up our special ScalaDays question, please describe yourself as a Monad.

I’ve been thinking about that since you gave me the hint that I might have to answer that question, I’m not sure I can describe myself as a Monad but I’ve been trying to fit myself into my favorite one - one I use all the time which is the Either class, it gives you either a left or a right and I used that one so much that I would love to find a way, a clever way of describing myself in those terms but I haven’t really managed it yet. So either I’m here or I’m elsewhere something like that but that’s definitely my favorite.

Jun 29, 2012