I am Sadek Drobi and I am here at QCon with David Pollak. David has a long experience in software and he did his own framework, Lift framework. So, Dave, why don't you tell us about yourself and what you have been busy with lately?

Ok, thanks for having me, thanks for inviting me for a chat with you. I have been working on the Lift web framework for the last two years, we released version 1.0 a few weeks back. Lift is a web framework that is built on top of the Scala programming language and Lift makes building real time web applications super simple. So you can build games, you can build chat applications; you can build messaging applications as well as your standard database crud applications with Lift. Lift gives you code economies of Rails, it gives you the performance of Java web frameworks. And it gives you type safety that is beyond other one.

So as you tell people after a long experience with Java and they discover Rails and Ruby and they were astonished with all the productivity and features it offers, and they start attributing this to Ruby as a dynamic language, then you came up with Lift, which is based on Scala which is a statically typed language, yet is as dynamic as Ruby on Rails. What's your take on that?

I don't think that having concise code and having a dynamic language mean the same thing. There are a number of values of Ruby that you also see in Scala that you also see in other languages. One of which is you can pass code blocks around. So in Ruby it's pretty easy to pass lambdas, it's pretty easy to create chunks of code that are passed parameters to other methods. And when you are starting to pass chunks of code as parameters to other methods, you can barely control structures in those methods. So your code is not the boiler plate of control structures. So you are not writing FOR loops over and over again. You are saying: "Here is the business logic, apply this business logic to this collection".

And if you come and sit down and you look at a lot of Java programs, a lot of Java programs are the boiler plate of controlled structures with very little business logic. It's probably a four or five to one ratio between business logic and boiler plate control structures. So Ruby gives you that conciseness. One of the other things, other annoyances at least for me about Java is the necessity of repeating what type of thing is. So if I say String foo equals "bar"; why did I have to tell the compiler that this thing is a string? Compilers can figure that out. The Scala compiler figures that out.

In Ruby it doesn't really matter because you assign a string to the unitype variable, and as long as it responds to the methods you are sending to it, it works. What Scala does, or at least the way I perceive it is Scala gives you the ability to embed certain tests right in your code. So the tests are encapsulated by the type system, but the type system doesn't get in your way, the type system basically is annotations in your program, for tests that you expect the compiler to run against your code. So if you take a look to where you have to define types in Scala, you have to define types as input of any parameters that you take in a method, you have to explicitly define the parameters of those.

Or quite frankly when I am writing Ruby code, when I am being nice, or document "This is supposed to be a string, this is supposed to be an array of something that has a close method on it or what have you". And the types that you declare in Scala reflect good behavior in Ruby. So you are not doing anything that is different then what you should be doing in Ruby except what you get out of it is a compiled test to say whether or not whether you are passing the right parameters. And quite frankly if I have a very complex assignment in Scala, I will explicitly define the type of the variable that I am assigning to because it is good documentation. I have a map that would filter in a fall, and that is all in one line, that is really an early stuff, what is it going to turn into, tell me in six weeks or two months or six months when I am picking up the code what I did here.

So where I find that I have to explicitly declare types in Scala is where I would have and should do so in Ruby. We have similar code conciseness, and actually Martin Odersky who is the father of Scala in his lab they recently did, they studied with some of the cognitive guys over the EPFL, and they hooked up people's eyeballs, like these eyeballs sensors, to see how they perceived code. And no matter whether they were looking at Scala, Ruby or Java, they spent almost exactly the same time per language token looking at code, so if you say how few language tokens can we use to express our business logic, that's how fast somebody can look at the code and perceive it, that's how fast somebody can write the code because no matter what language you are writing in you write about the same number of line of code per day. So if you have code conciseness like Ruby, but built in tests static typing, it seams like it's a nice blend of both worlds.

Is it true that Scala has a stronger type system than Java, yet it's as flexible as Ruby?

The first answer is yes, the second answer is not quite as flexible as Ruby. There are certain things that you can still do in Ruby that you can't do in Scala because we can't do them, I mean with Ruby you can change the dispatch table of an instance, the kind of meta programming things that you can do in Ruby you can't do in Scala, whether that is good practice or not, that is another question but there are some things that you can do in Ruby that you can't do in Scala, I mean Rails has a little trick when you pull an array of elements out of the database, if you delete something from that array it drops it from the database.

Even though you have an array Rails does some meta programming magic and that array's delete method or remove element method or whatever it is has a different behavior. Is that good? I don't know. Some people like to use it, I personally spent three hours chasing a bug before I realized that happened one night. But, by and large, in mainstream coding you get an extreme ability to define types and how they relate to each other. So for example when you are doing query building in Lift, Lift knows that particular columns in the database require particular types for queries, so, a varchar is going to require a string type.

So when you are building a query between a varchar column you can only pass a string in as a query parameter, if you got a datetime column you can only pass a date in as a query parameter, if you have a numeric column you can only pass a numeric type in. So Lift's query builder leverages off the Scala type system to be able to enforce these types at compile time so you don't get a runtime error: "Geez, I was expecting a string but you sent me an int, I don't know what to do with it". Or maybe I will lock it on and I will do a two string on it, well that works for strings but doesn't work for dates so you start moving into those things where having the compiler check this stuff for you and flag an error is nice, at the same time the user code for the query building in Lift is as concise as query building in Rails, the person who's writing the query doesn't have to specify any explicit types.

Scala's type inference says: "Oh this is a type of foo or a type of string, or actually it is a type that contains a string". And therefore the second parameter to the query must be something that may be converted into a string. And all that stuff is done for you with, I think it's like a three line type declaration in the library, the three line type declaration took me a part of the day to write but the library consumer, the guy who is writing the query, writes a query just like he would write it in a dynamic type language.

In the same way that Lift framework exploits the type system of Scala, all the paradigms that exist in Scala, but as a user do I need to learn all these paradigms like actors and functional programming?

No. You can pick up Scala, you can open up the Scala interpreter which is like IRuby in Ruby and you can start typing Scala code. And type 1+1 and Scala infers int 2, infers the type of 1+1. If you type 1.0+1, it will infer that that's a float. And you can create a list by literally saying List(1, 2, 3), Scala knows that it's a list of ints, you can then perform a Map operation, so you can say List(1, 2, 3).map(A => A+1), which will add one to each element in the list in order to turn into a new list.

So that kind of coding, even though there is a strong type powerful system, with type inference and everything strict underneath the covers, you as a programmer are writing almost identical code to what you would write in Ruby. I mean the difference between passing code block in Scala, and passing a code block in Ruby is the difference between curly braces and round braces, the difference between defining a list in Scala and defining a list in Ruby, whether you use square braces for your array or say List for your list. I mean it kind of looks and feels very similar. I actually wear two different hats: when I am doing consulting projects for clients, I, by and large, wear the library consumer hat.

The library consumer hat is one where I am saying "I am thinking about my code just like I would be thinking about Rails code". I did two years of Rails coding and Rails is a thing of duty, and it is a thing that contains a lot of great ideas. And I tried to incorporate a lot of those great ideas into Lift. But the touchdown for me is am I writing more code? Am I writing more complex code in Lift, than I would in Rails? And if the answer is yes, than I have done something wrong as a library producer. Then I take off my library consumer hat and put on my library producer hat and say: "How can I use Scala's complex type system, implicit conversions, view bounds, other Scalaisms and functionalisms to make the job of the library consumer easier?

Because the library consumer should be able to write code as concisely and easy in Scala as one would in Ruby as one would in Python. As a library consumer, as a person who is picking up Lift to use, as a person who is building applications in Scala, you should not have to know more that you know in Ruby but think about what Rails does with meta programming, you have to be a black belt kung fu meta programmer to get meta programming right in Ruby. Why the lucky stiff is a genius because he can do that sort of stuff and do it well.

There aren't a lot of people in the Ruby community, who are at wise level, Chad Fowler comes to mind, DHH comes to mind, those guys do meta programming in their sleep and they are awesome at it, and it is no different for people who are doing complex typing stuff in Scala. So the same kind of hard work that goes into meta programming in Ruby is the same sort of stuff that you would do in Scala with the type system. But the end consumer of Ruby code, doesn't have to touch the meta programming that goes on underneath the covers, to get the benefit of it just like in Scala you don't need to touch the type system, to get the benefits of it.

What are the Scala concepts that form in some way the identity of the Lift framework?

One of the things that I tried to do with Lift is abstract the http request-response cycle. Because as an application developer I should be less concerned about the plumbing and the marshalling of my data form here to there then I should be about what my business logic is. So one of the key stones in Lift is any user interface element in the browser is assigned a globally unique ID, that corresponds to a function on your server. So when that element is submitted, whether it is submitted over Ajax, whether it is submitted just with a regular form post, what have you, Lift then invokes the function with the value of that HTML element.

So using Scala's functionaliness, we have taken away the need to name elements explicitly, we have taken away the need to unpack http request parameters, what have you. And it's a higher level dispatch of the user did X, here is a block of code to execute because the user did X, and for those VB programmers in the audience, it's the same sort of thing, you do VB. What do you do? You open up a button thing and you say "When the button is pressed execute this code, when the user types something over here, here is the validation code for it and here is how you do the errors". You will actually see very similar looking code in Lift because it should be a lot more like running a desktop application.

It's actually also bi-directional, so Lift has a support for comments. So Lift has the ability to have server side state that changes and the change in server side state can be reflected in almost instantly in a browser, so when you say that a certain chunk of screen real estate in your browser is associated with a component on the server side, Lift inserts all the right JavaScript machinery to do long polling and on HTML 5 becomes available we'll do it with HTML 5 rather than long polling. But basically what it allows you to do, is it allows you to associate screen real estate with server side component.

You don't have to worry what the GUI, the id of the span that contains that screen real estate, is because Lift decides that for you, you kind of go through the things that you would do as developer to put all the pluming pieces in place to push stuff from the server back to the client. Lift does that for you, yes sure, you can do that yourself if you want to, you can have full access to the http request starting point, but by and large we are using the same patterns. And what I have done is I have looked at the code that I have written, the code that hundred of other developers have written and tried to abstract the best practices into the framework so that we can push from the server to the client, we can take from the client and go back to the server without having to worry about the fact that we are using http as a transport. I mean, conceivably you can use MQ and messaging as a transporter will work just the same way.

We know that in your experience you took a Rails application and you converted it into a Lift application. Can you tell us more about this experience?

Yes it gave me the opportunity to borrow a bunch of good ideas from. Anyway, there was actually more than two years ago and I was involved with a startup and I was also started the Lift web framework, and we were seeing performance problems, even at the low level of scale that we were going for, with Rails, so we decided to go with the J2EE stack and I discovered Scala a few months before that and decided that I wanted to have the benefits of Scala running in the J2EE stack and have conciseness of Rails so by and large what I did was I took the Rails constructs that we had and built the same constructs in Lift.

And then poured the code over so it was to a great degree a one to one correspondence, Lift's mapper which is our mapping layer is almost identical to active record and it got pretty much the same semantics as active record does, syntactically it's a little bit different so we use traits rather than X like or whatever, I can't even remember the particular construct in Rails.

Mixins and traits are very similar, but in Ruby when you define your class you can say X like, I don't know and it has many through relationships to another table you do that with a trait in Scala, rather than doing that with just an attribute in your class constructor. So Lift's Mapper is very similar to Active Record, we have an intercept mechanism, we have a re-write mechanism that is kind of like a dispatch table, but we also have the ability to use Scala's pattern matching to intercept web services requests.

So you literally write a pattern that says "Here is the first part of the query that we want to match, here is the second part, here is one part that we want to extract, and here is a guard that does stuff". So you will literally write Req("foo,," | ,, ID,, nill). That basically says: I want foo/| and I will extract the third parameter into the id variable and then you come at get request or post request which specifies what kind of request it is and that's all you have to do to do pattern matching on the request so it's a very declarative way and then on the other side of that, it just returns something that is XML or Jason or what have you and Lift converts it back into an http request that is piped over the transport layer.

What about performance, after doing this conversion of the website? Did you gain in performance and productivity?

Yes, we got a 6x performance improvement, we had the requirement of much lower code coverage in order to deploy successfully. So with Rails and Ruby we needed something on the order of 95 and 97% code coverage, in order to deploy without having errors in production. With Scala and Lift our code coverage requirements went down to about 60% and we were still able to deploy without having any problems in production. Buy a Feature, which is one of the projects that I am working on right now, it's not like a serious game, it allows Agile product managers to talk to their customers through an online game and listen to information about product features, we did a benchmark on Buy a Feature where we had over two thousand simultaneous users, on an EC2 large instance, we were serving 700 pages per second and a load average to 0.24 and we were actually running the database in the same machine.

So it's pretty good performance: shoveling 700 pages per second is good. But I saw a benchmark recently that the same hardware will do with Engine X it will serve ten thousand static pages per second. We are basically ten percent of that doing dynamically generation a lot of business logic behind the scenes and database accesses. That doesn't suck. So performance wise, we have seen the same kind of performance that one would expect to get from a well written J2EE application, but it's writing even with Wicked or doing stuff with the Spring framework, is going to cost you a lot more code wise then it will to do Lift.

Using the type system as a means of testing is very convenient, but you don't have any control over the error messages in some way over the assertion failures. How is that a problem for users when they get themselves into a NOT and they have to decrypt a compiler error?

Martin Odersky and his Scala team have done a lot to make compiler errors a lot more readable and understandable. They are not going to give you BDD style "This should do this but I failed when I tried" kind of error messages, but for at least a Java developer, they will get the line number, they will get a description of what the problem is so for example going back to the query example, if you passed through a string where you were expected to have a number, the error messages literally lined x in the program parameter y was supposed to be a number, but I saw a string and I don't know how to convert a string into a number. I a paraphrasing a little bit but literally you will see an error message since I was expecting x and I got y. Which is enough for most developers to go "Oh I passed the wrong parameter in, let me pass the right on in".

And when implicit conversions are required as well, it requires an extra layer of complexity?

It has an extra layer of complexity, but the error messages turn out to be the same thing, so if you have the implicit conversion, and a view bound or an implicit parameter that says here is how to convert from this thing into this other thing, the compiler literally says I got something x thing but I don't know how to convert that into a string, so I got a string and I don't know how to convert that into a number. So the error message is actually telling of the failure to be able to do the conversion, but it tells you what the expectation was and what the delivery was.

So you may as a programmer sit down and go "Well, I can solve the problem in two ways: it's telling me that I have to convert from an x to a y, maybe I should supply the y in the first place, or maybe alternatively I should tell it how to convert an x to a y". So for example if you are passing in a double into something that is expecting an int, then you might want to add the implicit conversion. No, you probably wouldn't want to, come to think of it.

But it's not back to STL days and C++ compilers where the type system frustrated developers more than anything.

About a year ago there was an influx of Java developers into the Scala community, and they complained very vocally about the compiler error messages, and there was a whole wiki where you could say "Here is a compiler message that I receive and here is the compiler message that I would have wanted to receive". And there was a lot of work in the EPFL, to take that seriously and the work continues.

And just quickly, you said you had different code coverage requirements on the Scala, the code cover tools that work on Scala source files.

Yes, anything that works on Java source files also works on Scala source files.

Actually can I jump in and dig a little bit deeper into the answer to that last question. Scala compiles down to Java bytecode, Scala interoperates with existing Java libraries, all existing Java libraries. So anything that you have ever written in Java you can call from Scala, any interface you have in any of your Java classes, you can implement as a Scala class, any classes that you have in Java you can subclass as a Scala class. And Scala bytecode looks exactly like Java bytecode. So you can even use the Java debugger, JDB to single step through your Scala code, to inspect your Scala variables, etc.

The vast majority of tools even AspectJ allows you to rewrite and intercept Scala code. So to the JVM and the Java libraries, there is no impedience miss match between Java code and Scala code, which is a really nice benefit, and because the Scala object model is a super set of the Java object model it means that you are not loosing anything coming from Java going into Scala.

But with this thing with the promise of integrity you get some kind of construct that is coming from Java and doesn't make sense for Scala. Like for example in Scala there is the option type at the same time. What do you think of this?

I think that when I am writing Scala code and bridge into Java code I convert nulls and Options. Scala has a construct called Option. Option is a container that contains either zero or one elements of a type so you can have an option of string and the option will contain either some string or none. But the nice thing about options is they can be used in a lot of different Scala constructs including the "for" comprehension. So you don't have to do an explicit null test if not null go do this thing, if not null go do this thing with options.

You basically say "for(v = Option of something) then do something" so you don't have to do the explicit test in one option case. There is no difference lines of code wise between doing a null test and using a full comprehension to iterate through an option. If you have nested options so you have five or six things that may be null, may not be null, may contain something may not contain something, you have one line for each of those until you get to your business logic line.

And so just like when I was programming in C half the time I ignored return codes, why should I test that return code, I know the IO is going to succeed, why test whether or not I get null back from this hash table that I am querying because I know the values in there. You kind of go through and you make these assumptions as a Java programmer, as a C programmer, Scala gives you a different way of approaching this assumption and just like checked acceptance in Java, I give you a mechanism for testing whether or not your IO operation succeeded rather than just making the assumption. Option give you the ability to test whether or not you received a useful value as a return from a call.

And so when I bridged down into Java code, I will wrap the stuff that comes back in an option if there is a chance that I will be getting a null, if there is a chance that the logic of the thing that I am calling says if I can't compute the value I will return a null to you. I will just do a simple null test like come back and actually have a helper method that says "Convert the thing that is coming back from whatever it is into an option of whatever it is. Which will give me built in null testing for nearly free.

As a matter of performance, under the covers options cost you very little, so even though you are putting a wrapper around your thing, around your object, your wrapper is very low cost from a performance stand point. But from a developer performance stand point you do a lot better because you have yet another built in test, an implicit test and you also have a way of expressing logically for your future self or for other members of your team that a particular method may not be able to compute a returnable value. So the paradigmatic example that I use is converting a string into an integer. There are lots of ways that it may not happen, but rather than throwing an exception or returning the integer equivalent of a null, or return a negative one if I can convert this thing into it.

You return an option, you say "None, ok, I couldn't do this" but it's a perfectly reasonable optional case that you can't convert a string into a number in fact it's a very common case but it gives the future you an ability to sit down and go "Oh ok this thing can logically fail, if this is going to logically fail here is the fail value, and then I can deal with the logical fail value in whatever consumes the code. And if I am doing something worth user input great I will juts say back to the user "I am sorry I didn't understand that". Or if you are doing something truly as exceptional, you can then raise an exception if you can get an option back. But it becomes your choice rather than something that you have to explicitly catch when the failure issue happens.

Normally in an enterprise we need an IDE to be able to work because I am not going to work with just a simple text editor. What can you tell us about IDEs regarding Scala?

They suck a whole lot less then they did three months ago. I actually spent a good chunk of yesterday with the person that is developing the Eclipse IDE plug-in for Scala. I have spent substantial time with the IntelliJ plug-in, and I use the NetBeans plug on a regular basis. The stability increases in the plug-ins has been remarkable, in Scala 2.7.2 which came out in November, the EPFL guys added a whole bunch of hooks so that the IDE components could hook into the compiler cycle, could hook into the AST to do simple test highlighting, a whole bunch of other stuff. And ever since then things have gotten a lot better.

So rather than each of the IDE developers having to write their own Scala parser, having to write their own AST interpreter, and having to figure out: "Well, I am doing everything that the compiler is doing except I am doing it myself and the language just got a new feature and I have to go and implement that", you don't have to worry about that anymore, you hook into the compiler. In terms of code navigation, I think IntelliJ has the other two plug-ins B, in terms of overall stability currently it is NetBeans, but the Eclipse plug-in is making significant inroads, it was the case up until recently that if you had XML literals in your code, Scala supports XML literals, the Eclipse plug in didn't like it very much unless you explicitly put parenthesis around your XML literals that requirement is being lifted. I am figuring it is probably going to be six months, maybe even a year before you have an IDE at the same level as a Java developer would expect with the full refactoring capabilities, full code navigation capabilities, etc. But the change has been remarkable and I am no longer reverting to Emac, when things get rough.

Is Scala a functional programming language or is it a component oriented programming language? What is Scala?

I don't know, I was at a dinner last night with a whole bunch of Scala people and we asked that question and I think Martin viewed Scala as a language that is a competent functional programming language but most of the bounce that are being pushed in Scala are being pushed on the OO front, and having traits in the language and traits that have composition rules, so basically in Scala you can slap an interface on anything so you might have an IO interface that you slap on to a hash table or something, it's meaningless but it's cool.

With Scala you can actually say a particular trait and a trait is everything that an interface gives you plus methods plus data. So you can simply say that a trait is a contract the class has to engage in like an interface or at the very other end of the extreme a trait is something that implements all of these methods like the Ruby mixin so that when you compose a class or a trait when you compose a trait into a class you get off all those methods for free. It's a lot like multiple inheritance but you don't have the diamond problem, because the way they do the mixins they liniarize stuff and so you don't have to worry about who super is because there are rules so you always know who your super is based on the order in which you chain the trade on to your class. So if you look at that, that is about how far into the forefront of OO, at least as I have experienced.

So a lot of the goodness that C++ multiple inheritance gives you, for the ability to mix in things together, Scala gives you that but it also overcomes a lot of the problems. If you intercept that with Scala's type system, you can also do a lot of nifty things at the type/OO intersection. So you have rules as to what you can mix your mixins into, you might have a type parameter you might have dependent types, and Scala supports both, and both are useful. Type parameters are parameters that you pass into the class on construction, dependent parameter is something you can define in a subclass.

So Scala blends the type parameter of functional style with the OO depended type style all together in a way that I find to be useful because I can pick and choose which one works best. So very long winded way of saying, I think that Scala is primarily an OO language, that has a lot of nice functional features such as pattern matching, immutable data structures, as first class citizens in the library, default citizens in your library, has the ability to pass functions, blocks of code around its parameters, and a flexible syntax but you are going to find a flexible syntax in Ruby too and Ruby isn't necessarily identified as a functional language. So hybrid that leans more towards OO is probably the way that I would characterize it and I am sorry to Martin if I have mischaracterized it.

There was a project to run Scala under .Net, did you hear about this project?

Yes every once in a while somebody comes on the mailing list and says "I tried to run Scala on .Net and it didn't work as well as I expected it to". The Scala compiler is a multi phase compiler so Martin Odersky, who is the father of Scala, also wrote Java C 1.1 through 1.4, so he knows his way around bytecode compilers. He put near the code emission phase he split the compiler so there is an internal representation of the code and there is one backend that will spit the code out to JVM bytecode, there is another backend that will spit the code out to CLR code. Enhancements can be made to the language and the compiler and either code can be spit out. The libraries on the other hand are primarily designed for the JVM, so Scala has got an actors library that is like Erlang actors, but that sits on top of Doug Lea's Fork/Join library.

There is no equivalent library in .Net, so the actor library in Scala is not available in the .Net implementation. There is a minor impedance miss match between Java strings and .Net strings. You can go through the list and you can get a running "Hello world" program in MSIL which runs on Mono or runs on the CLR, but I would not count on Scala as a production level .Net language, you don't have any IDE support, so all the IDE support that you have for Scala is JVM based IDEs and admitting JVM code. Quite frankly, if I were to do stuff on the JVM and I wanted the features of Scala, I would probably go with F#.