Scala, Functional Programming and Play 2.0

Recorded at:

1. Hi, I’m Michael Floyd, chief editor of InfoQ and I’m here with Sadek Drobi. Sadek, how are you doing today?

Pretty good. I’m here at QCon, I’m Sadek Drobi, I work for Zenexity as Chief Technology Officer, Zenexity is a web-oriented architecture with enterprise applications on top of the web as a platform and last year I was working very hard on the second version of Play, the architecture and implementation of the second version of the Play framework, a framework that runs on top of the JVM for Scala and Java.

2. So, I was going to ask you what you were up to, but that is pretty much all you are up to right now, is that right?

Yes, that’s right.

3. Ok, so one of the things that we have here, one of the tracks that we are doing here is Scala and the rise of functional programming. So let’s start with functional programming first. It’s gotten very popular, so functional programming’s origins date back to common Lisp, so it’s nothing new. So why do you think functional programming is of such interest right now?

Sometimes I like to think it’s for the wrong reason. In a way it’s one need that we have more cores and people are running more cores, they are trying to think of better models for doing concurrency and of course when you talk about concurrency everyone knows immutable models, immutable data is much safer when concurrency is involved, because you can parallelize these that. A lot of people have been talking about these kind of things, functional programming is much more than that, it’s a really good way of doing programming, very high level, it’s declarative, it has a lot of properties, but the fact that it works good for concurrency is a side effect of this good model of programming, so yes, people discover a lot of good ideas in there and I guess this is the main reason for the rise. And then people start to recognize, for some cases it models much better my model or it describes better the problem I’m trying to solve, it’s much more declarative so there are less bugs. Or it’s much safer because it doesn’t have mutation and mutation is this cancer that can spread into the model and destroy everything, the program becomes easier to reason about. All these reasons that we start discovering, starting with a need for multicore and programming and concurrency.

4. How do you think Scala kind of fits into this paradigm then?

So, as you said functional programming is really old, it’s there since the beginning, right? People would learn it in the school and then they would forget it and for them this is the most awful thing they learned at school, and I think that’s weird because the way we teach it at school and at university is the wrong way. There is a book, "Structure and Interpretation of Computer Programs", it’s a really good book it describes everything, it describes functional programming too, and it introduces it the right way. So, functional programming is like everything is a function, we are reading everything on top of a function.

There is another programming paradigm, object orientation, we can talk about objects, the state is encapsulated, some people like to think that both can provide different ways of viewing the problem, they solve maybe different problems in different ways, sometimes you have state and you need to hide it behind some actions, this is object orientation, sometimes you have to separate data and functionality, functions are separated from data and then functions operate on immutable data. Two completely different paradigms, really different ways of thinking about programs, then there are several languages that what about having these both different paradigms, among others, there are other paradigms, mutable and immutable and so on, what about trying to mix these paradigms, so that they have the right tool for the problem when they need it.

So F# is doing this from another angle, coming from the functional paradigm to add some object orientation, while Scala is doing the other way around, which means it’s an object oriented programming language that has functional programming inside and it’s really amazing how they integrated functional programming into an object oriented way, which means that if we just look at Scala, you see something like a function, you’ll see like function is nothing special, is an object with an applied method, right? It implements an interface, and the interface’s name is function and that’s all of it and some syntax so that you can create this object in a less verbose matter. For example, in Java you would create a class, anonymous class with action or invoke method, this is exactly what a function is in Scala. Except that you don’t have to write this much syntax to express something like +1, that’s why it just gives you the syntax, it’s just syntactic sugar together with closure another problem.

But it’s just syntax to be able to instantiate a function object. That’s for function. Pattern matching is the same thing, it’s just an applied method that you implement on your object and now you can use it for pattern matching, which is a very powerful thing, but again you try to implement it the object oriented way, it’s just an interface that has an applied and now you get pattern matching. And all the other things, like also case class will become an applied method which represents some data structure case class, it will present an algebraic data structure but they implement it through inheritance and this applied method. And again, every functional concept, you go and implement it in an object oriented way. So basically Scala is an object oriented programming language, introducing functional stuff but in the same paradigm. It’s really interesting, it really mixes well, it’s not like "Here are functions, here are objects, do whatever you want with them", it’s much more integrated than that, and there are a lot of examples of this integration.

So that’s basically what Scala is trying to do. And then, if you look at something like, the result is really good. If you look at the collection library, I programed in a lot of languages, in Haskell , in F#, in Java, in C#, and all of them have pretty good collection libraries, but Scala’s the best, I mean the best in the design way because things will act the way you want them to act. So, if you get a list of something and then you want to read it like a sequence of things, it just works, because of this inheritance together with strong static type system, they both work together into this well implemented collection. We’re talking about inheritance, most of the operation is defined at a very high trait, which is an interface with some implementation, and everyone inherits this implementation but specialized for maybe optimization on other aspects.

This is the dream you want to have for some library, I implement something abstract and then there are a few operations I have to implement down and everything will work as well, that’s what you get with a collection library. Of course, with a little more sophistication at the type level, people talk a lot about this sophistication of the signature of the types, I guess it’s very rare that it surfaces this kind of complexity, and I work a lot with the collection, and there is no time that I can think about something I need in a collection and I don’t find it.

5. So being a kind of a multi paradigm language, do you think Scala is becoming too multipurpose, is it becoming bloated?

Yes, people worry a lot about this and it’s reasonable to think about this like you have functional programming, you have a very powerful type system, you have object orientation, you have a lot more and now they are maybe thinking about adding macros and the user is "If I a look at the code could I reason it in a simple way, can I understand it what this code describes"? Because you have this line and you have Lisp which has a simple model, you build everything on top of one simple model, you have languages with much more exceptions and much more cases, and Scala is in a way in the middle but a little more to the more syntax from Lisp, but the thing is there is one thing that is good about Scala, there is some reuse of concepts, you have something like the for expression, the for expression is something you can use yourself by implementing map and flag map.

When they introduce syntax and they try to introduce the general format syntax and not the special form of syntax, that’s really good, because as a developer you need to understand that four does map and flag map, and now anytime you see map and flag map introduced you know the compiler is just calling maps and flag maps, you can reason about it this way. Same thing for pattern matching, you know it’s an applied, so you can reason about it some way, again is just one concept and reused around, like for something when you use curly braces a complex expression is not a simple expression, so again you can introduce it anyway anywhere where you introduce a parameter it’s enough one simple expression, you can put curly braces and so on. That’s the reuse of concept. From the other side, there’s functional programming, object orientation and as we said, other paradigms.

You have more things to think about, let’s say it is harder to start with. If you take something, like for instance Haskell, Haskell has its set of functionalities, a very small set of functionalities and then you can mix them and get things but it is fairly small, whereas in Scala it’s bigger than Haskell, it’s bigger than these languages, it gets implicit parameters, something like type classes of Haskell, together with inheritance, and these two things overlap and produce different things in different ways in different contexts. It is a lot of things, it can be abused, but in an enterprise, in my enterprise we have 40 developers and we work with Scala and if you introduce some set of rules of using Scala I guess you will not get problems. Of course, there are two things: some people say this but they don’t even understand the syntax, for them it’s completely strange, once you get introduced to this syntax, these concepts that I talked about, it gets much less a problem. Yes, you have a lot of concepts, quite a few concepts you have to deal with, you introduce some rules, for example in my team I say "No one introduce any converges, I know they confuse developers, let’s not introduce this.

Let’s not do the type trickery, the full type power, and try not to use these", and then even at the team, Martin Odersky is trying to have this set of rules so that people don’t get into teams where people try to introduce these things that are not even necessary. There’s a goal, you have to scope, you have to say "I’m using Scala as a user developer, I should just use these and not use all the other things that a library user could use", as a library user you need these kinds of things to do something as rich as collection libraries, but as an end user developer you just can be happy with another set of functionalities from Scala. At least this is the way we try to view it.

6. So you work for Type Safe, is that correct?

I work for Zenexity, we do Play2 in collaboration with Type Safe.

7. I see. I wasn’t aware of that. So, tell us a little bit about Play and what you’re doing with that?

Play started as a Java project three years ago, as a way to do web programming simply and closer to web rather like doing something fairly strange like a strange framework from web. It’s in a way like Ruby on Rails, or Jungle, these frameworks that are like "We know we are in the web so let’s take advantage of that instead of resisting and try to abstract web away". This is what Play tried to do in the last three years, to do this on top of the JVM, there was almost nothing on the JVM that acts the way Rails acts on Ruby. The problem is Java is not that flexible as a language. Java has this syntax and it doesn’t have a lot of possibility for a framework that should look like Rails. Play is trying to do all that is needed, like reflection, byte code enhancement, all these things to get it really simple for developers to do basic rules, start with prototyping and framework and then trying to deploy it in enterprise and all of that.

So that’s the first thing, to get something really simple, not bloated kind of framework. That’s the first focus. The second focus is to get something asynchronous, in a way that not each request gets its own thread, but rather in a way that, a socket is completely different, separate from a thread, a normal wire container on a JVM- every request will get a thread and it will get until the end of its life, the request’s life, which is kind of annoying when you have a lot of users because you have a limited set of threads and that’s it, afterwards you can’t get more users. Sometimes when you do some IO operations, like you call a web service, actually you’re not doing anything it’s the OS doing the thing and it will be notified when it has a response and it can notify you. The problem is there’s no way of doing that with classic wire container, that’s why you have to block the threads while you do these things, which is completely unnecessary.

So Play tried to introduce the other way of doing this by being more asynchronous regarding these things, so when you have to do a web service, just handle the thread to the pool thread and when you have something, now you get the thread back and you try to do something. Which means you can get as many users as you want, as many sockets you have and then you can allocate threads depending on what you need to do for these requests. So if the user is just waiting for some event to happen, the user should be consuming nothing, zero, because he is just waiting for the event, and when the event happens then you push some data on that socket. Play1 was trying to do this but it was a very basic model, because it was still on Java and we tried to do with an ecosystem on Java, that’s why it was a very basic model but doing some asynchronous. Play2 goes much beyond that, the core is implemented in Scala, the core of Play2 is implemented on top of Scala, just the core because the APIs we implemented in Java and Scala respectively.

What we did in Play2 it’s completely reactive, it’s based on promises and iterates, something I’ll explain more in my talk tomorrow, but promises is a construct that will allow synchronization, which means I will give you a promise and whenever I have the value, I will push into the promise and it will get called back. It’s a kind of call back mechanism. If the user hits for some URL, he’s waiting for a value, I will hand off to the framework, I will handle a promise, then the user is still turning, the browser is still waiting for a response, but on the server side there is nothing going on for the user. Imagine there is another user that hits another URL that will get that weighted value for that user, what will be done, I will get this value, I will push it in the promise which will push the data into the user. That’s all that happens, which means you are not consuming any resources while you are waiting, because you are not doing anything. This is the first thing promises, and then iterates, so the main idea is that you have the first thing is this awaiting value, it’s very static it’s not a stream of values, it’s one value.

Another thing is when you want to stream things, because now in web you have the web sockets, you have server sent events, a comments and all these kinds of things where you want to stream a lot of data through the wire, in the same way if you don’t have anything to stream you don’t want to consume resources but you want to control this streaming. The first thing when we talk about streaming, people would think about Java’s input and output streams, the problem with these two things is Java is they are blocking, which means that if I have an input stream I want to read, get me some data it will block it and I will be blocked while it’s trying to get data from somewhere, from a file or the web or wherever it is. And then when it has it, it unblocks me. This completely blocking doesn’t work when you want to want to handle a lot of data, users on your application, what you would rather do is be reactive, which means "I’m doing nothing, please call me when you have data", when there is data it will be pushed and it will then be pushed through the socket, it’s completely reactive.

The problem with reactive model is that you don’t have control, if someone is pushing some data into this thing that we call iterate, it comes from the Haskell world, so when someone is pushing data into it and you don’t want any more data, you open the file and you want to see just the first two bytes for example, and you want to just say "I’m done", you want the respondent to say "I’m done, stop, just close whatever resources you opened", because it’s very important to close resources in the beginning not in the end, not to read all the file when it’s closed, it doesn’t make sense, a big file of multi Giga, you don’t want to read everything if you only need a first bytes. So, that’s what iterate allows you to do, an enumerator is a source of data, every time it pushes any data into an iterate, the iterate will have the opportunity to say "ok, continue and here is how you continue", and it gives you a call back, or "done".

And when an enumerator as a source gets the done signal, it can do anything with it including closing resources, closing sockets, anything it wants to do, it’s a reactive model but without losing control, it’s an inversion of control, which means that the source that is controlling but we didn’t lose control, we can still say "I’m done, I don’t want any data". And that’s what we implemented on top of Play2, there are some applications where we do web sockets server sent events and this is what’s interesting, if there are no events, the server is doing nothing, no resources, nothing is being used, when that stream of data is completely reactive and completely controlled over from any layer of the chain of these kind of streams that you are throwing at the client.

8. So, from the user’s perspective, the developer that is using Play, are there some best practices that they should be thinking about?

Yes, mostly we get our clients, because actually we use Play2 internally at Zenexity, this is our way of doing web application and then we try to publish it as an open source project, the first thing we say is try to take opportunity of this kind of model. If you are calling web service, don’t do it on the thread where you are, try to schedule an unblocking, we provide libraries for doing that, you’ll do a nonblocking thing, because that can do it very well and notify you do this rather, this is the first thing. Of course you can have some legacy code that you have to do that, but you mainly think in terms of these kinds of things. Of course you can use Scala actors, for doing something like actions, like send and forget, you send a message, something to be done and that’s it, you handle the thing to the user.

The practices of using this kind of model of asynchronism, the streams, the user should be aware of the model and try to use it. That’s from the stream’s perspective. From the client side, if you’re doing a web application, you need to do some JavaScript and so on. The first question from some people that are used to Java, classing Java web framework, they ask "how can I do JavaScript, I want a tool that would generate JavaScript". Our vision is that you need to master what you are trying to do, JavaScript is very powerful on the client side and you should try to use what is fit for the client side. Because there are a lot of frameworks, like Wicket, trying to make an abstraction on top of Ajax or an abstraction on top of JavaScript and then very fast you will get limited, because you will serve some functionality that is not implemented by the framework and then you have to find a solution and then you start JavaScript and the complication and the complexity.

Why not start it from the beginning from JavaScript, doing some beautiful good JavaScript, because that’s really possible, that’s all we do, we did all sorts of crazy applications with JavaScript, and works really well. Then of course, you can use something like CoffeeScript, you get JavaScript less variables, we use this kind of thing, we prefer JavaScript because in the mode of development it’s much easier, you are in the browser, you refresh your browser, you change, your refresh your browser, it’s very fast, we prefer this, but you can still do these kinds of things. When we talk for example about NoSQL or MongoDB, the main way is to try to do some code block and then wait for the result. What we try to provide is an asynchronous way of talking to these databases too, again to take advantage of this asynchronous model, the more you do it, the less you will have a problem when you want to scale your application.

9. Just one last question for you. A new release candidate was just put out about a week ago, where is that, I mean is it stable?

Yes, we are at release candidate four, and we are going to release very soon, actually next week, I don’t know when the date will eventually be, we’re pretty much done, it’s final, we fixed some bugs and we have an application deployed for ourselves at Zenexity, we already deployed application with what we’ve got, we keep fixing bugs, 2.0 is done, we will release it next week, and of course then we will start 2.1 and so on, because there are a lot of things we want to add but it is there.