Keeping Scala Fresh(er)

| by Alex Blewitt Follow 4 Followers on Dec 09, 2011. Estimated reading time: 8 minutes |

Last week saw a flurry of activity in the Scala landscape, with a Yammer mail leak stirring up questions. Typesafe subsequently published a response, graciously noting the feedback and painting a picture where Scala would improve in the future.

Top of the list is the upcoming availability of the Scala IDE for Eclipse, which has been undergoing development for a long time. Originally started by Miles Sabin, the project is now resourced from TypeSafe's employees as well, and currently exists as 2.0.0-RC2. Like other pre-compiled Scala libraries, there are a number of downloads which are hard-wired to a specific Scala version; you can use any one of 2.9.1, 2.8.3-SNAPSHOT or 2.8.1.

Other commercial IDE vendors already support Scala; the recently-released JetBrains IntelliJ IDEA 11 contains Scala support out of the box for both commercial and community editions.

TypeSafe also responded to specific criticisms, such as the learning curve requirements, build times and performance as part of their getting down to work post. They note that Scala is a work in progress, and that whilst performance is important they are keen to avoid any premature optimisations as the language matures. For building, they recommend the Scala Simple Build Tool, or SBT, which operates in a daemon mode to save time between startups of the build chain. And, as part of increasing the community around Scala, have now moved Scala to GitHub.

However, their comments on binary compatibility did not satisfy many. Whilst TypeSafe offers a subscription plan, which guarantees a commercially supported and maintained release of the Scala+TypeSafe stack, they did not consider the greater community and framework angle that has caused the growth of other platforms such as Java, Python, PERL etc. Instead, they offer a closed-source Migration Manager which can statically determine whether there may be any compile-time variation between builds. They note they themselves use this tool to ensure that Scala 2.9.0 was backwardly compatible with 2.9.1. They also claim to aim to provide all frameworks a developer could need, such as common libraries for testing, IO and Software Transactional Memory.

David Pollak, creator of the widely known Lift web framework, isn't so sure that this is the right focus. In Scala's Version Fragility makes the Enterprise argument near impossible, he argues that the fragility of the bytecode is directly harming the growth of the Scala ecosystem outside of the TypeSafe stack.

An attribute of Scala is that the Scala compiler generates fragile byte-code. This means that all the code in an executable (JAR or WAR) must be compiled with the same library and compiler versions. If you're using Scala 2.9.1 in your project, you must compile against a version of Lift that's compiled against Scala 2.9.1 and all the Lift dependencies must also be compiled against that version of Scala. This is a reason that Lift has precious few Scala library dependencies.

Scala's version fragility creates two huge costs for complex enterprise-type systems:

  1. All dependencies must be compiled against the exact same version of Scala and every referenced Scala library which makes in-house builds when there are multiple teams amazingly complex or time-consuming
  2. External libraries (like Lift) cannot have multiple layers of dependencies because you need to match the exact version of every library and Scala version all the way down


Unlike others (who maybe have opinions but no direct Scala experience), David is one of the few who can boast five years involvement working with Scala as a language and with the wider community. The operation of the Lift framework project, along with its guiding principles, is a model of excellence in the Scala (or indeed, any) world as how a community should operate.

The problem this impacts is not an issue when you have small development teams, or you are the only team in your organisation using Scala. If you're doing a greenfield migration to Scala for an initial project, the cost of migrating that team to a new release is a matter of less than a day's work. However, throw in two teams and suddenly there's a co-ordination issue that wasn't there before. The problem degrades exponentially:

So, if you're in an organization with more than 2 or 3 development teams that are all generating internal Scala code, you can generally agree on the version of Scala and the various libraries that you're going to use. When a new version of Scala or the libraries are released, you can get a couple of people in a room and choose to upgrade (or not.) But once you get get past 3 teams, the coordination efforts become significant. Trying to get 10 different teams to coordinate versions of Scala and Lift and ScalaZ and Rogue and Dispatch, etc. is going to be a significant cost.

As a library builder, the version fragility issue means that our library is not nearly as good as it could be and we can't service our community the way we want.

David has voiced his concerns before, in InfoQ's Barriers to Scala Adoption article published in October. However, he feels that the issue still isn't getting enough attention and that the closed-source bytecode checker is a plaster on the symptom rather than an underlying fix.

What have I done about the issue? Over the years, I've raised the issue privately and it has not been addressed. I have tried to organize a project to address part of the issue, but haven't had the volunteer uptake for the project.

A year+ back, during the 2.8 development cycle, some of us got together on the "Fresh Scala" project. Basically, the idea was to have continuous integration of the major libraries in Scala-land and weekly milestones so that the community and the ecosystem could give better feedback on new versions of Scala. Due to a lack of volunteer time, we never got the CI system built and here we are a year and a half later with the same Scala version fragility issues.

The Fresh Scala initiative didn't get much traction a year and a half ago, but Josh Suereth of TypeSafe claims responsibility in his Scala Fresh 2.0 post. In it, he announces Scala Fresh 2.0, a reboot of the project with a final destination of GitHub:

What happened to Scala fresh?

People may not realize but Scala Fresh failed because *I* failed Scala Fresh. I had very little time between a more than full time commitment at Google, writing commitments, and kids. This should not be an issue in the future, thanks to Typesafe taking binary compatibility seriously.

I've begun work on what can only be called "Scala Fresh 2.0". That is, a place where community libraries will be built and deployed against the latest version of Scala. This can serve two purposes:

  1. Ensure that future versions of Scala do not break community libraries
  2. Ensure that the core libraries of Scala are available for every major version of Scala.

Josh suggests that binary compatibility is the responsibility of library developers instead of the platform:

Binary compatibility is a community effort. I know Typesafe is doing what it can with its resources, and I'm personally tackling as much as I can (probably trying to juggle too many balls). However, if you want to help, please email me!

The last point is worthy of note; it is possible to write Java code which evolves in a backwardly incompatible way as well (removing public methods, changing method signatures etc.). Having libraries evolve where methods are removed or modified will cause backward compatibility issues whatever the language. On the other hand, designing libraries to be backwardly compatible is something that many Java developers automatically do, and tools like PDE's API usage scan can help perform a similar role.

However, where the evolution of Java libraries is somewhat predictable (by virtue of the class implementation being in one file), Josh highlights that it's possible to introduce incompatible changes via traits in Scala, since they are (in effect) an interface+implementation:

 trait Foo {    def foo = "foo" } trait Foo { // backwardly compatible with the above   def foo = "foo"   def bar = "bar" } --- trait Bar {   def foo = {     " foo"   } } trait Bar { // not backwardly compatible   def foo = {     bar + " foo"   }   def bar = "bar" } 

As a result, it's possible to have source code compatible changes (a class which has the Bar trait will still compile in both cases) but clients compiled against the old version of Bar will not have a bar() method, so newer clients will attempt to call a method which has no implementation.

What causes some pain is that Scala's key libraries suffer from this all the time. The Scala collections libraries are tweaked with each release of Scala, and since most APIs use the Scala collections libraries, there is a hard-coded dependency on the version of Scala used to compile (and the core libraries to compile against) in the generated bytecode. Some have started asking whether it is a better idea to ignore Scala as a compiled language, and treat it as a 2.10 development path, and no sign of the backward incompatibility being addressed, it is safe to say that if you want to move forwards with 2.10 it will require another recompile of everything in order to be guaranteed to work. InfoQ asked TypeSafe for comments last week after their getting back to work piece, but no response was received prior to the publication of this piece.

Rate this Article

Adoption Stage

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Why? by Victor Tales

Why another programming language?

Re: Why? by John L

Standard question. I think the web needs a standard answer. How about this: ?

Seems totally out of proportions by Serge Bureau

You do not have any guarantee in Java either of binary compatibility ?

In fact Traits are like interface with implementation, and as an interface if you modify it after you release it to the public, it is like an API, it cannot change. You can only write a new one and eventually deprecate the old one.

So how is it worst than in any languages ?

Re: Why? by Dan Tines

GW-BASIC is all the programming language anybody would ever need on a computer

Re: Why? by Serge Bureau

Amusing, I am not the only one asking why.

New languages do need justification, but not according to you. A mess is preferable ?

Re: Why? by Luis Espinal

For a blanket question, a blanket answer:

Why another programming language?

For the same reasons we don't code (mostly) in COBOL and FORTRAN anymore. Seriously, it's not rocket science, and the reasons should be self-evident to anyone that codes for a living (specially for anyone who does Java development.)

Java (the language, not the Java/JVM technology) has its warts with a clear cost associated to them. That's the main technical/business reason for another JVM language (change the language, but keep the tried-n-true technology under neat it.) On a professional level, most of us who have been doing Java for the last 10-12 years (or more), there is no professional satisfaction in doing Java, nor in working the same class of problems and tasks associated to them. So that is the second (and less important reason of why another language.)

Anyone worth its salt in software knows that a language is just a tool, and that different problems require different tools. In the same sense that you don't see someone saying "I'm a hammer carpenter", you are not a Java developer or a C# developer, or a C++ developer. You are a developer, and you should be cognizant of what tools best to use for solving what type of problems.

Beyond Clojure, there are little alternatives to large-scale, message passing, multi-core development. So, in that very important type of scenario, Scala comes in.

For other type of problems, say. to put a web-face on a database, Scala shouldn't be an option. Heck, Java shouldn't be an option either (it never has been). There are better, simpler, more cost effective alternatives out there (Ruby, Python comes to mind.) We have industrial numbers to prove this assertion.

Then you consider the JEE legacy. It is an standard that unifies development common knowledge and practices. But it also has its deficiencies. That is why you see things like Spring Framework (which itself is rooted in JEE.)

For some situations, you need less (think RoR). In other cases, you need something completely different (think Akka.)

So the answers for why another language (or stack/platform) are varied and context-specific. They aren't rocket science or hard to infer.

Re: Why? by Luis Espinal

GW-BASIC is all the programming language anybody would ever need on a computer

How do you implement a distributed, airline reservation system with it? Or a DSP algorithm that needs to go on a FPGA?

Re: Why? by Luis Espinal

GW-BASIC is all the programming language anybody would ever need on a computer

BTW, I did work (as in real work, not play around) with GW-BASIC (and another primitively bestial thing known as PICK BASIC). Thank God we have other languages.

Re: Why? by Dan Tines

New languages do need justification

Oh really? So where's this committee that I should submit my idea to before embarking on my new language?

Re: Why? by Dan Tines

Sorry the joke was kind of subtle and a play on "640K should be enough for anyone".

The parallel by Joao Pedrosa

Just like a developer does better when the developer can keep all the architecture of a program in his head, I guess a language like Scala does better when all of the dependencies are loaded up in an IDE.

Gilad Bracha has talked about something I find dear that is how hard it can be to moduralize when you try harder to prove types. Something about leaking type information or... You can find more at

Before commenting on it I'd rather folks read the comments on his blog.

I think the problem with Scala community efforts is that programmers don't want to be fixing one another's source codes without being payed for it a.k.a maintenance programming. Scala doesn't make it any easier. :-) Type safe or not.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

11 Discuss