JetBrains Meta Programming System Supports Language Oriented Programming and DSLs
Meta Programming System (MPS), a new Language Oriented Programming tool from JetBrains, allows the developers to extend programming languages as well as create Domain Specific Languages (DSLs) for enterprise applications. JetBrains development team recently announced the release of beta version of MPS software.
Language Oriented Programming (LOP) is a new style of programming where the developers can create specialized languages just as they can create classes or methods in a conventional language, use them to develop software, and extend them when and how required. Martin Fowler blogged about Language Workbenches to use as IDE tooling to make language oriented programming a viable approach. He also wrote about MPS as a language workbench tool.
MPS is also useful in creating DSLs where you can define custom language editors and other constraints for any new language. Domain experts who are not familiar with traditional programming can work in MPS with their domain-specific languages using domain-specific terminology.
MPS software is free for all users and a major part of its code will be open-sourced under Apache 2.0 license. It started in 2003 as a research project and JetBrains developers have been using it for developing some of their new products. The team is targeting for Beta 2 Release of MPS next month and version 1.0 release some time in the first quarter in 2009. JetBrains also released new versions of their IDE tool IntelliJ IDEA (version 8.0) and the team collaboration tool TeamCity (version 4.0) products back in November. More concepts on how MPS works and the new beta version download are available on their web site. There are also a user guide, tutorial and a blog site to learn more about the tool.
InfoQ spoke with Konstantin Solomatov, core MPS developer at JetBrains about the MPS software features and the future road map of the project. InfoQ asked how MPS compares with other modeling and code generation tools like Eclipse Modeling Framework (EMF) or openArchitectureWare (oAW).
MPS is based on concepts similar to EMF and oAW. All three technologies allow creating a meta-model, describing its editor and other aspects such as completion, model transformation (which we call 'generator' in MPS), etc. EMF models are usually edited with a graphical editor where you create a diagram representing entities using the box-and-line style. There are domains where this approach works nicely, for example, ER diagrams. However we do not think it is good for creating any relatively mature software.
The most interesting part of oAW is xText framework that allows defining textual DSLs that are parsed into EMF model. But this way of creating languages has its limitations too. If you want to combine languages inside of a file, you have to be sure that the resulting grammar isn't ambiguous, i.e. that for each input there is only one way to interpret it. It's hardly possible in text based languages. For example, two different vendors might want to create a language adding support for monetary values into Java. Both of them add a new Money type to the language. Consider a code that uses both of the new types. When we parse a file like that, containing a variable of the Money type we can't decide whether this type is from the first or the second language.
MPS, in its turn, stores its models as abstract syntax trees and allows editing them directly. In MPS, code looks like text and in many aspects it behaves like text. But since we never convert code to text and back, we don't have to worry about grammar ambiguities.
Since MPS isn't text based, it doesn't have any text-based grammar. You define a language by defining the structure of its abstract syntax tree, its abstract grammar. Since we don't have text-based grammar, there are no language compatibility issues, as they are compatible by definition. With Oslo and Ometa both being text based, there are problems similar to xText.
Comparing with Oslo and Ometa that only allow defining parsers for languages, MPS provides additional features such as editor, constraints, type-system, data flow analysis and generator.
What are the new products the JetBrains development team is using MPS as the modeling and development tool?
One of the new products is JetBrains issue tracker, code-named "Charisma", which is due for public release in Q1 2009. It is completely written in MPS. Charisma is already used as an issue tracker for MPS itself, for TeamCity and other products. While creating Charisma we developed a full stack of web development languages that serve as MPS' counterpart of J2EE.
What is the future road map and the new features for MPS project?
The main goal of the Beta is to gather feedback from MPS users and decide on the future roadmap. We have a plan for 1.0 but there are no big changes and new features planned till 1.0. We will publish a more distant roadmap once we release MPS 1.0 in Q1 2009. There you can expect debugger support, and a couple of new languages such as a language for defining UI.
mixing languages using common parsers
>> resulting grammar isn't ambiguous, i.e. that for each input there is only one way
>> to interpret it. It's hardly possible in text based languages.
Why do you think it's hardly possible? (see my comment below)
>> For example, two different vendors might want to create a language adding
>> support for monetary values into Java. Both of them add a new Money type to
>> the language. Consider a code that uses both of the new types. When we parse
>> a file like that, containing a variable of the Money type we can't decide whether
>> this type is from the first or the second language.
This sounds like a library problem not a language problem, or are you talking of introducing two different money-literal expressions with the same syntax?
Of course with common parsers you need to have an unambiguous grammar but it depends on how much syntax you define on a global level (i.e. the lexer) whether it's easy or even possible to do so. If for instance you use a scanner-less (en.wikipedia.org/wiki/Scannerless_parsing) approach things are straight forward.
Another way to face this issue is to share a common lexer (containing common things like identifiers, etc.) beween the languages.
Still it won't be possible to have the same syntax for two different expressions, but I don't think this is a problem in practice, because this would confuse the users as well, wouldn't it?
That said, MPS is certainly an interesting technology and it's good that such smart people like the Jetbrain guys work on a DSL-technology. Still I don't think the advantages (always unambiguous language extensions) of their approach are worth accepting the disadvantages (non-text based).
But we'll see. :-)
Oslo and the M framework
The tech preview contains a couple text representations that map to this AST, MGrammar and MSchema, others are available as samples, and users are defining new ones all the time. MGrammar is a text modeling language like lex/yacc. MSchema defines structural types with constraints. It also provides query and transformation over data. An MGrammar definition and generates a syntax directed editing experience in both Visual Studio and Intellipad, a text focused tool.
Regarding ambiguities, it is often the case that two fragments of a program can be interpreted in different ways but the context determines which one is valid. MGrammar can tolerate such "local" ambiguities. If an ambiguity is not local (there are two interpretations of a whole program), the intended meaning can be selected using priorities. No need to refactor the grammar.
The M specification is available under OSP licensing and downloadable at msdn.microsoft.com/oslo. Martin Fowler wrote some nice words about Oslo as well martinfowler.com/bliki/Oslo.html. Also, James Clark wrote down some "random thoughts" blog.jclark.com/search/label/Oslo.
Re: Oslo and the M framework
Why was the world flooded with XML in the first place? It's markup overhead and human readability issues were not a problem with the techniques used for cross platform collaboration prior to XML. What were those techniques? DSLs.
Anyone feeling dizzy yet? We're going in circles here.