00:16:19 video length
Bio Alex Shatalin is the JetBrains MPS project lead. Vaclav Pech works for JetBrains as a senior software developer and a technology evangelist.
The Code Generation conference is the leading event on the practical applications of Model-Driven Software Development (MDSD). It offers a high quality learning experience by combining discussions and debates with a diverse range of practical tutorials and workshops. It will also give you the opportunity to share experiences and knowledge with fellow developers.
Vaclav Pech: Hello I am Vaclav Pech and I work at JetBrains as a software engineer on the MPS Project.
Alexander Shatalin: Hello my name is Alexander Shatalin, I am team lead of MPS Project actually.
Vaclav Pech: Well MPS stands for Meta Programming System, so it’s a Language Workbench and that is kind of specific in the way that it approaches code that it gives you a projectional editor, so it’s not a Language Workbench based on a parser but on projectional editing. The way you edit your code, you don’t edit it through a normal editor where you would have free text that you can freely edit, but you get the AST that represents your code projected on the screen, so you edit it in a controlled way. So you can’t create invalid, so to say, AST or invalid code that wouldn't be parseable.
Vaclav Pech: That is exactly the task for the language designer to give you an editor or multiple editors, that give you way to edit the AST indirectly, so you don’t touch the tree itself, but you have some predefined editors which might look very close to text editors actually and in many cases they do, but they can differ from plain text editors, they can provide constructs like decision tables for example or state machines sort of abstraction to edit it in a more comfortable way.
Alexander Shatalin: Currently not unfortunately, but we are planning to implement this feature in the future, so it’s kind of on the roadmap of the project.
Alexander Shatalin: You know that from the side of MPS all these elements are just a kind of rendering functionality for particular node of the abstract syntax tree, so we have a specific node rendering a specific element in editor definition rendering abstract syntax tree node as a table. So that is how it works basically. And nearly the same approach would be used for diagrams, so we will introduce additional nodes in the language which you can use to define editors which will allow you to create diagrams with nodes and links between them, somehow edit them. We'll use or tightly integrate this and which definitely what we have currently for editing stuff in MPS so you will be able to use Completion and the rest on diagram nodes. That is how it should be actually.
Vaclav Pech: But the tables you can view them as just a different way to arrange things on the screen. We might take a different example, let’s say an If statement, right, so you have If and a condition and typically you have a body and then you have Else and underneath you have the alternative branch. But with Projectional Editing you can decide that you would like to see these two branches not below one another but next one to another and we can project it on the screen any way you like and we can give each branch a different background color or different font, or whatever. It is basically playing with the visual appearance of the code on the screen. But it is all textual, is not that you have to take the mouse and draw a table and then fill in stuff, you do it from the keyboard and you refactor the code as if it was just a normal piece of code, so you can wrap the table, take the table and assign it to a variable or have it come back as a return value from a method.
Vaclav Pech: Yes, I would agree with that.
Alexander Shatalin: Yes, have some specific frames around that.
Alexander Shatalin: We have a proprietary XML-based format for storing that code, but basically you as a user of MPS should not think about that format at all, because we are hiding it absolutely from end users. We provide you with the possibility to work with AST nodes directly from the editor and we provide all the tooling around that with the possibility for example hide corresponding XML files when you commit or check out sources from the version control. So we do provide kind of similar projectional editing covering all the Version Control interoperability.
Vaclav Pech: So this low level format should never leak out, you should never see it even when you check into Version Control System it’s still hidden and you have the projectional editor and really you work with code and not with the XML representation of it.
Vaclav Pech: Yes, you don’t see the format that is used for persistence, it’s so solely used for persistence on disk, for nothing else.
Vaclav Pech: There are a couple of stages that you have to go through. So if you want to design your own language, first you define the abstract structure of your language, the abstract syntax, so the nodes that should form the abstract syntax tree. So if are building a version of Java for example, you would create a node that represents an If statement, but that node need a condition so it would have a child element that would represent the condition. It would typically be an expression and perhaps you would somewhere say it should be Boolean. And then you would say an If statement needs an If branch so that is a list of statements and then it has an Else branch which also is a list of statements.
And then you have to define what a statement is, what a list of statements is, so you would define these nodes and how they can be combined into a valid tree. And then you design the editor which we talked about, as the representation on the screen, so how the If statement should look on the screen, so we get an If text here then a place where the user can edit the condition, and then the place where the statements go. Once you have that you need to generate code, so you have to translate your language in to either some other language, so that is model to model transformation, so you say: "My abstract syntax tree needs to be changed into that abstract syntax tree".
For example my loop statement that I have my fancy loop statement needs to be somehow translated into the ordinary For loop in the target language, so you translate the concepts you have into the concepts that are in the target language. So you gradually simplify your language, I mean your construct into simpler and simpler languages and then you generate text. That is what you generate and then you run a compiler against it to create a runnable application. So that is the core that you have to have something runnable and there are extra things that you also need to define.
Alexander Shatalin: We have a lot of IDE specific features around that like you can basically start with Type System Rules for example to define some type system aspects for the language and then specify a lot of other stuff like data flow analysis and do some more IDE specific integration with the environment. So your actual DSL will look like a normal programming language in MPS, so you use all those features during the editing process and in addition to that I have to say that we have definitely some integration with the ANT scripting language where you can create actually an And script and call the MPS code generator from there so you’ll be able to basically get the source code on build servers somehow from the models. So it’s more or less integrated so this lifecycle is covered as well.
Alexander Shatalin: You mean the models in MPS? Basically you can think of the models like the sources in Java, because sources in Java or in any other programming languages actually intermediate representation of your program as well, so if you finally have to compile that and then you get something executable, so it’s something similar but in MPS instead of thinking of Java sources like as a single source of information, we actually think of Java sources as about some intermediate state like intermediate object models used by compiler.
Alexander Shatalin: It’s mostly used inside of the IDE itself, so we’re mostly using that for providing IDE functionality for your language, for providing actually a software developer with the necessary assistance from the side of MPS, because once you have the program working your data flow, you control that and it’s ok, I mean you don’t have any Null pointer warning, then it’s most probably going to the final Java sources which will be compiled properly and no problem there. You have to be sure about that.
Vaclav Pech: Data flow is pretty much an analysis tool that you run inside your IDE when you edit the code so it will analyze your code on the fly and tell you "Hey, this piece of code can’t be reached, you never get there or you might get a Null pointer exception from here". But you know once the data flow is happy then you don’t need to run it further down the chain, you just generate Java and provided the generator is correct and then the data flow related problems in the generated code.
Alexander Shatalin: Basically, that sounds funny but we are trying to reduce the number of languages distributed with MPS obviously because we are not going to address all the domains in the world, so the main focus for MPS is developing new domain specific languages. So first we are selling or we are distributing those languages which are used for describing new domains specific languages in MPS. Obviously since we are based on Java platform we have our good and powerful support of Java platform so it’s not a problem for you if you want to create some extensions in MPS going down to Java, the rest is kind of outside of the scope of the project a little bit. And I have to mention that we have a couple of powerful extensions for the Java language, right?
Vaclav Pech: So it's not the only Java that we distribute which is basically a projectional clone of Java so to say, but we also enhance Java with something that we consider to be very useful, so things like working with collections in a functional style or having closures as first class citizens in the language or working with regular expressions in a convenient way and date and time. So these are all composable extensions you may or you may not use them (it depends on you), but we typically use them because we enjoy the power of these extensions. The thing is that even using these extensions you still generate Java 1.6 compatible source code, so you don’t need any runtime dependencies on MPS in your code if you generate it from MPS.
Vaclav Pech: All the closures are translated into anonymous inner classes and working with collections gets translated into loops.
Vaclav Pech: Yes you can build on the Java that we have, so we can built on Java with collections and closures and create whatever you like. Or you can start from the ground up so you might say "I assume there is no language and I want my own language that looks totally different from Java or C", so you can start from that to build your own language and then someone else can create extensions to your own language.
Alexander Shatalin: Or you can pick one of the existing languages which was already built for MPS by somebody else for example by Markus Voelter, who actually created a C language support in MPS and actually you can pick this language because it’s open-source right now and stack your own extensions up on that language, similar to the way you do that with Java in MPS just out of box.
Alexander Shatalin: I doubt to make sense to build Kotlin with MPS, because Kotlin is addressing a little bit different use cases as far as I know, but basically what we are going to do is we are going to extend MPS with additional language constructions, more or less reflecting the power of Kotlin but based on Java.
Vaclav Pech: But if you wanted, you could build from the ground up a Kotlin language or any language you like in MPS so you could have the projectional version of any programming language that you’d like to create, but Kotlin is an independent activity of JetBrains so it’s kind of a different group.
Vaclav Pech: About MPS itself it’s an open source project, and it’s under the Apache 2 license, so you can get it to use it for commercial purposes if you like, and actually we have commercial users who use MPS to do business and I’m very happy to see all these people here at the conference, we have pretty large set of customers and users here at this conference. Typically people in Academia use MPS as well to do research because again there are no restrictions in how you use MPS.
Alexander Shatalin: So actually MPS is open source and this is a good platform to start something upon it, so we are open for any requirements and requests from the community and we are willing to make this community grow, so that is basically the state of licensing.