We are here at Ruby Conf 2006, with Jay Fields and Zak Tamsen from ThoughtWorks. Why don't you tell our viewers a little bit about yourselves?

I'm Jay Fields, I've been with ThoughtWorks almost 2 years now, started out doing .NET development with ThoughtWorks and quickly moved into the Ruby space.

Zak: I'm Zak Tamsen, I've been with ThoughtWorks about 3 and a half years. Just recently I started doing Ruby work and I've been working in the DSL space for a while. I've been writing software for nearly 10 years.

The reason we are here is to talk about DSLs. We've been doing some pretty cool stuff, domain specific languages, can you tell us a little bit about why that's such a hot topic ?

About 8 or 9 months ago now we had a project that somebody came to us at ThoughtWorks and we estimated out as a pretty large project. And they were interested in getting it on the market a lot faster. So you actually came up with the idea to do the whole project as a DSL, and basically let the subject matter expert put together the domain logic.

Subject matter expert in this case being a financial analyst and in general we talked about domain specific languages, we mean that we're actually writing executable code, which i is readable and maintainable by someone like a financial analyst. Can you say if that's the case?

Zak: Sure, a domain specific language is a language that is specific to a given domain; generally speaking the domain in the business world would be something like a business subject matter expert that you are targeting for. That person can write code in this language and that code is turned into other executable code and then runs and does whatever it needs to do.

Why is there a connection between the topic of DSLs and Ruby?

The first one that we implemented was an internal DSL, Ruby because of its conciseness and no need of things such as parenthesis or semicolons, gives you a more friendly language for domain experts. It does still enforce Ruby syntax but when you have talented subject matter experts you can get around that by structuring a well written language.

Zak: There are a couple of things: one is the difference between internal and external DSL. Internal DSL implies that you are actually executing Ruby code, the language itself, the DSL would actually be Ruby code that's executable and then eval and then does what it needs to do versus external DSL where you deal more pre-processing to get it into the Ruby language and then be executed. So there's a lot of advantages of Ruby. It's a very expressive language, it's a lot more expressive and it's a dynamic language that tends to be more expressive than some of the more traditional type languages like Java and C#. It has open classes, it has an eval method that allows you to eval a string of code, with mixins and closers and blocks, making it a lot more expressive for business users.

But wouldn't business users be afraid of a programming in a programming language?

When I say it's expressive for business users I mean ultimately the developers have to design the language so that it's something that's easy for business users to use. And so having an expressive language allows the developer to do things more effectively and more concisely and present a language to the business users that is a lot simpler for them to use, so they can focus on the actual domain. So if it's a financial domain they can focus on typing up rules that are in their language and less on Ruby syntax.

Is there a language designer role?

Absolutely.

What does it take to be a good language designer?

Definitely takes iterative development first of all, you have to work closely with the subject matter experts defining the language one word at a time basically.

Can you expand on iterative in that context?

Jay: Sure, start out with the smaller subset of the problem and define a syntax that works well for the subject matter expert, and then try to work within the language to minimize required tokens, and basically the subject matter expert shouldn't even know that it's written in Ruby code, they should just read it as their language, that's the key to the domain specific languages. It's their language it's not just Ruby code.

Zak: You need to be able to speak the language with them and have a real eye for how to keep the language simple. If the language starts getting complicated it gets hard to implement for the developer and it gets hard to understand for the business user.

How did you get into that kind of role?

Jay: I just came on board to the project and set closely with the subject matter expert, learnt the domain. The thing that I like about domain specific languages is that you really learn the domain. You are not just creating an application, a feature at a time, that may or may not solve the problem. With domain specific languages you have to understand the domain and so you build off previous experiences, and put together a language that makes sense.

Zak: It's binding offpieces of the language one at a time.

This code that's in DSL, that's either written or at least the writing of it is assisted by a business user. How does that fit into Rails code or other programming code? Do you write a whole application in DSL?

Jay: You definitely don't write the whole application in DSL. The whole idea of the DSL is to solve a very small focus problem, you can have multiple DSLs within one application, that way it doesn't become a general purpose language, which would increase the complexity to the point where you might not even want to do it with the DSL.

Zak: You keep the DSL localized, like Jay said, you may want to have multiple DSLs per application. That maintenance of the DSL, putting in the rules of the DSL, we would use something like Rails to actually build up a web application around the DSL itself and maintain the DSL. The DSL i could be as simple as a text screen on a web page. And the business user sits down and types in rules, and saves them. And then it's the rest of the application that does the editing of the rules, saving, deleting, searching or even simulations. We can simulate rules for instance, that is a regular Rails application and has nothing to do with the DSL itself.

How is this approach compare to using a rules engine? A lot of people might be thinking "why go through all that trouble when there are off the shelf packages that you can get to let the business user define the business rules of their application"?

Zak: The quick response is 'have you ever sat down with a business user in front of a rules engine?', they are generally very complicated. You get splitting hairs in terms of what's a DSL and what's a rules engine and what are the things in between. Generally speaking you want a DSL to be very specific, you want it to be flexible so you can have multiple rules, but the idea is that it is much more focused on solving a specific problem, and that makes the DSL ultimately a lot easier for the end user. Rules engines are usually for solving more general problems. The more general your language becomes, not only the harder it is to build, the harder it is for the end user to use.

Jay: And rules engines aren't easily extendable, they can give you features 1 through 100 but you may need 101 and it's not always an option. So while you don't even need 50% of the features, the one that you really need isn't available.

Zak: The DSL sets a more customized approach, for example if you need 15 features you might be able to get 10 of those 15 from an off the shelf rules engine but it's those last 5 that are really going to provide the business differentiation for you.

And there aren't any off the shelf or open source libraries that do this sort of thing?

Jay: People are working on stuff, Intentional Software, JetBrains those guys are all working on IDEs, and I think the end goal that Martin has written about is an IDE where you can check the domain specific language files into source control, and we can all work together in the same IDE, but we're a ways off from that. So one of the areas of interest for Zak and I is basically web language work benches where you can build in. If people need different abilities; we've done that, versioning workflow, basically whatever the business needs to insure that domain specific languages are controlled by them instead of technology.

Zak: I think what you'll see is a movement towards work benches in general, and one of those is going to be a marriage of a rules engines and custom DSLs, coming closer together, and making that easier to do. But right now, there isn't a lot of good work in the gap between custom DSLs and off the shelf rules engines.

Let's talk a little bit about the challenges that we ran into with our first shot at doing this.

Jay: It was highly successful but we had a highly technical subject matter expert, so our DSL contained underscores, commas, hierarchy among the statements in the DSL.

Zak: Things which are specific to the Ruby language.

Jay: Correct, because it was an internal DSL. But the problem with that is that the technical user could get promoted anytime and then somebody else has to come in. So you have raised the bar to become successful with a domain specific language.

After I rolled out that project you kept going for a little while. And did you find that the technical user had to author new DSL code?

Jay: He was. The domain logic didn't grow too much after you left. We worked together with that user to build up the initial set. He has been able to successfully build off what we have created originally. Despite the fact that some people think that subject matter experts are afraid of blank screens, they don't know where to start. Again, we built iinto the web language work bench the ability to clone an existing rules set.

Zak: Part of that is that iterative development, having the business user in the development of the language and from the get-go. He gets more and more comfortable with it, and also starts to see the technical person's vision for where the language is going, and help design by making it grow.

Jay: By the end of the project he was defining the language we were just working on what the keyword meant, after he came up with it.

Zak: And that's what you want, it's the end-goal or a good bench mark of how successful your DSL is, is the business user being able to understand it and say "ok, I want a rule that looks like this". And that's the key. The second project was more focused on making the language simpler, keeping it simpler, for the business user, and we were able to do that by doing more external DSL stuff.

Would you say you tried to make it more natural?

Jay: Absolutely, as close a natural language than we can get.

Zak: Much more business natural language

Let's talk about that term. On your blog, and in our usage at ThoughtWorks you've been introducing this three letter acronym BNL.

Jay: Working with you we had a lot of conversations about DSLs and it's fantastic when you and I can talk about it because you know exactly what I'm interested in, and that is empowered subject matter experts. Rake, it's a cool DSL, I love it for myself personally, but it's not going to change the enterprise tomorrow. On the other hand, everybody has been looking for a way for subject matter experts to basically specify the business logic. I was driving down this path and I wanted to write about it a lot, talk about it a lot, and I felt like my conversations were dominated by people saying "your DSL it can have a hierarchy, it can be concise". And it's true, any DSL can, but when you are writing one for a subject matter expert you want them to be able to express something very simply. We are not talking about putting together DSL for graphic designers, who is possibly more technical. We are much more concerned with the guy over in marketing, who has no idea why an underscore or a comma, or any other symbol that he is not familiar with needs to be in the document.

Wouldn't a typical IT response to that be "you shouldn't be giving him a textual interface to begin with, you should be giving him a GUI"?

Jay: We ran into that. I don't think you're going to find a lot of people in marketing who are comfortable with a text box. So we basically started a demoing our system, a previously written system, and they are on board immediately. It's not really the IT side that pushes back, because they would love to have somebody else maintain that stuff, when you have a well functioning language workbench, your deployment process becomes the workflow. So IT no longer has to be involved. You do get an initial push back from the subject matter expert until you show them that it works.

Zak: There are a couple of important things there: when subject matter experts talk about their subject matter they don't actually talk about it in GUI terms, or "I want to click here and move something over here". They actually talk in English and say "I want to do XYZ if my profit score is a certain number". And they do talk in English, they are normal people, and actually I think that for them, once they see it working, the English, the text box and just writing things that are close to English is much more natural and more expressive, because they are not clicking around on a GUI The maintenance of it is a very interesting issue. It's a bit scary for technology people not so much that it's not a GUI, but they got some business user who they don't trust completely writing rules, and that's why the workflow becomes very important, having a QA process where rules go from scratch to simulation to actually being in production.

That's executable code right even though it's in the database

Sure.

We've blurred the distinction between data and executable code?

This is a lot more uncomfortable for technical people than it is for the business user.

Why is it uncomfortable?

Zak: Because there is a concern that if they are writing executable code the quality management and being able to actually put that to a QA process is harder and more difficult to do. In fact, it's not, we have shown that you can have a regular workflow process, go through the normal steps you would as long as you treat it like executable code.

Jay: It is executable code to us but again to the subject matter expert it's just putting code in a text box. To him, it's far less dangerous.

Zak: And for him it empowers him to make changes and simulate those changes a lot more quickly, rather than have to go through a hard process of changing actual code, recompiling and then putting it into a test environment, doing a lot more than the traditional deployment things that you would do with code, it's much quicker for him to test, write some rules, change things, simulate and skip results back. The feedback is really incredible.

Jay: This goes back to It departments actually being excited about this, because they no longer get requirements, have to implement those requirements have the subject matter expert verify those requirements now it goes to QA and then deployment. They are basically out of the whole process. And the subject matter expert is empowered to handle all that on his own. The quote that I love the best about the work that we've done is not that "Ruby makes me 10 times more productive", it's that the subject matter expert came to us and said "a change that used to take 2 or 3 days, now takes me 5 minutes, including running it in a test environment". This is another aspect of the web language workbench that we put together, is a test environment for them to test the language that we're putting together.

A change that used to take 2 to 3 days.

Zak: Instead of going to a developer and say "I have this change request and then I go and talk to the technical person, and say "ok allocate time to the developer to make the change", make the change that go through its natural QA process, and then you have the business user verify that it was the right change.

I'm actually challenge that quote a little bit, not the quote itself, but being familiar with the project, a change actually took more than 2 or 3 days, because they didn't deploy.

Zak: Sure, you are right, it's actually better than that, the benefit is even greater. Once you've got the change that these users verify the change and code is the correct thing, and then there is a whole issue: "when do I deploy this next?", and so forth. With the DSL, once you've got your test suits and your QA around your DSL it allows you to change go through your auditing process to make sure it's a valid change. And then it's a one step process and it's there, it's in production. The idea for the application we are working on is that they can change the rules every single day and push them into production every single day.

Before it was a 30 day window?

Something like that. This is huge business value for them.

Is there a tie-in between DSL techniques like we've been discussing and Agile?

I think there definitely is, it's going back to iterative development, there's no way you can sit down ahead of time and say "we need the application to do these following things, so here are these features". Let's take a step back to general purpose languages. There are a lot of people who say Agile doesn't work for general purpose languages and I'm comfortable with that because you do have to put a lot of features into a general purpose language before you can release it.

What do you have to put into that?

Jay: You have to put all the normal features into it but also make a breaking change whenever you want because there's a large code base out there. I'm talking about going from 1.0 to 2.0.

Like writing Ruby 2.0 using an XP team?

Jay: Like taking Java and saying "I like everything that's in here except now I want to make annotations to be just like C#", for whatever reason.

But you can do that?

Jay: Right, especially if you already put annotations out there the way that they were. However if you have programmers on a team and the subject matter expert who's writing the code, the programmer can go in and basically do all kinds of scripting, whatever they need to do, to change around the existing files and just add a new word if necessary. It becomes easy to make changes to the language incrementally.

There is a catch there isn't there? In the sense that even though it's not general purpose, you still have a body of scripts.

Sure but they are controlled. You have a body of scripts in one place telling everybody to go back and change their annotations it's just not a viable option.

You are saying this because of the scope?

Zak: If you're working within a specific domain, it's customized for that domain. I think the big issue with it is that coming up with the language is a collaboration between the developers and the business users. It doesn't make sense to sit down at a meeting and say "here's how the language will look, we are going to spend 8 hours to design a language to come out of this". I don't think that's how business users in particular think. They want to express an idea "ok, here, how do I go about doing this?" And you code it up and show them this is how it would work, and then you continue to do that. The iterative approach makes it a lot simpler, to build up a language naturally and get the business users much more comfortable using that language.

Jay: Again in learning the domain you may not need just one DSL. You may need multiple DSLs and that's hard to identify up front before you have any kind of application written. For example, for our last application we had two high level concepts that we were going to put into DSL with no problem, and we ran into a road block and we actually didn't know if we wanted to introduce another DSL or add another aspect to the existing DSL. It turned out that there was enough that was reusable, that we introduced this new feature into the existing DSL instead of breaking it in a part of its own. But we would have never known to do that up front, in fact the subject matter expert said "why don't we just do it this way, add these keywords to the language?". And we put it in and it worked.

The skeptics will say right now "it sounds like a fantastic silver bullet" but why is it not a silver bullet?

Jay: It's not a general purpose language, as soon as you need to do a lot more than you can accomplish, with a couple of concise statements, it's time to look at something, maybe a rules engine, maybe an external DSL that you are going to compile, or a complete parser.

Zak: The language has to be simple and concise as Jay said, if the language requires to be more general purpose or need full parsing abilities then you don't want to be doing an internal or even an external DSL, it's too hard to manage.

Jay: And if you need some kind of hierarchy in your documents, I think that's when you need to start to push away from it honestly because the domain experts don't think that way, they don't need to say "first I need to declare this, and then I need to do this". An Excel macro doesn't requiresomething to be set up first, it just runs. And once you need to do something complex enough to have multiple set up before you can execute than you need to abstract that out of the language which is possible, or you need to move to some other type of solution.

Zak: There are limitations on the scope and breadth of the problems that you want to solve. Some of that you can get around by having small more focused DSLs, multiple DSLs in an application. But some of it is there are certain things that don't fit the problem domain well.

What is the future for Ruby and DSLs?

Jay: Zak and I have been working on some stuff that is really cool, where it basically takes a DSL a business natural language specifically, converts it down from unparsable text into parsable text, and then just executes on that. We've had success creating inline statements that run quickly and also delegating back to databases for putting out the high intensity work. I think we are going to see more of that; it is going to become a lot more main stream.

What was the performance profile of our project?

Jay: The first application we put together they had a manual process it was all straight in SQL. It wasn't a bottle neck but it took 2-3 hours.

It was a legacy that we replaced.

Jay: Correct. In the end we had some multi threading going on, running multiple SQL statements at the same time - 6 minutes. It doesn't sound believable, somebody is probably not going to believe it but that's fine, it really did work that way.

Zak: As for the future I think there will be more work in this stuff. These toolkits to help you get customized DSLs outside of the box and a little bit more of marriage between an easy Ruby parser that goes with a nice syntax and the heart of DSLs and the parser. And I think the workbench will help a lot.

Jay: Absolutely, the workbench is going to mature faster than everything else. And it's going to feed the fire.

Zak: Besides maybe adoption of DSL, people started to use the DSLs more and more.

You guys are certainly contributing to that future

Hope so.