BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Metadata-Driven Design: Creating an User-Friendly Enterprise DSL

Metadata-Driven Design: Creating an User-Friendly Enterprise DSL

Bookmarks

 

 

Even though it goes without saying, an IT department and the users of their written software form a symbiotic relationship of mutual benefit, where the coordinated efforts of such a group can combine to create a gestalt of their collective work. In the best case scenario, the software developers and architects can truly comprehend and even predict the needs of their fellow stakeholders. In the end, they create more than just simple applications for the users; they are able to create essential tools that empower their clients and that help to keep those productive engines humming with life.

Often, we think of these tools as simply applications that assist with the daily maintenance of data, like a GUI with various CRUD capabilities or (for the more advanced user) a simple script that can be run to accomplish a discrete task. However, if we use our imaginations just a bit, we can create something unconventional yet powerful for our customers.

Sometimes, we don’t even have to imagine something completely new. Instead, it can be a transformation of something that has simply existed in the same way for years. Take, for example, something as mundane as a business rules engine.

For those unacquainted with the typical IT pipeline of data, there are usually a number of steps necessary in order to import data from an outside source into a production system:

These discrete steps can be split among several processes or can be aggregated into one, but for now, we’re concerned with the Record Processor module. This module will perform various steps on a presented record, validating and possibly manipulating the data before it will be submitted for permanent storage (like a database table). In particular, we are keenly interested in those steps that will process the data with specific business logic in mind. For example, we might want to save only product records of a particular type, and if we get one that mentions the type “Death Star”, we would want to reject it.

For a module like the File Parser, we could create a GUI application that loads a data file into a spreadsheet, where the user can map certain columns to properties in a data structure. In that way, the user has some control over the pipeline. However, for other modules (like the Record Processor), the users have little to no direct control over the back-end aspect of the architecture. In order to make any alterations within the Record Processor module, the users must follow the more traditional approach of providing specifications to a developer, who will then code those directives into it. The package that enforces the business rules (i.e., the rules that validate and/or manipulate business data) is usually a poignant example of such a case.

Of course, there have been helpful, relevant tools applicable to rules engines for decades, from legacy companies like ILOG and Pegasus Software. However, this form of implementation creates an end product that is not directly accessible to those who usually possess the actual business knowledge about the company’s proprietary data; there are no tools available that allow them to interface with these rules on their own.

What if we could create a language that could be easily understood by the layman but yet enforce those rules that apply to our business domain? What if we could abstract away some of the obvious yet necessary processing as implicitly applied in the language’s expressions, so that only the most important examinations need to be explicitly expressed? What if a snippet of this language could then be interpreted and performed at runtime, without the need for any recompilation or redeployment of the system? Can we really build such a maintainable domain-specific language (i.e., DSL) for a saavy but non-technical crowd? As shown in the brief example below, the answer to such a question is yes.

Even though some may argue that users generally should not be granted permission to alter such an important piece of functionality, there is usually a small sect within every company that is granted elevated permissions (i.e., superusers) since they possess a superior set of business knowledge. This group could become more productive if they could maintain the business rules on their own, without the required involvement of any developers. In order to provide them with such a capability, we could attempt to teach them a more elementary form of programming, or even introduce them to a popular scripting language.

If your general platform of choice is Java, you could pick JavaScript as the taught subject for these superusers, since the Nashorn JavaScript Interpreter has taken the stage with Java 81 and will become even more integrated into the JDK with time. So, let’s temporarily entertain the idea of teaching JavaScript to our superusers, so that they can write some business rules on their own. In the end, it would probably resemble something like the following snippet:

function ExecuteBusinessRules(IncomingBookData, CurrentBookData) {

  var Result = true;
  
  var IncPublishedDateString  = IncomingBookData.GetData('PublicationDate');
  var CurrPublishedDateString = CurrentBookData.GetData('PublicationDate');

  if (IncPublishedDateString.length > 0) {
    var IncPublishedDate = ConvertToJSDate(IncPublishedDateString);
    	      
    if (CurrPublishedDateString.length > 0) {
    
        var CurrPublishedDate = ConvertToJSDate(CurrPublishedDateString);

        if (CurrPublishedDate.getTime() < IncPublishedDate.getTime()) {
         
            var tmpDate = new Date(IncPublishedDate);
            tmpDate.setDate(tmpDate.getDate() - 4);

            var NewPublishedDateString = CovertToJsString(tmpDate);

            IncomingBookData.data['PublicationDate'] = NewPublishedDateString;
        }
    }
  }

  return Result;
}

So, let’s first assume that the Record Process module of our diagram is written in Java and utilizes the Nashorn package, ready to execute the ExecuteBusinessRules method in a specific JavaScript file. This snippet of JavaScript is a simple manipulation of dates. The function ExecuteBusinessRules takes two instances of a JavaScript object containing book data and then compares the PublicationDate property on both of them. (The Business Rules Engine itself would prepare and then provide the objects IncomingBookData and CurrentBookData to this executed script.) If the incoming date is older than the current date, it will subtract 4 days from the incoming date before submitting it to the Record Persistor module.

In order to accomplish such a simple task, we’ve already had to write nearly 100 lines of JavaScript, including the definitions for objects (like IncomingBookData) and functions (like ConvertToJSDate) not seen in the snippet. We would likely have more with proper error-checking. In addition to the developers’ support of these common libraries, your superusers would also need to become fairly proficient with JavaScript just to implement the body of the ExecuteBusinessRules method.

On top of the code’s elevated size, there is also no easy way for us to automatically validate this work on behalf of the superuser. They would then need to become skilled with tools like Firebug in order to debug their own code, and even then, there’s no straightforward option for checking other types of mistakes related to our system (i.e., misspelling the key name ‘PublicationDate’). Unfortunately, this choice also creates an opportunity for danger, as the superuser could mistakenly (or intentionally) create a piece of JavaScript with disastrous effects upon the system. In summary, various pitfalls and injuries can await stakeholders who pursue this particular route.

Instead, if we desire to truly empower the superusers and to create an analog of a sandbox, we should look to create a domain-specific language that is simple yet powerful enough to become the grammar of our pidgin for business rules. Consequently, I would nominate the use of metadata-driven design for such a project. What is metadata-driven design (i.e., MDD)? As mentioned in an article2 about building a MDD architecture, MDD is a design method built upon domain-driven design, with which one can create a system (or subsystem) with the intent of its metadata determining both the data model and the functionality. Through the use of amorphous data structures, we can create a malleable foundation on which we can build other software layers. By simply adding new rows to the metadata, we can then enlighten the system so that it can become aware of new data points and how to deal with them.

An implementation of such a system inherently possesses a lexicon that provides useful terms for a domain, and in some cases, the metadata’s implied functionality can serve as a primitive sort of grammar. As interesting as this grammar might be, we will ignore it for now; this mentioned lexicon is our primary interest, since it can provide us with the basis for our own homegrown DSL solution.

By bringing MDD to the forefront, though, we can remove many of the obstacles found when the superuser adopts a scripting language like JavaScript. Since MDD is simply an incremental version of domain-driven design (DDD), we have an established vocabulary of terms that describe the various data properties of our desired system, and we can then use them in order to create a DSL for a custom rules engine. (In this particular case, we will not ponder over any specific implementation of such an engine, like whether or not it should utilize the Rete algorithm. Instead, we will simply focus on the language file that feeds the engine and the general expectations of its execution.)

If you’re unfamiliar with the ideas behind a DSL, I recommend viewing Martin Fowler’s dissertation3, where he describes DSLs as “limited forms of computer language designed for a specific class of problems”. In the seminal lecture, Mr. Fowler describes how XML configuration data can be utilized as a simple DSL for Java programs and frameworks. (Of course, this method of utilizing XML can now be found in various Java frameworks of today, with Spring and Struts being just two examples.) In doing so, he mentions several advantages to using such a DSL, including the lack of requiring recompilation for many cases and its approachable usability for normal business users.

Both of these advantages fall on our wish list. The lack of any needed recompilation makes such an option extremely compatible with a MDD implementation, since one of MDD’s primary goals is for code to be rarely altered. Second, we strongly desire approachable usability, since it’s our intended goal for the business rules to be easily written (especially with the help of a simple user-friendly tool). If modern Java programming overwhelmingly vouches for this choice, why can’t we also leverage such a strategy in order to create a robust solution for our business-savvy but non-technical stakeholders?

So, let’s temporarily revisit an artifact from my initial article about MDD, specifically the spreadsheet that describes the data properties (i.e., Attributes) for a proposed system:

Here, we can observe the metadata that describes the important qualities of our data properties. Since this schema enables us to provide dynamic functionality to our system by simply altering or adding new rows of metadata, we can enable our superusers by exposing the power of this metadata through the creation of a simple DSL. Again, as Mr. Fowler elaborated in his talk, we will also employ XML due to its inherent structure and due to the plethora of widely available tools for parsing the format. Now that we have both a primitive lexicon in our metadata and grammar in the form of XML, we can establish the fundamentals of our new language with a first iteration. We will choose to port our previous JavaScript example with the PublicationDate as the basis for a prototype:

In contrast to our JavaScript version, this snippet is much smaller in size and more straightforward, omitting the various details that are necessary when employing most languages (like validating the date format). Of course, these details and the general execution of this RuleTree must be implemented somewhere, and in fact, they will exist as code within the engine itself. (Even though the total implementation of such a rules engine and its associated interpreter are beyond the scope of this article, we’ll provide some brief insight into such work later on.)

Much like with the JavaScript example, the Business Rules Engine will have an exposed Java interface with an ExecuteBusinessRules method to be called by the Record Process module, where the Current and Incoming objects will be provided. This code, though, will be hidden from the business superuser, who does not need to know (and likely does not care) about such idiosyncrasies.

More important than a reduction in size, we have gained a number of advantages from the perspective of someone who must maintain and enhance this architecture. First, by using XML to structure our business rules, we now have an easily verifiable grammar that we can safely impose upon the editing superuser. If they need assistance with correcting any mistakes, there are many applications (both free and proprietary) that can help with both debugging and repairing most syntactical mistakes regarding the layout of the XML. Two, by using the Attribute names previously mentioned in our spreadsheet, we can integrate our coded rules engine into the cited MDD architecture and then use its metadata when validating and invoking these business rules.

Of course, we will still need to create a rudimentary lexer of some kind in order to understand the expressions contained within tags like <eval>, and these basic expressions (like “IS BEFORE”) are useful tools that can be utilized and developed fairly easily. However, the true power comes from the accessible metadata. Much like how we can validate the rules’ structure by examining its XML, we can now validate and evaluate the various expressions by examining the mentioned Attribute in the context of its use.

One, while the DSL file is being created, we can now create simple validation tools for the editing superuser, so that we can ensure certain values exist as their designated type. For example, the Attribute ‘PublicationDate’ can be used with the operator ‘IS BEFORE’, but if the same operator were used with the Attribute ‘PriceValue’, our validation tool could warn the superuser of an invalid action being invoked on an Attribute of the type ‘double’. Two, inside the Java code of our Business Rules Engine, the code for the ‘IS BEFORE’ expression can now appropriately and safely compare dates, encompassing such an operation with all of the needed complementary code (like error-checking).

This particular example shows how MDD can execute simple business rules that apply to actual data, but we can create even more functionality if we incorporate other dimensions of our metadata into the DSL. In the case of the cited MDD architecture from the February 2015 article, the design had included an auditing layer; this layer was composed of metadata structures called Fields that gave us the use of permissions and locks. With some carefully planned additions to our project, we can now write other types of expressions in our DSL, ones that further assess the state of the product data:

Now, in addition to qualifying and acting upon the data itself, we can now do the same with any supplemental information that we might possess regarding the record. As shown in the example above, we can now evaluate whether or not the current PublicationDate of a product is locked (i.e., not accessible for alteration). Any subsequent iterations of the cited MDD architecture can induct additional dimensions to its schema’s metadata, and every introduced dimension can now be an inclusion within our DSL.

So, how do we build such a Business Rules Engine? Surely that must be some sort of herculean effort? Actually, it’s not as bad as one might think. Again, we won’t venture too deep into any given implementation, but we’ll present the basic ideas. First, as shown in the diagram at the beginning, the DSL Rules Interpreter module will need to read the rules file and then create a valid representation of its contents. In this case, we’ll load all of the information into a class called RuleTree. This RuleTree will be composed of logical blocks called RuleSets, and the basic definition of such a RuleTree would be the following:

Using our DSL example above, a RuleTree produced using that snippet would contain two instances of a RuleSet: one with its Description set to “PublicationDate Rules” (i.e., the <if> tag) and its child (i.e., the <leaf> tag). If we needed some level of complexity, we can create layers within our RuleTree by nesting RuleSets. After producing our RuleTree, we would then pass that structure along to the Business Rules Engine, and applying it against two business objects (or product records, in this case), that module would subsequently create a RuleTreeReport with detailed information about the results from invoking the RuleTree (including a status code that indicates successful processing or the occurrence of a critical error). Even though these succinct steps merely summarize the work necessary, the important message is that with a little less than a couple thousand lines of Java code, you can have your own working rules engine.

Of course, the amalgamation of XML and some elementary, custom expressions (like “IS BEFORE”) could be just the first iteration of a DSL based on MDD. Even though such a subject is beyond the scope of this article, we could create other further enhancements by having expressions with useful features that are standard (like operators). It would be possible to build such functionality by simply choosing a popular language as our foundation (JavaScript, Perl, Python, etc.) and then constructing another layer on top tied to our metadata. We could even port our custom operators like “IS BEFORE” and use metadata to bind its execution to a specified library or a JIT block of code, so that it could be dynamic as well! As you can imagine from here, there are a number of different ways to refining our DSL, as long as we have MDD as the basis for our approach.

About the Author

Aaron Kendall is a software engineer in New York City, with nearly 20 years of experience in the design and implementation of enterprise data systems. After beginning as a developer of device drivers and professional software, he became passionate about software design and architecture. He has created innovative business solutions using a variety of platforms and languages, as well as numerous freelance software projects that range from open source packages to game design and mobile apps. If you would like to read more about his work, you are encouraged to visit LinkedIn and his blog.

 


1 Zeigermann, Oliver (15 April 2014) Nashorn - The Combined Power of Java and JavaScript in JDK 8 InfoQ

2 Kendall, Aaron (19 February 2015) Metadata Driven Design - An Agile Bridge Between Design and Development InfoQ

3 Fowler, Martin (31 October 2006) Introduction to Domain Specific Languages InfoQ

 

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • XML?

    by Tom Flaherty,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    XML is a step backwards.

  • Eclipse XText

    by Michele Mauro,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    What about XText and all the work on parsers and tooling made in that project?

  • Re: Eclipse XText

    by Aaron Kendall,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    That's an interesting idea! I've only read about the XText project, but if it could be integrated with a proprietary metadata format (like the one mentioned in the article), then that could make an excellent and elegant solution!

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT