Orchestrating RESTful Services With Mule ESB And Groovy

Posted by David Dossot on Aug 25, 2009

Sections

Architecture & Design,

Development,

Enterprise Architecture

Topics

Java ,

REST ,

SOA

Tags

Mule ,

Groovy

Over the past couple of years, the REST style of software architecture has gained in popularity, mainly because it typically gets reified in systems that require less moving parts, exhibit loose coupling and are more resilient.

Having more REST resources available in the enterprise landscape increases the chance that orchestrating them in some way will be needed. For example, a business activity will typically consist of creation of a resource followed by subsequent lookups and creations of other resources.

In essence, interacting with RESTful services is pretty simple: we need to form and send proper requests (headers and entity) and analyze the responses we get back (headers and entity again). For this, no special tools or frameworks are needed, besides a good HTTP client library. Besides, as the different entities involved in RESTful interactions are defined by so-called micro-formats, the capacity to parse or output these entities easily can also be very helpful.

Orchestrating the interactions with several resources becomes slightly more involved. We need to define the orchestration, handle errors and retries properly and our system must behave gracefully under load. As an integration framework Mule provides all of this (and much more).

Let's consider an example, a merchant system where the processing of a new order implies the following orchestration:

1. Create a new order by posting an XML entity to a service.
2. Lookup the newly created order resource and extract confirmation information from it.
3. Instruct an email gateway to send a confirmation message to the customer, based on these confirmation information.

In this article, we will detail the interactions for each of these steps and will consider what particular Mule moving parts and Groovy features we have used to achieve such an interaction.

The overall orchestration will consist in a chain of Mule services connected with specific routers, filters and in-memory message queues (aka VM queues). You can learn more about routing messages in Mule by reading reading a recent InfoQ article on this subject.

REST support in Mule

Mule provides a simple, yet extremely powerful way to interact with RESTFul services using Mule RESTPack.

The Mule RESTPack provides a comprehensive set of connectors and guidance that help you create new RESTful applications. As of this writing, this pack offers three transports based on popular REST-related frameworks: Abdera, Jersey and Restlet. This is very handy for exposing new resources but what about integrating existing REST resources?

The good news is that with Mule's stock HTTP transport spiced up with Groovy support from Mule's standard scripting module, you get all you need for successfully interacting with RESTful services.

 

POSTing with Mule

Let's get started with the first interaction. An order is created by HTTP-posting an XML entity to a specific resource, as shown in these excerpts:

POST /orders HTTP 1.1
...
<order xmlns='urn:acme:order:3:1'>
   <customerId>123456</customerId>
   <productId>P987C</productId>
   <quantity>2</quantity>
</order>

To which the server replies in case of success:

201 Created 
Location: http://acme.com/order/O13579 
... 
<order id='O13579' />

This interaction is achieved in Mule with a simple HTTP outbound endpoint. Note that the interaction itself is triggered by sending a map containing the required values (customer and product IDs, quantity) to the order creation service. So what does this service look like? Here it is:

<service name="OrderCreationService"> 
  <inbound> 
    <inbound-endpoint ref="OrderCreationQueue" /> 
  </inbound> 
  <outbound> 
    <chaining-router> 
        <http:outbound-endpoint synchronous="true" 
              responseTimeout="15" method="POST" 
              host="${acme.order.hostname}" 
              port="${acme.order.port}" path="orders" 
              user="${acme.order.username}" 
              password="${acme.order.password}" 
              contentType="application/vnd.acme+xml" encoding="UTF-8"> 
              <transformers> 
                <transformer ref="OrderMapToMicroformat" /> 
              </transformers> 
              <response-transformers> 
                  <message-properties-transformer> 
                        <add-message-property key="OrderPlaced" 
                            value="#[groovy:message.getStringProperty('http.status','ERR')=='201']" /> 
                        <add-message-property 
                            key="OrderResourceLocation" 
                            value="#[groovy:message.getStringProperty('Location','')]" /> 
                  </message-properties-transformer> 
                  <object-to-string-transformer /> 
              </response-transformers> 
         </http:outbound-endpoint> 
         <outbound-endpoint ref="OrderCreationResultQueue" /> 
   </chaining-router>     
  </outbound> 
</service>

That's rather dense XML so let's look closer at what we've got here:

• A message is received in a channel named OrderCreationQueue (which can be a VM queue or a JMS queue),
• The received message is directly passed to a router that will chain the result of the HTTP POST to another service in charge of analyzing the outcome of the call via a channel named OrderCreationResultQueue (an asynchronous VM queue).
• The HTTP post is performed using a standard outbound endpoint on Groovy transformers steroids:
  • The required order microformat is created with a specific transformer detailed in the next section.
  • The result code is extracted and compared to the expected value thanks to a tiny script. We perform the comparison here in order to isolate subsequent services from pure HTTP concerns: the boolean OrderPlaced property we create is truly neutral and bears a name that is relevant to the ongoing orchestration.
  • Similarly, we copy the Location header under the more contextually meaningful name of OrderResourceLocation. While doing so, note how we handle the fact that this header may be missing (in case of failure) and default to an empty string to avoid adding a null property to the message (which makes Mule kick back at you).
• We use an object-to-string-transformer to "detach" the HTTP response which comes back as a stream. Thanks to it, the stream gets fully consumed and its content is transformed to a string, using the encoding relevant to the HTTP exchange. When this stream gets closed, the HTTP connection gets released: we don't want to keep it open, waiting for the subsequent service to pick-up the response message in the OrderCreationResultQueue.

Groovy's MarkupBuilder Goodness

The heavy lifting in this service is really done by the OrderMapToMicroformat transformer, itself powered by Groovy's MarkupBuilder. The MarkupBuilder API offers a natural way to produce XML entities compliant to our particular microformat:

<scripting:transformer name="OrderMapToMicroformat"> 
   <scripting:script engine="groovy"> <![CDATA[ 
        def writer = new StringWriter() 
        def xml = new groovy.xml.MarkupBuilder(writer) 
        xml.order(xmlns: 'urn:acme:order:3:1') { 
          customerId(payload.clientId) 
          productId(payload.productCode) 
          quantity(payload.quantity) 
        } 
        result = writer.toString() ]]> 
    </scripting:script> 
</scripting:transformer>

Notice how the values from the map payload are used to populate the XML elements: some of the naming convention mismatches are resolved here (for example, clientId becomes customerId).

As desired, this transformer produces this kind of output:

<order xmlns='urn:acme:order:3:1'> 
   <customerId>123456</customerId> 
   <productId>P987C</productId> 
   <quantity>2</quantity> 
</order> 

 And, alongside the right content type, it is all what the order RESTful service was expecting for creating a new resource.

 Analyze this

Let's now look at the service in charge of asynchronously analyzing the outcome of the order creation and deciding if it is worth going further with the orchestration:

<service name="OrderCreationResultProcessor"> 
  <inbound> 
    <inbound-endpoint ref="OrderCreationResultQueue" /> 
     <selective-consumer-router> 
        <message-property-filter pattern="OrderPlaced=true" /> 
     </selective-consumer-router> 
     <logging-catch-all-strategy /> 
  </inbound> 
  <outbound> 
    <pass-through-router> 
      <outbound-endpoint ref="SuccessfulOrderQueue" /> 
    </pass-through-router> 
  </outbound> 
</service> 

We use a selective consumer to accept only messages that bears a header confirming that the order was successfully placed. If this header is true, we then route, via an in-memory queue named SuccessfulOrderQueue, the message to a third and final service dedicated to handle successful order creation messages. Note that, in this example, we simply log non-compliant messages but in reality we would send error messages to a dedicated queue either for later forensics or for immediate feedback to the application.

GETting with Mule

The final service that composes our orchestration takes care of HTTP-getting the newly created order that contains extra values needed to form a valid message for the email gateway. Here is a sample interaction:

GET /order/O13579 HTTP 1.1 
200 OK 
Content-Type: application/vnd.acme+xml 
... 
<order xmlns='urn:acme:order:3:1'> 
  <customerId>123456</customerId> 
  <productId>P987C</productId> 
  <quantity>2</quantity> 
  <customerEmail>foo@bar.baz</customerEmail> 
  <estimatedShipping>2009-31-12T00:00:00Z</estimatedShipping>
</order>

The good news is that Mule's stock HTTP transport contains a component (named rest-servicecomponent) that facilitates the interacting with REST resources from within a service. Thanks to this component, we do not need to chain the result of the GET operation to a subsequent service but can perform everything within a single service. Moreover, it supports the usage of expressions in its configuration, which enables connectivity with dynamically built URLs.

Here is how this last service looks like:

<service name="SuccessfulOrderProcessor"> 
<inbound> 
  <inbound-endpoint ref="SuccessfulOrderQueue" /> 
</inbound> 
<http:rest-service-component httpMethod="GET" 
   serviceUrl="#[header:OrderResourceLocation]" /> 
<outbound> 
  <pass-through-router> 
    <outbound-endpoint ref="EmailGatewayQueue"> 
      <transformers> 
       <object-to-string-transformer /> 
       <transformer ref="OrderMicroformatToEmailMap" /> 
      </transformers> 
    </outbound-endpoint> 
  </pass-through-router> 
</outbound> 
</service>

After receiving a successful order message, this service uses the REST service component to generate an HTTP GET request to the URL that has been previously stored in a property (aka header) named OrderResourceLocation. Note how we use the Mule expression evaluation framework (with the #[...] syntax) to inject a dynamic URL in this component.

Right before passing the response of the GET request to the service in charge of communicating with the e-mail gateway, we perform two transformations on the XML order entity:

• We "detach" the response entity, as discussed before.
• We use a transformer to parse the XML entity and build the map expected by the next service. For that, we rely again on Groovy's power.

Groovy's XmlSlurper Happiness

Groovy's XmlSlurper is a dream tool for parsing XML microformats, thanks to its DSL-like API.

Here is what is inside the OrderMicroformatToEmailMap transformer:

<scripting:transformer name="OrderMicroformatToEmailMap"> 
  <scripting:script engine="groovy"><![CDATA[ 
    def order = new XmlSlurper().parseText(payload)
				.declareNamespace(acme: 'urn:acme:order:3:1') 
    result = [emailId:'OrderConfirmation', 
		emailAddress:order.'acme:customerEmail'.text(), 
		orderId:order.@id.text()] 
   ]]> 
  </scripting:script> 
</scripting:transformer>

Yes, that's two lines of Groovy and we have a namespace-aware XML parser and a map builder. It's almost too good to be true! But that's all we needed to create the map that the email gateway service is expecting.

Finally RESTing

In this article, we have followed a pretty much pre-defined orchestration path, with the new order resource URL being the only dynamic bit. It's not hard to imagine how one could leverage the demonstrated transformers and expressions to support more dynamic interactions, as you could get if you decide to walk the HATEOAS path. If that would be the case, other Mule's smart router would become handy, like the one allowing you to create idempotent receivers. You've seen how Groovy's capacities and conciseness combined with Mule's communicating abilities greatly simplify interacting with RESTfull services. Both Mule's expressions and tiny Groovy scripts are efficient ways to dynamize your Mule configuration. As an added bonus, the fact we use asynchronous VM queues to chain the different steps of our orchestration allows our solution to gracefully degrade under peak loads, thanks to the intrinsic SEDA architecture of Mule.

With all these power tools in hand, integrating REST resources will surely not make you restless.

Enjoy!

The full configuration and related test resources are available as a standalone Maven project that you can get here: http://dossot.net/datastore/mule-groovy-rest.tar.gz

1. Back to top

   Clarification: No RESTPack

   by David Dossot

   The title of this article is misleading: Mule's RESTpack hasn't been used here.
   
   D.

   Reply

2. Back to top

   Re: Clarification: No RESTPack

   by Dilip Krishnan

   The title has now been fixed

   Reply

3. Back to top

   Full Configuration download link broken

   by kevin liu

   Hi, I really would like to see the full configs. If you can please fix the download link: dossot.net/datastore/mule-groovy-rest.tar.gz
   
   Thank you

   Reply

4. Back to top

   Re: Full Configuration download link broken

   by David Dossot

   Sorry about that, the link is now back online.
   
   D.

   Reply

5. Back to top

   Re: Full Configuration download link broken

   by Naresh Podi

   The article is good. But I am looking for the following details.
   
   "I have an external REST webservice, which takes three headers Content-Type, Content-Length and Authorization."
   
   Now I would like to understand
   1. How to configure the external REST webservice in Mule.
   2. How to get the Header's information. so that I can use those headers for an another purpose. that means I should be able to access those headers in a Custom class.
   3. If we call Mule ESB's url, which class will be having all the request information.

   Reply

• Older >