BT

Getting Started With Spring Integration

Posted by Joshua Long on Jan 20, 2009 |

Why Spring Integration

Spring Integration is an API from the creators of the Spring Framework that's geared towards Enterprise Application Integration (EAI). When it comes to integration, there is no shortage of “solutions”: hard coded Java clients, other ESBs, and more traditional application integration technologies like message queues. Spring Integration offers improvements on every case, sometimes in dramatic ways! Spring Integration is very lightweight, and easily tested. It has almost no barrier to entry, and is conceptually going to be simpler than any “I'll just write it myself” solution. In the long run, it's much more flexible and resilient. The instinct to just write it yourself can be suppressed. Spring Integration works with standard technologies like EJB, RMI, and JMS, and enhances them by allowing you to model complex solutions in one place. This simplifies the use of those technologies considerably. Because Spring Integration is so lightweight (you deploy the Spring Integration server with your application; you don't deploy your application to Spring Integration) and so focused on the development life cycle (XML schemas to facilitate configuration, POJO-friendly APIs, and strong integration with the Spring Framework and JEE), you'll find Spring Integration a better fit than a lot of other ESBs.

Spring Integration is powerful in its own right, but it can't be denied that it benefits from the strength of the Spring framework. The configuration format, for example, is nothing more than Spring schemas, which in turn abstract bean instantiation for you. There's very little magic to using Spring Integration, and you could conceivably write a main(String [] args) method that did everything your XML configuration was doing. A lot of the support for RPC and messaging that is available in Spring Integration is built on top of the Spring framework's support. Everything in your Spring integration configuration file is still a standard Spring application context, and benefits from the same dependency injection and aspect runtime available to regular Spring beans. With Spring Integration, the application context is the bus. This enables solutions that depend on application context events, for example. This the other reason there's no free standing “runtime,” as the bus exists as long as the context is available.

Background

Enterprise Application Integration (EAI) is an application of technology defined as the integration of data and services between applications. The problems it addresses are boundless and the solutions almost as myriad. Engineers have been building these solutions for decades. Only in recent memory have we identified the best practices of the discipline and categorized them.

The patterns of modern day EAI are usually attributed to Enterprise Integration Patterns [1], by Gregor Hohpe, et. al., that categorizes and explains many of the integration patterns common to integration solutions. Hohpe, et. al., list four types of integration styles:

  1. File Transfer: two systems produce files whose payload is the message to be processed by the other system. An example of this is polling a directory or an FTP directory for a file and processing it.
  2. Shared Database: two systems query the same database for data to be passed. An example of this is when you have two EARs deployed whose entity classes (JPA, Hibernate, etc.) shared common tables.
  3. Remote Procedure Call: each of two systems expose services that the other can call. An example of this is EJB services, or SOAP and REST services.
  4. Messaging: two systems connect to a common messaging system and exchange data and invoke behavior using messages. An example of this is the familiar hub-and-spoke JMS architecture.

These styles are disparate because no one solution will work all the time. This has given rise to a whole field of middleware that seek to enable solutions based on these patterns, typically called an Enterprise Service Bus (ESB). An ESB is the ultimate middle-man: it knows how to talk all languages, over all protocols, and mediate messages being passed.

These architectural styles are different, each with its own unique advantages. Often, the standards in JEE (or any other modern day development platform, frankly) fall short, and don't provide solutions when integration with other systems. It's surprising - given how many projects are maintenance projects – how little of the technology toolkit in a shiny new platform is devoted to leveraging old services, or functionality.

JEE - and Spring after it - made great strides in simplifying the enterprise programming model. JEE standardized and made commodities of solutions to common enterprise problems: database access, remote procedure invocation, transactions, authentication, directory services, etc. EAI solutions have no direct support in JEE, aside from the basics: RPC, messaging.

JMS, REST and SOAP are supposed to be platform agnostic, but assume a homogeneous message protocol. This is impossible when – for example - the solution requires integrating with an old mainframe application whose input and output are batch files deposited on some FTP endpoint.

Succinctly: modern day middleware handles the normal cases well, and falls short at handling exotic use cases. Take the subscription process for most bulletin boards or email lists. Typically, the user sends an email to some application that eventually receives the email, parses it for the word “subscribe” and extracts the sending email, and then sends a response back after enrolling the user in the mailing list. The first instinct might be to build some sort of scheduled application on CRON or via Quartz that polls for email. Or, perhaps, to check an FTP for batch files, which you then consume. This will quickly become tedious, and fragile.

The complexity rises precipitously after that. As time moves on - and an application becomes more important - the burden of integration with trading partners, other applications, or other platforms becomes more expensive. Each integration each adds a new point-to-point channel to the system that has to be maintained. Eventually, the converging channels to integrate every endpoint with every other endpoint becomes an untenable mess, “spaghetti architecture.”

SpringSource's Spring Integration [2] improves upon typical ESBs by simplifying the programming paradigm.

How to Consolidate and Clean up the Architecture

There are many patterns for Enterprise Application Integration, and just as many protocols that need to be dealt with (endpoints). Spring Integration provides the ability to model ESB - style solutions using the same idioms and conveniences that you're accustomed to from the Spring framework. ESBs provide more than just the ability to model messaging solutions over disparate technologies/protocols, however.

Services of an ESB

There are several things that are common to most ESBs. Some of the most important ones are:

  1. Routing: the ability to route messages on conditional logic, or configured routing rules.
  2. Messaging: transforming the message's payload from one type to another – is crucial to any kind of complex solution. In messaging, the canonical data model [3] pattern requires that systems vend a common format. Processing messages is also another important part of an ESB, naturally.
  3. Mediation: adapters and service mapping.
  4. Complex Event Processing (CEP): the ability to process events on the bus as an aggregate by correlation.
  5. Invocation: This one should be the most obvious. An ESB needs to be able to consume and provide services.

Typically an ESB will contain mechanisms for logging, auditing, authentication (security), and management, etc, above and beyond the services listed above.

An ESB provides a lot of value if your requirements are even mildly complex. It's worth contrasting what you've already got in the JEE platform with what an ESB would give you. The following table compares some of the common use cases, suited to ESBs and the alternate, JEE-ish solution.

 

ESB

Traditional JEE Middleware

Message Queues

Configurable by XML, supports all the usual Messaging patterns

Not too involved if you're using the Spring JMS support, for example.

RPC

Typically can consume, vend, and abstract most RPC services

Equally infinite possibilities, but no standardization of approach.

Integrate Heterogeneous Systems

ESBs are designed to isolate the trickiness in consuming, normalizing, and then vending disparate messages.

JEE supports a few use cases quite well. The solutions do tend to get complex, quickly, however. SOAP messages to FTP? Batch file records to EJB calls? JEE offers half of the solution, in each.

Security

Good support

Good support

Legacy mainframe systems

Strong support for all manners of interop supported by JEE and more, including batch files

Beyond CORBA, JEE doesn't offer much support for older systems unless those systems have been front-ended with something like SOAP

Flexible routing

Routing decisions delayed as late as possible across all supported components

JMS, EJB, etc, all have different configuration specifying availability and routing. Hard to get a “birds eye view.”

Ease of Integration With Non Standard Requirment

Relatively easy, particularly with Spring Integration as every thing's POJO and you're not integrating with an application server; conversely: your solution won't be standard across all ESBs

Requires in depth knowledge of JCA [4], or a system like BEA's Tuxedo [5]. Solution will be standard (results may vary) across all JEE servers.

 

Popular Solutions

Spring Integration is new, and does not come unchallenged. Mule [6] is a very popular ESB that deserves a close look. Mule seems to have the most mind share and - in terms of flexibility of the solutions - has most impressive repertoire. The open source extensions available through MuleForge [7] make it a compelling option for almost every problem. It is a standard server: solutions are deployed to it and then run. The Maven plug in facilitates the development lifecycle.

ServiceMix [8] also has a lot of mind share. ServiceMix was originally based on the Java Business Integration (JBI; JSR 208 [9]) specification. JBI is a JCP specification for building ESBs. ServiceMix is not quite as flexible as Mule because of its JBI ancestry, but that is changing. The container is being moved towards an OSGI infrastructure.

I haven't exhausted all the other worthy ESBs out there, but don't let that dissuade you from learning more about them if you get the chance. Some are quite worthy, and worth investigating.

Spring Integration has a strong showing in functionality out of the box, and is particularly easy to work with. The development use case is very compelling: if you've become spoiled with POJO and test-friendly frameworks in recent years then this framework might be right up your alley. You can, if you like, use interfaces or standard framework classes, or you can code entirely to POJOs and your domain model, or, you can work with a mixture of both. I've chosen to use some framework classes in the solution in this article to make apparent what's happening and to provide consistency. Sometimes too much magic -while productive to the initiated - is confusing.

Getting Started

A Crash Course On Enterprise Application Integration

We're going to demonstrate the development cycle by actually building a very trivial application. The example will be simple enough to understand, but not frustratingly contrived. Plus, it's kind of a cool tool to have! The requirement: permit emailing a blog to a known address which then publishes that blog for you. There are a few good reasons for doing this.

Blogger ([10]Google's blog service) already has this. However, I run Apache Roller ([11]) on my blog, so I could use such a solution. The inertia required for writing a blog is largely a side effect of not having the time to get into “publishing” mode. Roller's software compels the user to log in and compose a blog entry. Doing this is messy, especially if you have to fight the WYIWYG editor. The second, best reason for building this solution is that it provides a simple integration with as few moving parts as possible. We can reasonably hope to dissect this in this article. It's contrived, but practical.

Building integrations is easy, and your best “IDE” is a piece of paper and pencil. Diagrams can be drawn in many tools. It's a simple matter of translating the diagram into your favorite ESB's configuration format, or tool. ESBs use the same jargon. Knowing the jargon is more important than knowing any one tool or ESB.

Let's review some ESB 101 definitions, just as a refresher. A message is the payload bound for some endpoint. An endpoint is the end of a channel. A channel is a named connection of two endpoints. Often, messages will come in from a messaging system to be delivered to an application that doesn't understand that messaging system. Or, the same could happen the other way around: an application might vend data but not understand a messaging system. For these problems, a channel adapter is needed.

And, that's it! Those are the terms you'll need to know to talk about a solution. Other terms are variations or specifications on top of these terms, or they relate to patterns of integration, and not to integration itself.

Overview of Spring Integration

Spring Integration applications are simple Java programs that are configured using Spring's schemas. You could configure everything with regular Spring configuration - and no schemas - if you were so inclined. The schemas make things easier, in much the same way that the transactional functionality with aspects in Spring are easier to configure using schemas. Spring Integration provides convenient schema for general concepts (the integration namespace), as well as adapter specific configuration, like those for email, or file types.

Spring Integration applications deal with the notion of a message that's traveling through channels. The message starts its life at an endpoint, typically greeted by an adapter. As the message moves through the processing pipeline, it may be transformed, routed to other channels, split up, responded to, or aborted and sent on a dead message channel, all inside the bus. If you program against the Spring Integration interfaces - we will to a large extent to keep things as consistent and as clear as possible -then you will deal with a Message object, which looks something like figure 1.

image1

The message class is a wrapper around the payload of the message being processed. It's a convenient place to get access to the payload itself - which can then be cast - as well as the message headers - which you can inspect or change. Spring Integration doesn't require that you use the Message interface. Your application might expose methods expecting parameters of the same type as your message payload. For example, messages coming from a file adapter (an adapter that can send messages from the file system that have been picked up off a file system) may be routed as an instance of java.io.File.

Let's look at a sample integration, one which takes emails delivered to an email address, transforms them and then publishes them to a blog.

The configuration in our example is in src/main/resources/integrationsContext.xml. The entire source code can be downloaded from here. The file looks normal at the outset. At the top of the XML is a bean whose job it is to macro replace variables in a properties file into this XML configuration script. Then, an import of another spring file that contains simple services that we want to use in this integration. Nothing too interesting.

Then starts the interesting bits, 4 beans are declared in succession: the service activator (newBlogPublishingServiceActivator), a transformer (emailToBlogTransformer), a handler (outboundBlogPostHandler) and a filter (emailsInFilterAgainstWhitelist). The beans will be used later in the configuration.

The first stanza of actual configuration is the poller element, which is the default poller for the document. A poller is exactly what it sounds like: it's the mechanism that polls different endpoints for changes and, if some thing's perceived to have changed, let's the adapter react to it as an event. We configure one as a default poller for the entire Spring Integration configuration because it's easier than repeating it. See figure 2.

image2

The next three elements are the declarations of three channels. They mean nothing – they're just named channels. A channel without endpoints or adapters is useless. See figure 3.

image3

The next line configures the email adapter. The email adapter looks for email coming into the system and puts them on the channel named emailsIn, which we declared above. You know that email systems don't broadcast events when an email comes in. While there are exceptions (IMAP IDLE, for example, which Spring Integration also supports) normally you need something to poll the email account for new emails. If there are new messages, they get read one at a time and passed to the next component in the processing pipeline. See figure 4.

image4

The message is now traveling on the channel emailsIn, destined for the next component on the channel, a transformer bean. The transformer takes whatever is given to it, changes it somehow (we'll talk a bit more on that in a bit), and then sends on its way. In this case, the Transformer is given a Message containing a payload of type MimeMessage,which is used to create an object of type BlogPost. The BlogPost is a domain class specific to our system. We'll use it as a DTO here, but it should be evident that it doesn't matter where it came from – it's acceptable to work in terms of your domain, if you want. This is the best use of the transformer element: transforming from messaging format to a format that's common to your system. This is related to the canonical data pattern.

The resulting BlogPost is wrapped in a new message that contains the headers from the original message. Headers are just what you'd expect them to be: values specific to the request that can be interrogated for extra meta data. Internally, the Spring Integration Bus assigns everything an ID, for example. That ID is exposed as a header value, for your use. The ID is guaranteed to be unique.

The message is then sent to a filter, emailsInFilterAgainstWhitelist, which interrogates the payload for the sender value. This is to ensure that you can't spam your way on to the blog! This example is absurdly simple, like the whole solution. You can imagine querying Spring Security or LDAP or something to do a better job than this. See figure 5.

image5

The final bit of XML configuration is the service-activator element, which takes what's given to it and does work with it. In this case, we're telling the Spring Integration to call the method pubishIncomignBlogEntry on a bean with ID newBlogPublishingServiceActivator. This method does the work of actually taking the payload, and sending it to a service that'll publish the blog entry.

And then we're done. It may seem like a lot... but it's essentially 4 stanzas of relevant XML. The declaration of the channels is for the clarity of the example. The poller element is defined just once so that everything that needs a poller in the context can make use of a default one.

Possible Extensions

We defined three channels and four components. By designing the components that make up the solution the way we did, we made them reusable in other integration solutions. The integration flow isn't limited to this example. We could potentially reuse that transformer, filter and even the service activator in more complex integrations. The only refactoring would be the Spring Integration XML file. There are a lot of possibilities. That's what a good ESB does: it delays the flow design and configuration until as late as possible.

It's easy to imagine where we could have taken the solution from here. What follows are a few possible additions.

  1. Use jBPM ([12]) to add support to the system for some sort of audit work flow. Suppose we wanted to add Business Process management (BPM) to the solution, such that when the message came in, it is stored in some sort of work queue for an editor to approve. The editorial process could proceed as required inside the workflow engine. When the final blog content is approved, the editor could send the finalized entry to the same email with some sort of keyword or identifying keyword which could be used as a correlation id. The correlation ID could be used to let the integration solution proceed, favoring the updated entry. See [13].
  2. Use Spring Integration to “multi cast” the blog. Sure, the blog's been published. But there's another issue here, which is best put using an age old philosophy question: “if an RSS feed is updated in the woods, and no one polls for it, did it really get updated?” Eh, perhaps I'm paraphrasing! The blog's title could also be announced via Twitter, the blog aggregators ([14], [15], etc) could be notified, etc, using the recipient list pattern ([16]).
  3. Update the filter example to authenticate against some sort of real authorizing service. Perhaps using LDAP or Spring Security.
  4. Diversify the support for publication beyond simple email. FTP, WebDav, file systems (I've even left a simple file directory adapter configuration commented out in the source code under the email adapter.), etc, are all viable inbound types. A more advanced example might enable SMS messages from mobile clients. (Of course, I'm not sure how many people are writing blog entries from their cell phones. You never know. Only one way to find out, however!). There's no support for this yet, but scrutiny of the Spring Integration source code will reveal that's it's easy to build your own adapters. You might use something like SMSLib [17].

Where Spring Integration May Fall Short

Spring Integration's brand new – and very powerful. You can be confident it's got the weight of SpringSource behind it, and will evolve. However, that doesn't mean it's perfect -far from it! Application integration is not a new field and there are decades of solutions and architectures to accommodate. Some uses of application integration have traditionally involved heavyweight adapters like those for integrating with SAP, or JDEdwards OneWorld. Spring Integration doesn't directly support some of these specifics cases.

While Spring Integration does support an awful lot out of the box, it is lacking support for some pretty typical adapters like SFTP, HTTPS, or AS2. Some of the proprietary solutions will support these requirements better, at the moment. Some solutions can be very expensive, so you might try adapting a third party library and writing your own adapter for Spring Integration. You'll be pleasantly surprised as writing an adapter for Spring Integration is pretty trivial, if you're interested. Just take a look at jSch [18], or Jakarta Commons VFS [19], or Jakarta Commons Net [20] for a wellspring of possible starting points for your solutions.

Finally, Spring Integration may not be the best solution for you (right now). If not, don't fret. The concepts you learned in this article translates well across other solutions. Go, integrate with impunity!

Conclusion

Spring Integration represents a clean, concise approach to EAI and a strong alternative ESB. Modern day ESB solutions are heavy, and hard to introduce in some environments. Spring Integration represents a powerful, low-friction alternative that can tame some of the most involved integration problems.

Resources

1 Amazon.com page for "Enterprise Integration Patterns"
2 Spring Source's Spring Integration, http://www.springsource.org/spring-integration
3 Enterprise Application Integration, "Canonical Data Model", http://www.eaipatterns.com/CanonicalDataModel.html
4 JCA Specification, http://java.sun.com/j2ee/connector/
5 BEA Tuxedo, http://www.oracle.com/products/middleware/tuxedo/tuxedo.html
6 Mule home page, http://mule.mulesource.org/display/MULE/Home
7 Mule Forge, a foundry for extensions to Mule, http://www.muleforge.org
8 Apache ServiceMix, http://servicemix.apache.org/home.html
9 JSR 208, Java Business Integration, http://jcp.org/aboutJava/communityprocess/final/jsr208/index.html
10 Blogger, Google's blog service, https://www.blogger.com/start
11 Apache Roller, http://roller.apache.org/
12 JBoss.org's jBPM Information, http://www.jboss.org/jbossjbpm/
13 Enterprise Application Integration, "Aggregator", http://www.eaipatterns.com/Aggregator.html
14 Dzone blog aggregator, http://www.dzone.com
15 JavaBlogs blog aggregator, http://www.javablogs.com
16 Enterprise Application Integration, "Recipient List", http://www.integrationpatterns.com/RecipientList.html
17 SMSLib, a Java SMS library, http://smslib.org
18 jSch, http://www.jcraft.com/jsch/
19 Apache Commons VFS, http://commons.apache.org/vfs/
20 Jakarta Commons Net, http://commons.apache.org/net/
21 My blog, http://www.joshlong.com

 

About the Author

Josh Long is an enterprise architect, speaker, and a loving husband based out of Phoenix, AZ. When he's not hacking on code, he can be found at the local Java User Group or at the local coffee shop. He maintains a blog at http://www.joshlong.com ([21]) and can be reached at josh@joshlong.com

Download the source code here.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Re: Getting Started With Spring Integration by Sasa Vender

Nice article, thanks.

"Vend" or "Send"? by Dave Macpherson

I'm assuming the the word "vend" throughout this article is "send"? Perhaps a quick "vind and veplace" is in order ;)

Great article by Srini Penchikala

Hi Josh, Great article. I was able to run the sample application w/o much extra setup effort. The example is simple but it's a good introduction to Spring Integration framework architecture. SI brings the power of Dependency Injection (DI) that Spring Framework has been famous for and EIP patterns implementation together to nicely manage the resources (there are more than few of these resources when you are working on a real-world asynchronous messaging system) and lets you focus on business logic rather than messaging infrastructure code.

Spring Integration vs. Apache Camel by Lari Hotari

How does Spring Integration compare to Apache Camel (camel.apache.org/)?

Camel uses Spring for configuration (integrates nicely to spring based approach) and it's possible to deploy Camel "applications" to a OSGI container (cwiki.apache.org/confluence/display/CAMEL/tutor...). Is Camel a direct competitor?

Re: Getting Started With Spring Integration by Gaetano Miranda

Nice article. I'd like to remark that I don't think SOAP(doc/lit) and REST are examples of RPC integration style.

Thank You

Do we even need ESB? by Oleg Zhurakousky

"Because Spring Integration is so lightweight (you deploy the Spring Integration server with your application; you don't deploy your application to Spring Integration). . ."

Well said and raises an interesting point.
Do we even need ESB as a product onto which we deploy applications specifically designed around such product or EAI in general should be looked at as an enabling framework used inside of your JEE or any other type application allowing integration concerns to be handled as they arise. Spring Integration allows just that.

SOA without the Complexity by Tom McCuch

Great article, Thank you!

SOA is an architectural pattern. A pattern, by definition, is the encapsulation of a complex system into a reusable component. Patterns are meant to describe "building blocks" for YOUR solution - they are not meant to be solutions in of themselves. Patterns therefore lend themselves to be best implemented by lightweight, embeddable frameworks that serve to support your solution, not heavy, commercial-off-the shelf products that aim to control it. This is where the big vendors have all gone wrong with SOA - and where SpringSource, yet again, has gotten it right with Spring Integration.

Great article... by Adam FitzGerald

Thanks for the great article, Josh. I have added it to the SpringSource community web site. I also want to point out something regarding adapters. If users of Spring Integration want to share the adapters they develop with others they should check out the new
Spring Integration Adapters project in Spring Extensions.

Adam FitzGerald
SpringSource

Distributed Spring? by Dhirendra Negi

Are we heading towards more heavyweights?!

Very interesting by Tai Truong

I am also watching on Spring Integration (SI) from the very beginning and since its release I have succesfully used SI for implementing a lightweight integration platform. It is in production for some customers and runs pretty stable :-).
One of things I like in SI is how they it takes care of many thinks to make integration transparent: it wires all the endpoints together into an integration flow and encapsulates the message processing. It is component-base and I can implement by translators, processors and any other endpoint without a dependency to any Spring class. This I haven't seen from other API like Mule or OpenAdaptor.

In the near future I will post my experiences in my Blog. If you are interested I have also posted a summary about the concepts of Messaging and EI:
taitruong.wordpress.com/2009/04/20/enteprise_in...

Cheers Tai

Re: quite similar to the new OpenAdaptor by Sarwar Bhuiyan

Hi there,

I was wondering if you had a look at the new Open Adaptor as it is also spring based and for the most part you can use their out of the box components in a spring xml file. If you need to write processors and endpoints, you can extend some basic interfaces that they have. I guess that may or may not always be necessary but it's still quite flexible in case you have a lot of custom requirements.

Sarwar

Application architecture by Kshitiz Garg

Hi Josh,

Thanks for your post!

I am aiming to create a notification server (a pub/sub and p2p model) and excited to use Spring Integration's features. To make that notification server an isolated black box, I m thinking to expose its functionality thru rest based APIs. Since ActiveMQ is highly configurable so probably going to use it for the message management part.

Do You think that it's a feasible & a wise decision to go for an architecture like this considering scalability & extensibilty ?

External systems <--> Restful APIs on Tomcat <--> Spring Integration acting as lightweight ESB <--> ActiveMQ (+MySql for message persistence and tomcat i.e. not the 'embedded' jetty) <--> Spring Integration <--> Restful APIs <--> External systems

What all pros & cons do you suggest in this scenario.....

Many thanks,
Kshitiz</--></--></--></--></--></-->

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

12 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT