Mainframe Integration with JBoss ESB and LegStar

Introduction

In the October 2008 issue of Dr Dobbs titled Is Your Next Generation Language COBOL?, Michael Swaine mentions that "billions of lines of new COBOL code are being written every year" as a testimony to the ongoing size and strength of legacy mainframe applications. No wonder that in this context, integrating mainframe applications with distributed systems, particularly Java-based ones, remains an important topic.

Mainframe integration is a domain where the industry has gathered considerable experience over the last few years. One lesson learnt the hard way is that tight coupling is bad. This is of course true for all integration systems but with mainframe applications, the technological gap is even wider: COBOL, CICS/IMS, TSO, on one side, Java, J2EE, Eclipse on the other. Creating too much dependency between applications with very different life cycles, developers mindsets, tooling, etc. leads to brittle and unstable systems.

Mainframe integration vendors have very early on adopted Service Oriented Architectures (SOA), leading to a first generation of SOA-inspired mainframe integration solutions. Most of these solutions are mainframe-centric in the sense that binding COBOL to XML and SOAP messaging run directly on the mainframe.

The recent adoption of Enterprise Integration Patterns by a large IT community and the emergence of the Enterprise Service Bus (ESB) as the SOA infrastructure of choice, have created new architectural opportunities for mainframe integration. In this article we advocate a new, ESB-centric, view of mainframe integration and we introduce LegStar for JBoss ESB, an open source solution that implements such an architecture.

First generation, mainframe-centric, integration is showing its limits and cost

Michael Swaine mentions that the COBOL language itself is evolving to support XML and that systems such as CICS now offer SOAP support. Besides IBM, there is a long list of vendors providing COBOL binding to XML and SOAP stacks on the mainframe. For several years, analysts, specializing in legacy integration systems, have favored this mainframe-centric approach to integration with XML and SOAP running natively on the mainframe.

Because no native mainframe SOAP stacks existed at the time, most mainframe-centric integration vendors felt compelled to develop homemade SOAP stacks. Some going as far as developing their own XML parsers.

However, SOA and Web Services have evolved rapidly since then. A few years ago providing basic XML processing and SOAP 1.1 was enough to claim Service Orientation. Since then, customers expectations and awareness of SOA has evolved and new standards, such as WS-*, has been adopted. As a consequence, the maintenance cost of homemade SOAP stacks grows rapidly.

Mainframe developers lack the large communities and pooled resources that open source provides. As a consequence, several mainframe SOAP stacks do not yet offer SOAP 1.2 or WS-Addressing, which are pre-requisites for many WS-* specifications. As a comparison, the latest Axis2 release from Apache is a 20 Mb download, including 59 libraries covering SAAJ (SOAP 1.1 and SOAP 1.2), XML Schema, WSDL 1.1 and WSDL 2.0, WS-Addressing, Fastinfoset, WS-MetadataExchange, MTOM, WS-Policy and more.

Furthermore, new technologies are emerging such as RESTful Web Services. With support from Google, Yahoo and Amazon, this type of architecture is gaining popularity and the adoption of JSR 311, the Java API for RESTful Web Services, has contributed to its credibility. JSON also gains momentum as a lightweight alternative to XML. Axis2 for instance, now has support for JSON. Integrating mainframe applications with RESTful Web Services and JSON support are likely to become requirements in the near future, adding yet more items to an already long backlog. Beyond the basic Web Services capabilities requirements for orchestration or scripting languages have emerged putting yet more pressure on mainframe-centric development teams.

Mainframe-centric solutions are not only lagging behind in term of features but they often suffer from poor design decisions taken hastily when SOAP emerged.

For instance, mainframe-centric solutions often tightly couple Binding, Messaging and Transport layers. The open source community on the other hand made the important decision to cleanly separate language binding (Java to XML binding for instance) from transport and messaging. In the Java world, JSR 222 Java Architecture for XML binding (JAXB) is typically implemented independently from any Web service technology; you can use it with SOAP or RESTful Web Services. Similarly, Messaging and Transport are usually clearly separated, making it possible to ship SOAP messages over JMS or TCP sockets in addition to the traditional HTTP/HTTPS. As a consequence, mainframe-centric solutions are harder to adapt to the changing SOA landscape.

Mainframe-centric integration solutions were also designed considering the mainframe as central to the IT infrastructure. Distributed applications were perceived as peripheral. This has lead to an asymetric vision of mainframe integration where the only need, or at least the most important one, was for these peripheral applications to consume mainframe resources. As the importance of distributed applications grows though, mainframe applications cannot stay isolated any longer. Many mainframe-centric integration vendors have recently perceived the threat and started offering outbound capabilities but few have undertaken the fundamental overhaul that is needed to support a more balanced vision of the role of mainframe applications in the IT infrastructure. SOAP Client features are typically very weak and limited on mainframes, severely restricting outbound access to distributed services.

Finally, an important aspect of mainframe-centric integration is cost. Software licenses and maintenance fees are expensive on mainframes. Open source has not penetrated that world yet and may never do so. In addition, XML and SOAP processing are CPU intensive activities. CPU costs on mainframes have remained comparatively high, despite several initiatives by IBM to alleviate this fact.

In summary, it becomes harder and harder for mainframe-centric integration vendors to catch up with the latest Web Services standards, respond to new requirements and adapt to the changing role of mainframe applications as well as the new cost structures popularized by open source software.

Next generation, ESB-centric mainframe integration

In the open source world, Enterprise Service Buses (ESBs), have emerged as the best infrastructure to support SOA. They offer support for many transports, registry services, sophisticated message transformations, orchestration and more. In a sense, they cover the ground previously owned by Enterprise Application Integration (EAI) proprietary platforms.

An ESB-centric mainframe integration solution leverages these advantages by building on the ESB architecture. For example, binding COBOL to Java or XML is handled as any transformation activity within the ESB. Exchanging messages with the mainframe uses the ESB standard transports or build on existing ones. Mainframe applications have no awareness of the ESB, they could as well exchange data with another mainframe.

ESBs are a very dynamic segment of the open-source market and are much better understood by the younger developers community than the mainframe. ESBs benefit from large open-source communities and are directly exposed to emerging technology challenges such as BPM, dynamic languages and RESTful Web Services. BPM and BPEL, which remain challenges for mainframe-centric integration solutions, are widely available on ESBs. With ESB-centric integration, mainframe interactions can readily participate in complex business processes.

Like their mainframe-centric counterpart, ESB-centric mainframe integration tools feature design time and runtime capabilities. Design time tools are needed to map COBOL to open world complex types. The difference is that ESB-centric design time tools produce artifacts specifically targeted at an ESB. Once deployed in that ESB, these artifacts will convert streams of mainframe data into open world objects. Conversion takes place as intra-ESB tasks and benefits from the multi threaded, scalable architecture of ESBs.

ESB-centric mainframe integration also benefits from the wealth of XML optimization techniques that are available today in the open world to reduce memory footprint, as well as CPU consumption of XML-related activities. Arguably, CPU is already cheaper on distributed platforms than on mainframes but on top of that, sophisticated tools, such as profilers, which are widely available in the Java world, are much less common on mainframes. ESB-centric solutions are better equipped to monitor and control CPU or memory consumption, reducing the overall integration cost even more.

In an ESB-centric solution, generated artifacts are consistent with anything that can be deployed in the ESB (Java, XML, jars ...). This means these artifacts can be managed with the abundant management tooling of the open world (source control, build control, dependency management, continuous integration ...), they can be unit tested, instrumented and secured. Integration artifacts are development artifacts and highly sensitive at that. They are often crucial parts of high volume transactional systems and have access to sensitive data. In ESB-centric solutions, they can be treated as first class citizens of the development ecosystem.

ESBs also offer a better level of insulation than mainframe-centric SOAP stacks. To illustrate this point, consider the following use case: mainframe code needs to consume a service provided outside of the mainframe, on a distributed platform. As distributed environments become richer in features and take on a more preeminent role in the IT portfolio, it is more and more common that reference data and processes originates from distributed environments. In such situations, the remaining processes on the mainframe need access to these distributed functionalities. In a mainframe-centric integration solution, the COBOL program would have to issue a SOAP request. With an ESB-centric integration solution on the other hand, the COBOL program has no knowledge of the particular messaging or even that it is talking to a distributed platform. It just drops mainframe data in a WebSphere MQ queue or in an HTTP payload. Knowledge of the target service and protocol is kept within the ESB, which is precisely the role of an ESB. ESB-centric solutions offer a higher level of decoupling which is the most desired quality of an integrated system.

LegStar and JBoss ESB, an ESB-centric mainframe integration solution

LegStar for JBoss ESB, an open-source mainframe integration technology, is one of the first to specifically target ESBs as integration platforms. In this article we demonstrate how Redhat JBoss ESB, an open-source ESB, can be leveraged using LegStar to give mainframe COBOL programs access to the outside world. The combined architecture uses synchronous or asynchronous messaging, with minimal mainframe technologies involved, effectively decoupling mainframe programs from the distributed processes.

Use case

We will expose a Java object (a Plain Old Java Object or POJO) to COBOL consumers.

LegStar for JBoss ESB can also cover the more classical use case where a COBOL program is exposed to Java consumers but there are several unique challenges when COBOL code needs to act as a client, particularly when loose coupling is paramount.

LegStar for JBoss ESB also allow COBOL code to consume ESB Services and Web Services. We have chosen to consume a POJO to keep things simple.

Architecture

On the mainframe side, we assume COBOL as the development language and WebSphere MQ as the transport mechanism.

On the server side, we use freely available, open-source technologies: JBoss ESB and LegStar.

The deployed architecture looks like this:

Several things are worth mentioning about this choice of architecture:

WebSphere MQ transports raw data to and from JBoss ESB

Alternatively, you can use HTTP to converse with the ESB but WebSphere MQ is widespread on mainframes and offers better support for asynchronous messaging than HTTP. The COBOL code could be running in TSO, as a Batch program, in CICS or IMS. The WebSphere MQ API is available for any of these environments and more.

The COBOL program drops raw mainframe data in the WebSphere MQ Message. This content is not translated on the mainframe. This means JBoss ESB will be receiving raw mainframe data. There are two advantages in doing so:

• there is less CPU consumption on the mainframe (no data conversion)
• the mainframe does not need to know that it is talking to a non-mainframe platform

This second point is particularly important in term of decoupling.

Although the architecture diagram shows a single WebSphere MQ Manager on z/OS, there will probably be another one on the distributed platform. We did not represent that additional manager for simplicity and also because it is not strictly necessary.

From the ESB perspective, we use a standard Listener that comes with JBoss ESB. This is a high performance, multi threaded listener capable of supporting high loads of incoming, concurrent, requests from the mainframe.

On the mainframe side, the client code interacts synchronously or asynchronously

The COBOL client code is free to wait for a response, using standard WebSphere MQ APIs, or just return after posting the request. An asynchronous messaging pattern is implemented using WebSphere MQ standard triggering mechanism where another COBOL program can process the reply.

Asynchronism is desirable as it creates less dependency between systems. This type of exchange pattern also fits the ESB model well.

The JBoss ESB Service is a proxy between the COBOL client program and the POJO

JBoss ESB supports various transports and provides a versatile pipeline concept that we leverage here.

The JMS Gateway, a standard JBoss ESB listener, monitors a WebSphere MQ Queue and triggers a series of actions when a message is received. JBoss comes with several out of the box actions such as the Object Invoker or the JMS router that we use here.

The transformation actions that are needed to marshal/unmarshal raw mainframe data are generated by LegStar at design time as described in the next section. The artifacts that are produced by LegStar are specific to, and optimized for, JBoss ESB.

To a certain extent, the ESB Proxy Service shields the COBOL client from changes that might affect the target POJO. First, the COBOL client is not affected by changing the POJO location or package name. You could also add new properties to the POJO without affecting the client. More profound changes, such as properties being deleted or changing type, would still have an impact on the COBOL code of course.

Step by step

Here we describe the major steps that a developer would follow to create the ESB Service Proxy to a POJO. LegStar for JBoss ESB comes with Eclipse plugins that simplify the process. Alternatively, the same tools are available as ant scripts but we have chosen the Eclipse way because it is friendlier.

The completed Eclipse project is available here.

Target Java Object (POJO)

We use a Plain Old Java Object (POJO) for the purpose of this article but it could as well be a JBoss ESB Service or an off-ESB Web Service.

It is not just any Java object though, we expect the Java class to expose a single, coarse grain, method taking a single complex object as input, producing a single object as output and raising an exception if something goes wrong. This behavior relates to the Remote Facade pattern. In the context of this pattern, input and output objects are known as Data Transfer Objects. We will refer to such objects as data objects in the rest of this article.

The unique method is shown here. The input data object representing the request is called JVMQueryRequest. It contains a collection of environment variable names. The output data object, called JVMQueryReply, will contain a collection of values for the requested environment variables as well as a few locale specific values.

public class JVMQuery {

    public JVMQueryReply queryJvm(JVMQueryRequest request) throws JVMQueryException {

        List  envVarValues = new ArrayList ();
        try {
            for (String envVarName : request.getEnvVarNames()) {
                if (envVarName == null) {
                    throw new JVMQueryException("Invalid variable name");
                }
                String value = System.getenv(envVarName);
                if (value == null) {
                    envVarValues.add("Not found");
                } else {
                    envVarValues.add(value);
                }
            }
        } catch (SecurityException e) {
            throw new JVMQueryException(e);
        }

        JVMQueryReply reply = new JVMQueryReply();
        reply.setEnvVarValues(envVarValues);
        Locale locale = Locale.getDefault();
        reply.setCountry(locale.getDisplayCountry());
        reply.setLanguage(locale.getDisplayLanguage());
        reply.setFormattedDate(
                DateFormat.getDateTimeInstance(DateFormat.FULL, DateFormat.FULL,
                        locale).format(new Date()));
        reply.setCurrencySymbol(Currency.getInstance(locale).getSymbol());
        return reply;
    }

}

Mapping Java data objects to COBOL structures

When raw mainframe data reaches the ESB, action must be taken to marshal this data into a Java data object. This is where LegStar comes into play. LegStar provides design time tools to map COBOL structures to XML Schema and Java. It also offers a runtime binding framework to perform marhaling/unmarshaling of raw mainframe data.

LegStar comes with a set of Eclipse plugins that allow users to pick up a Java object and request automatic mapping to a COBOL structure:

The result is an XML Schema with special COBOL annotations. Here are a few element descriptions resulting from mapping the JVMQueryReply data object, as presented by the Eclipse standard XML Schema editor:

You may notice that some COBOL properties have default values. For example, character strings have a default size of 32. This is because COBOL only supports fixed size strings. Similarly, COBOL does not support unbounded arrays. You can easily change the default values in the XML Schema editor.

Generating COBOL binding classes

LegStar COBOL binding classes are the Java classes that perform marshaling/unmarshaling of raw mainframe data into Java data objects.

In Eclipse, a plugin extracts complex types from XML Schemas. It then allows you to select the complex types you want to generate binding classes for. In the following example, we select the input and output types:

The LegStar COBOL binding mechanism builds upon the Java to XML binding framework known as JAXB or JSR 222. The binding process generates JAXB classes with additional COBOL annotations. If you open one of the generated JAXB classes, you will see how each property in the Java object holds a COBOL annotation:

Generating JBoss ESB actions and sample configuration

The final step will produce a set of artifacts to help with deployment and testing of a JBoss ESB proxy service. In Eclipse, we select the WebSphere MQ generation option and specify where the target JBoss ESB service should be deployed:

LegStar generates sample JBoss ESB configuration files with a complete actions processing pipeline ready for testing. It also generates COBOL client sample code that can be used to call the JBoss ESB service and jump-start your own development.

Conclusion

LegStar for JBoss ESB is an example of next generation, ESB-centric, mainframe integration solutions.

This is a summary of the characteristics of this new type of architecture:

• It is truly bi-directional, the mainframe code acts as server or client depending on circumstances.
• Mainframe COBOL programs are effectively loosely coupled to distributed services. In the example shown, the COBOL program addresses a target service using a ubiquitous WebSphere MQ queue name. As far as the COBOL program is concerned, the target service could sit on another mainframe, be implemented in Java, PHP and, for that matter, anywhere on the Web. The ESB knows where the service is implemented or at least how to forward requests to it.
• The ESB is in charge of insulating the mainframe code from the target service idiosyncrasies. In a world where technologies evolve rapidly, there will be less need to modify the COBOL programs when the target service changes.
• Mainframe programs can readily participate in complex workflows. Instead of the simplistic action pipeline shown in this article, JBoss jBPM or Active Endpoints ActiveBPEL can provide process workflows and service orchestration with mainframe programs as active participants.
• Use of standard, open, technologies such as XML Schema, JAXB, JAX-WS, JMS or HTTP guarantees that a large community will keep them alive. This is in stark contrast with the current set of mainframe-centric SOAP stacks, some written in assembler, which are black boxes, proprietary and slow evolving.
• All artifacts produced are open world friendly: Java code, XML Schema, XML configuration files, etc. This allows integration artifacts to be managed with the powerful tooling that has now become widespread in the Java community.
• The footprint on the mainframe is reduced. In our scenario, there were no foreign technologies at all on the mainframe. A COBOL compiler and WebSphere MQ were sufficient (Alternatively, HTTP can be used). All CPU-intensive activities, such as conversions and XML processing, occur off the mainframe on platforms where CPU is cheaper.

COBOL and mainframe applications will be around for a long time to come. Many such applications fulfill their purpose quite effectively. In parallel, open world applications are growing in size and criticality, with SOA and ESBs as fundamental building blocks. In such a context, ESB-centric mainframe integration brings unique advantages that are worth considering.

About the authors