Incorporating Enterprise Data into SOA

Posted by Boris Lublinsky on Nov 22, 2006 01:00 AM

Community

SOA

Topics

Data Access

Tags

Design Patterns,

Data Binding

Introduction

The majority of today's SOA design techniques ^1,2,3 are centered around definition of services. They use 1service-oriented decomposition, based on the business processes, enterprise business/functional model, required long term architectural goals and reuse of the existing enterprise functionality. This approach usually incorporates one of the most important assets of the modern enterprise - enterprise data as an afterthought. In this article we will revisit a typical SOA architecture, outline the complexities of dealing with the enterprise data and discuss several design patterns for incorporating of this data into SOA implementations.

Typical SOA implementation

A typical SOA design approach leads to implementation of the enterprise services in a form of a specialized layer rationalizing existing enterprise functionality (applications) against "ideal" enterprise business model ³.


Figure 1 Typical SOA implementation

Such architecture (Figure 1) defines multiple layers in the SOA architecture ⁴:

• Enterprise Resources and Operational Systems Layer. This layer represents the portfolio of existing applications (i.e., legacy, COTS and custom built systems).
• Integration Layer. This layer uses various technologies to expose existing enterprise resources and operational systems so they could be used by business components.
• Business Components Layer. Enterprise Business components are deployable units of software that provide the functionality required by the business services. These components can be either newly developed or "wrappers" using integration layer to access the functionality of existing enterprise resources.
• Business Services Layer. Business services provide high level business functionality throughout the enterprise. This layer effectively bridges between the "ideal" business model and the existing enterprise IT assets - applications and business components.
• Business Processes Layer. Business processes allow for creation of business solutions through orchestration of business services.
• Customer Facing Layer. Customer facing provide support for customers (both inside and outside the enterprise) to view and control the execution of the enterprise business processes and/or services. These customers can be either humans using Web or rich clients or B2B connections supporting intra enterprise business processes.

In order to enhance services interoperability, such architecture usually defines semantic messaging model - enterprise-wide business objects, used for the services interface definitions. This semantic model is typically derived from the same enterprise business model, which is used for the services definition, and consequently assures "common language" used by all services invocations. As a result typical SOA implementation effectively introduces two different data models ⁵ - "outside data" exposed by the service interfaces and "inside data" - enterprise data used by service implementations.

Enterprise data access issues

Although typical SOA implementations hide enterprise data behind service interfaces, it still requires solving of the following data access issues:

• Consolidation of data between multiple applications. Today's enterprise data is typically scattered between multiple siloed applications. Every application contains only a subset of the enterprise data, dedicated to the problem that the application is trying to solve and the data is often duplicated between applications. This data redundancy between applications creates an inaccurate representation of enterprise data and requires periodic data synchronization between applications, each of which is often considered to be a "master" data store for particular functionality/unit. In addition, data representations themselves are different between different applications. As a result, it is often difficult to reconcile data representation between individual applications. As the individual applications evolve independently, the complexity of the problem increases. As an SOA implementation is attempting to represent enterprise-wide functionality it needs to operate based on a well-defined enterprise data model. This means that enterprise data access from service implementations is required to correctly align and consolidate data from multiple existing applications and ensure propagation of data changes to all applications, using this data.
• Ownership of enterprise data by services. Foundation of the modern service definition technique - functional decomposition is not easily mapped to the enterprise data. For example, the notion of the customer (and corresponding data) is usually shared between multiple functional services. The problem is that functional and data decompositions are driven by completely different rules. Functional decomposition is defined based on the enterprise business processes - enterprise functionality, whereas data decomposition is defined based on the enterprise data taxonomy - underlying enterprise data model. As a result, aligning of the enterprise data with enterprise services becomes a daunting task.
• Interface definitions. Because service invocations are always remote, service design strives towards large granularity interfaces, aiming at minimizing of the amount of service traffic (chattiness) between service consumers and providers. Data access, on the other side, can require both high and fine granularity of interfaces, depending on the data access requirements. Finally, data access typically implements pure CRUD (Create, Read, Update, Delete)^a where as enterprise services implement business meaningful interface, like rate policy, etc.

Patterns for incorporating of the enterprise data in the SOA implementations

There are several design patterns aimed at support of the enterprise data in SOA implementations, some of which are well established and some are emerging, based on the experiences in the current SOA implementations. I will cover three most important ones below.

Aligning enterprise data support with the business services

This is the mainstream SOA implementation pattern today well aligned with the overall SOA implementation, presented above (Figure 1). In this pattern enterprise data access is incorporated in the business service implementation. Data is accessed through the integration layer and the service implementations themselves are responsible for defining and supporting rules for validation, storing and retrieving of a particular data set. The advantage of this design pattern is that it is well aligned with the overall service implementation approach. The disadvantages of this pattern are:

• Service implementation has to support not only business functionality of the service, but also all of the data validation and access and coordination logic. As a result business implementations are often intertwined with data access and transformation code.
• Sharing enterprise data between multiple enterprise business services requires additional design considerations and typically leads to additional coupling between enterprise services. If multiple services need an access to the same piece of an enterprise data the following solutions are possible:
• Duplicating of data access logic between multiple services. This allows every service to access data independently, but requires duplication of the data access, transformation and synchronization logic between multiple service implementations. Although, theoretically, it is possible to encapsulate such logic in a separate component, reusable by multiple services, in practice it rarely works. Multiple service implementation platforms, for example, J2EE vs. .Net, vs. mainframe will require different implementation of such component. Additionally, because components are not independently deployable, every change in such component requires redeployment of every service dependant on this component.
• Extending interfaces of a particular service to include CRUD methods for manipulation of the data controlled by the service. This approach leads to decreasing of the service methods granularity and diminish business meaning of the service methods. It additionally creates additional dependencies between enterprise services, breaking their autonomous nature.

In reality most of the SOA implementations are using the mixture of the above approaches. Although this pattern is typically applicable for small-size initial SOA implementations it does not scale well for enterprise-wide implementations.

Enterprise data access as a business service

One of the approaches for overcoming drawbacks of the previous pattern is to treat enterprise data access as a first class business services. In this case a special type of a business service - data service - encapsulates all of the logic required for access, coordination, validation and transformation of the enterprise data. The advantages of this pattern are:

• Explicit separation of concerns between implementation of the service functionality (business logic) and enterprise data support logic. Enterprise data services effectively create abstraction layer, shielding business functionality from the data access details.
• Better partitioning of implementation by factoring out extremely stable data access code from services implementations.
• Similar to the other business services, an interface for the data services is often based on the enterprise semantic model. This can significantly simplify consumption of the enterprise data by business services.

The disadvantages of this pattern are:

• Because the majority of the enterprise data come from the enterprise applications, implementation of the data services is typically similar to the implementation of the business services - it is build from components, implementing data access logic and components using integration layer to access data existing in the legacy applications. This makes it often difficult to distinguish between integrations aimed at the functional reuse and integration aimed at the enterprise data access. Enterprise data access integrations are typically not going directly against enterprise data store - they usually leveraging existing legacy implementations for both access and validation of data. As a result it is not always easy to classify a particular implementation as pure data or business service.
• As we have mentioned above, a typical interface design for data services is CRUD which is considered service interface design anti-pattern ⁶. Additionally, because data services need to support a wide range of enterprise data access options, required by business services, their flexibility has to be very high. This requirement usually manifests itself in low granularity of data service interfaces.
• Centralization of access to all of the enterprise data through specialized data services requires business services to use them for all of the data access. Such implementation can lead to fairly "chatty" implementations and creates strong dependencies of business services at the data services, which to some extent breaks services autonomy. One of the approaches to minimize these dependencies is creation of the business processes (composite service), which manages all of the enterprise data, required by underlying services and passes all of the required enterprise data to business services as a set of parameters. Although this approach improves business services autonomy it usually leads to "bloating" of the process state and services messages, thus negatively impacting business process scalability and service invocation performance.

Enterprise data bus

One of the most popular approaches to SOA implementation today is usage of the enterprise service bus pattern ⁷, allowing to "virtualize" enterprise services access ^b. Similarly, enterprise data bus allows for virtualization of the enterprise data access. In a lot of ways this design pattern is similar to the enterprise data access as a service pattern, described above. The difference is that instead of promoting data access to a first class business service, in this case it is treated on the level of the integration layer, thus providing a direct access from any service to any piece of the enterprise data Figure 2.


Figure 2 Enterprise Data bus

The origins of the enterprise data bus can be traced back to the federated database technology - virtualization of access to disparate databases in the enterprise. Unfortunately this technology is rarely applicable for the SOA implementations. The issue here is that existing applications effectively control enterprise data access through implementation of the data access and validation logic. Bypassing these applications and going directly against databases requires reimplementation of this logic, which is typically not cost effective. As a result data bus is typically implemented using integration layer (Figure 1). The new products on the market, for example IBM Information Server ⁸ provide support for both direct database access and integration approach. Such products expand flexibility in the enterprise data bus implementation by allowing to "mix and match" integration solutions with the direct database access ^c. Effective implementation of this pattern requires separation between integrations providing enterprise data access and integration providing enterprise functional capabilities. Considering that both data and functional integrations get/return enterprise data ^d, the most straight forward approach is to consider all of the integrations as data access integrations ^e. This leads to modification of the overall SOA implementation. (Figure 3)


Figure 3 Typical SOA implementation with a data bus

The implementation (Figure 3) adds additional layer - enterprise data bus, providing access to the enterprise data, contained in either enterprise applications or their subordinated databases. It also modifies implementation of the business components layer; in this implementation it does not contain "wrapper" components any more. Business components, in this case are comprised only of the components implementing service functionality based on the existing functionality and data accessible through the enterprise data bus. The advantages of this pattern are:

• Explicit separation of concerns between implementation of the service functionality (business logic) and enterprise data support logic (similar to the previous pattern). Enterprise data bus effectively create abstraction layer, shielding business functionality from details enterprise data/functionality access.
• Through encapsulating all of the access to enterprise data/functionality enterprise data bus provides a single place for all of the transformations between enterprise semantic data model and data models of enterprise applications.
• Because any of the service implementation, in this case, has an access to any of the enterprise data it requires, such implementation allows to significantly reduce coupling between service - service invocation contains only data references (key), which change extremely rarely, while the actual data access is implemented by the service itself using enterprise data bus. This means that if the service implementation requires additional data for its processing, it can access it directly without impacting its consumers.

The disadvantages of this pattern are:

• Due to the amount of accesses (synchronous) performance becomes one the most important characteristics of the enterprise data bus. Any performance degradation of the bus can effectively destroy SOA implementation.

Conclusion

As the scope of SOA implementations expands from limited in scope departmental implementations to enterprise-wide undertaking the issues of enterprise data access are quickly starting to become one of the most important implementation issues. If not architected correctly from the very beginning, enterprise data access can become a major problem down the road. Design patterns presented in this article define different approaches to dealing with the enterprise data in the SOA environment along with the drawbacks and advantages of each approach.

About the author

Boris Lublinsky has over 25 years experience in software engineering and technical architecture. For the last several years he focused on Enterprise Architecture, SOA and Process Management. Throughout his career, Dr. Lublinsky has been a frequent technical speaker and author. He has over 40 technical publications in different magazines, including Avtomatika i telemechanica, IEEE Transactions on Automatic Control, Distributed Computing, Nuclear Instruments and Methods, Java Developer’s Journal, XML Journal, Web Services Journal, JavaPro Journal, Enterprise Architect Journal and EAI Journal. Currently Dr. Lublinsky works for the large Insurance Company where his responsibilities include developing and maintaining SOA strategy and frameworks. He can be reached at blublinsky@hotmail.com

References:

1. Back to top

   Layering

   Nov 24, 2006 6:59 PM by Hong Du

   "You can always handle complex problems by layering".That's true.

   Reply

2. Back to top

   The advantages of Enterprise Data Bus

   Nov 27, 2006 1:36 AM by Hu Hu

   "Because any of the service implementation, in this case, has an access to any of the enterprise data it requires, such implementation allows to significantly reduce coupling between service - service invocation contains only data references (key), which change extremely rarely, while the actual data access is implemented by the service itself using enterprise data bus. This means that if the service implementation requires additional data for its processing, it can access it directly without impacting its consumers." Can you talk more about the "data references(key) mapping to actual data"? How does the service use the key to get the actual data through Enterprise Data Bus?

   Reply

3. Back to top

   Doesn't this confuse logical with physical? And miss a pattern?

   Mar 1, 2007 1:17 PM by graham berri

   In Pattern 1 I'm pretty sure the so-called business services will include at least some services that Boris would regard as data services in other circumstances. I don't mind that, except that it undermines what follows. In Pattern 2 If only some data access routines are presented as business services, then pattern 2 is no different from pattern 1. If *all* data access routines are presented as business services, that seems like a pointless academic exercise. Pattern 3 This seems to add another tier in the platform, meaning extra message passing and network use, and impose another a layer on all applications, meaning extra dev and test effort. For what? I see the benefit to vendors - more service bus sales. I see the benefit to enterprise architects - one technical architecture imposed on everybody. The benefits to app dev teams are obscure. Extra tiers and layers can complicate more than they simplify. Missing Pattern 4 Where BEA sell to an enterprise both an ESB (enterprise service bus) and a DAT (data abstraction technology), then what is to stop the enterprise implementing some "data services" on the ESB and some "business services" on the DAT? Nothing, unless Boris can redefine for us a data service so as to distinguish it from a business service which (directly or indirectly) accesses enterprise data. Can you tell the difference when you look at the invocation/reply interface? If you can tell, then how? If you can't tell, then how can you judge whether somebody has implemented any of these patterns or not?

   Reply

4. Back to top

   Re: Doesn't this confuse logical with physical? And miss a pattern?

   Apr 8, 2007 9:45 AM by Boris Lublinsky

   Thanks Graham for detailed response. One of the things that was important to me when I was writing this article was to try to distinguish purely data service from the business services (I probably did not do it clear enough) Business services are autonomous and are designed to minimize coupling, where coupling has many many facets, including temporal one. This means that asynchronous invocation is preferred interaction paradigm for business services invocation. Data services, especially data retrieval often play by completely different rules, they are often synchoronous in nature and require very fast responce time. This alone makes you want to logically separate the two. The other issue, typical with update services is requirement for multilevel transactional support, which is not the major strength of services implementations. These two alone vouch for trying to approach data services differently from business services.

   Reply

5. Back to top

   Re: Doesn't this confuse logical with physical? And miss a pattern?

   Apr 28, 2008 10:30 AM by Eric Roch

   Related post (mine): Data Services

   Reply

• Older >