mySOA: Agile, Governed and Sustainable

Posted by William El Kaim on Jan 18, 2010

Community

SOA

Topics

SOA Platforms ,

Model Driven Engineering ,

Messaging ,

ESB ,

Master Data Management ,

Governance

Tags

Business Architecture

SOA is a topic extensively covered in the literature. After having read lots of books, articles, software vendors’ white papers’ and blog posts, I was still wondering how to make it real. The main objective of this article is to present a journey into SOA, described with our “words” and adapted to our constraints.

We call it “mySOA” approach.

• SOA obviously because we will try to build Agile, Governed and Sustainable business and technical services across the enterprise, and even beyond (cloud, B2B, etc.).
• “my” because the Web 2.0 revolution will make the services more and more used for social interactions, than for point to point conversations. “my” also because it is specialized to our needs, aligned with our company business.

We do not pretend to provide the state of the art solutions in this domain, or even to provide a unified methodology (terminology was the result of a join work between internal teams!) but we hope it will provide a foundation to help you build your own mySOA approach or to encourage you to extend your body of knowledge on the subject.

Finally, in order to be as concrete as possible, we will cite some tool vendors or tool platforms we used. We really encourage you to make your own proof of concept on your particular context before buying any technical solution. One size does not fit all, especially in mySOA.

mySOA: Agile, Governed and Sustainable

The mySOA approach is based on three pillars: Agility, Governance and Sustainability

mySOA approach should be agile

Agile meaning that we will not always follow the “you must” and “you should” found in the literature (you must have a business strategy, you must have the support of the top management, you must have do a top down approach, you should make big design upfront).

We will do it by iterations, leveraging small teams in charge of all aspects of a particular set of services. The objective is to create what we call “service blades” than can be plugged and reused in different business or technical contexts.

Agile

• Because the business has to be agile to survive in a fast, and flat interconnected world.
• Because our development teams are using SCRUM and moving to agile is so natural.
• By necessity, because the financial crisis prevented us to have a full fledge SOA project budget for several years.

mySOA approach should be governed from day one

We used governance as a way to support both business and IT new forces, but also as a way to enforce collaboration and alignment. Trust was, as always, a big part of the challenge; since teams had to delegate some of their decision power and accept some rules (for the benefit of all). Agile means also that exceptions management should be part of the governance process DNA.

mySOA approach should be sustainable

SOA will allow for a progressive and sustainable overhauling of functional and technical silos so as to design reusable or versatile services that will be called in various business processes. Sometimes, more than reuse, we will look for the adequate assets to provide value to the business, for an uncertain time. If we have to develop a portlet for enabling a business service to only one particular client (like a CO2 calculator), then we will do it. It could be reused or not, but should be thought to last.

Sustainable means also that people embarked in this journey should be ensured to keep their job, even if they have to move to another role, due to mySOA maturity and new business models or organization. Too often, the result of an SOA project is reduction of cost, outsourcing (off shoring is may be a better word) and destruction of teams that helped building it. mySOA is based on people, inside the company and should use them as champions to increase its value. mySOA focus on creating added value to the company.

mySOA – Choose your path to maturity

SOA encourages setting up an agile enterprise, but the path to mySOA nirvana is up to you. It is really important to position uniquely any of your SOA projects and initiative to ensure that all stakeholders understand the cost, the risks and the possible benefit level. Figure 1 shows a particular SOA project categorization model, coming from the sustainable IT community, which has the advantage to clearly position your project in cosmetic, overhaul or extended SOA.

1.    Cosmetic SOA.

• Non intrusive to existing asset: services are exposed with help from existing systems;
• This is not a rewriting of systems yet;
• This SOA is reliant on quality of existing systems;
• This SOA allows for obtaining some limited quick wins.

2.    Overhaul SOA

• Re-structuring existing applications with help from services;
• IT infrastructure can be fully used;
• Business rules and business logic stay encapsulated in the application code.

3.    Extended SOA

• Using solutions that enhance systems agility: Business Rules Management System, Master Data Management, Business Process Management

Figure  1: Path to SOA maturity (from Sustainable IT)

The extended SOA is the most “expensive” one (time, resource, money), but is the one that provides the best future proof results. The extended SOA is the target (to be model). In order to follow an agile approach and to progress to the target, the best path is to go through transition phases where services will be built by ensuring that master data, business rules and semantic mapping are clearly separated, and not entrenched on applications you can access through web services.

MySOA relies on a Competency Center

The first thing we did was to create an Integration Competency Center (ICC). The term “SOA” was not chosen on purpose; “integration” is better understood and positively seen by the business. It does not mean, that it will not evolve in the future (sustainability again, we build always on what we did before, and agile!).

ICC is a distributed organization, composed of two main groups as shown in Figure 2. The solution center that ensure a sustainable support for all projects over the time, and the center of expertise which provide agile distributed support on demand to projects (or recommends consulting firms).

Figure  2: ICC Organization is Twofold

The ICC charter was the first document created, and define that it should acts in the following fields:

• Business level
• Business Integration and Innovation: Socialize and evangelize the value of business integration capabilities between Business Units and sold products.
• Business Domain Governance: support the business in defining SLAs with supplier and key functional needs (list of business services to be provided, independently of the technology).
• IT level
• Technical Governance: Create technical standards and define rules to validate them, especially around Service Design and Interface Standards. ICC has developed a minimum set of technical standards that defines: the key architectural principles to follow, XML schema standards, WSDL schema standards, Service Level Definition template standards. ICC also provides a dedicated intranet and a wiki for questions and discussions.
• Service Development Lifecycle Governance: The ICC is not developing services by itself so it need to be associated at agile validation point to ensure things are done accordingly to the governance rules (and evolve if needed, the governance process is also agile).
• Service Delivery and Run-time Governance: Deliver managed services globally, by taking care of security, SLA and building adapted virtual “services” (or endpoints) when needed. ICC manages all technical supplier interfaces.

The solution center is composed of a distributed agile team composed of full time employees spending some of their work time on SOA. This was a way to avoid the “Tower of Babel” syndrome and to keep the team busy on trying to make mature real projects and not only do governance or slides. It can rely on contractors if needed.

Figure 3 features the services offered by the ICC.

Figure 3: ICC Organization is Twofold

Looking for my Services

One of the key question coming over and over is how do we find services? How can we define the right granularity? The answer is as usual not easy. We followed three different approaches:

• Top-down - Business Service Elicitation
• Agile – Follow your agile development process
• Bottom-Up – Brownfield development

Business Service Elicitation (Top Down)

It is outside the scope of this article to detail how to discover business services. We can recommend you anyway to read the Sustainable IT Architecture[1] book or to access documents and training content available for here (free license under creative commons).

You can follow the following very general steps:

• Define product offering and their dependencies: what are the business value chains for a limited business domain (stay agile);
• Define business objects, their lifecycle and their relationships;
• Define master data;
• Create business services that maps the Business Objects or Business Interaction lifecycle;
• Define or forecast variants of your service as soon as possible.
• Agile – Follow your agile development process

Another way to do it is to leverage the agile development process, by doing “Forwards Versioning Strategy”. Work within the integrated team, make priorities, develop and test.

Of course building service interfaces in several iterations can cause some issues, since the services are evolving and service users can be unhappy to have to follow the releases. This is a tribute to pay for having a service that suits your needs quickly.

Each team should understand that if we want to avoid a big design upfront, which by the way does not always prove to be efficient, they have to follow some rules. This could also be avoided by imposing some technical standards and testing often. Communication, as usual is key.

No silver bullet here, but agile was a really astonishing way of finding the “right” services at the “right” time for the “right” business needs (being on time is more important than being just right).

Brownfield Development

Brownfield development is a term commonly used in the IT industry to describe problem spaces needing the development and deployment of new software systems in the immediate presence of existing (legacy) software applications/systems. This implies that any new software architecture must take into account and coexist with live software already in situ. For more information, we can recommend to read Eating the IT Elephant[2].

In that case, what we recommend is to:

• Mine your data. Look at all data you have in your systems and try to get the most value of them.
• Control their lifecycle. This is a pre-requisite. For example, we used Convertigo to get access to our legacy back office data, without breaking the existing processes, business rules and security constraints.
• Ensure their quality. Do not neglect that part, this is the most common mistake made.
• Define one version of the truth. Define who the master is, and who will work with replicated data. You can use MDM tools or build a federated MDM (in general coupled with some middleware on top of it to manage data synchronization and distribution).
• Service-orient your data. That’s the latest part of the job. Not always the easy one, since it will create more load and complexity to tune on the DBMS side.

Informatica will propose with its new platform V9 a unified approach to data management, data integration and data service enablement. They had intense discussions with architects, including me, from different client companies since more than a year, to understand our issues and build their platform. I encourage you to read some of the lessons learned of this dialog here. Open source data integration vendor are also rushing to this Eldorado like Talend or Xaware.

Some Best Practices

Look at relative publication concerning Service Identification, like Accenture SIF, BPM and SOA Handshake, InfoQ article on SOA identification, this one for collaborative work and of course check Thomas Erl encyclopedia.

Below is a set of best practices that could be helpful to know.

• Avoid general purpose services. A service must have value to someone.
• A service with multiple variants and versions validates its usefulness.
• A service provides a specialized view of information.
• A service follows always a lifecycle, and it needs to be governed.

 mySOA - Agile governance by necessity

Since the number of services to be managed can grow quickly, we decided to focus on the most “important” ones (again it is a team decision). Importance is based on business needs (client request, new product functionality), and technical needed commodities (security support, REST support, Content Delivery Network, etc.).

We had then to create a governance ladder and align service governance effort to service importance:

1.      Fully governed: ICC Managed Service – A service that is deployed for global use and critical to the business that conforms to all the ICC governance rules.

2.      Known and governance delegated: Local Managed Service – A service that is sponsored by a local interest that conforms to the local governance rules.

3.      Known, no governance (use at your own risks): Unmanaged Service – Any service that does not conform to the appropriate governance rules or monitored.

4.      All: Monitored – SLAs are tracked and reporting is possible.

5.      Snapshot mode: Conform to Governance – Technical and SLA requirements have been measured once and on demand and validated.

In all cases, the services should be declared and categorized in a unique repository.

A service provider is the owner of the service, and can be internal or could be external to the company (suppliers). Whatever the service governance level chosen, the service provider should follow a minimum set of rule. A service provider:

• Is fully responsible of all aspects of the service development lifecycle.
• Is familiar with the standards defined by ICC;
• Ensure service design conform to the ICC standards;
• Ensure service implementation conform to SLA;
• Ensure support level 2 and 3 are available;
• Ensure, that no more than two consecutive versions are in production;

ICC is in charge of managing the agile garbage collection of services. It checks that the service is still useful and used. This is key to avoid having a never ending catalogue of services.

Taxonomy of Services                                                             

In our SOA vision, all our internal business units should contribute to delivering services. The architectural blueprint can be used for the design of new systems as well as the refactoring of existing systems.

• For the business decision maker, understanding the business value of a component and its commoditization level makes it easier to make build vs. buy vs. lease decisions, and may expose business opportunities for making a service available for others.
• For the architect, having a good grasp of the different categories assists in classifying existing or new services, as well as helps define the appropriate functionality to include in a particular service in order to promote composition and reuse.

ICC is using a unique categorization for all services, as shown in Figure 4. Service call direction is the following ones:

• Service calls always go down the stack, never up
• Service calls may skip layers if intermediate layers are not required

Figure  4: ICC High Level Service Taxonomy

Technical/Infrastructure Domain Services

Technical / Infrastructure Services are common facilities that do not add any explicit business value, but rather are part of the required infrastructure for the implementation of any business process in an SOA. They are typically purchased or custom built and centrally managed.

Let’s cite for example:

• Communication Services transport messages into, out of, and within the system without being concerned with the content of the messages.
• Utility Services provide generic, application-agnostic services that deal with aspects other than transporting application messages.
• Security Services provide required security-related capabilities like identity, privacy, confidentiality, non-repudiation, etc.

Business Domain Services

Core Business Process Services tie together the data-centric and action-centric building blocks to implement the business processes of the organization. They are in general statefull services (manages Business Process state). An example of a Process Service is a purchase order processing service. A Business Service will never consume a Core Business Service, but a Core Business Service may consume a Business Service

Business Services implement the business-level capabilities of the organization. They are in general stateless services and represent the action-centric building blocks (or “atomic verbs”) which make up the organization’s business processes.

Business Entity Services unlock and surface the business entities and their different lifecycle states in the system. They are in general stateless services. Business entities can be thought of as the data-centric components (“nouns”) of the business process: employee, customer, sales order, and so on. Business Entity Services are the operations on those data objects that could be:

• CRUD operations: Create Retrieve Update Delete;
• Search functions;
• Sustainable operations based on the business entity lifecycle states.

Application Domain Services

Application Processes implement the specific business-level capabilities or some other action-centric business logic elements (“building blocks”) which are unique to a particular business application context.

• Stateless/Statefull Service;
• Not managed by ICC.

Service Lifecycle

Our service lifecycle is as simple as possible. It is composed of several basic steps represented in Figure 5.

1.      Define. Determine if a new service is required, collect functional requirements (user stories in agile) and Service Level Requirements. ICC can support or participate to these steps.

2.      Develop. Code business logic and document (iteratively in agile) or rent access to the service (on demand). In any case, populate the registry with all necessary assets.

3.      Validate. ICC is in charge of controlling design time governance. The work to be done could be more or less important depending on the governance level associated to the service. Return to Define step if necessary.

4.      Deploy. ICC setup run-time governance (SLA control, billing control, and Business Activity monitoring).

5.      Decommission. Remove from production, Remove from SOA registry,

Figure  5: mySOA governance process

If you prefer to follow a well established process adapted to service lifecycle, like RUP shown in Figure 6, you can of course do it also. You can also benefit from some specific vendor tooling (UML profile for software service and RSA SOA plug-in) and consulting support. You can be agile or not, that’s again still your choice!

Figure  6: RUP process adapted for SOA

Business Service Blade

A Business service blade

• Is grouping all necessary services to support a Business Process or key value added functional or technical features.
• Is created and configured independently of all other blades (i.e. as far as possible no dependencies between blades should be made).
• Is self contained hiding for the client its specific “projection” on technical platforms.
• Its usage can be tracked and it can be billable.
• Is created by all BUs or groups or by 3rd party suppliers (available on demand).
• Is managed by ICC or not;
• Can be versioned entirely.

Business Service blades were created to enable granularity adaptation between service needed and service available, as resumed in Figure 7. It was a way to align the existing set of business services or application services we had, and the ones needed in business process.

Figure  7: ICC High Level Service Taxonomy

mySOA, How we did it?

For sake of simplicity, this article describes linearly the different aspects of mySOA and the work to be done. Of course, in real life, we did experience successes and failures and adapted the approach to business and IT maturity and willingness to make things right. Our journey was nevertheless simplified by some key enablers.

mySOA Platform requirements

In order to builder our Enterprise Service Platform, we tried to look at best of breeds tools and see if we could make them interoperate seamlessly. The main requirements were:

• The enterprise service platform should be built iteratively, and based on off the shelves component. The business was clear on that stake; our job is not to build an SOA platform!
• A unique manageable, scalable, fault tolerant platform;
• That leverage the existing infrastructure and licenses as far as possible;
• That enable retirement of legacy systems and data when possible;
• That could be leveraged for all services categories and integration flows;
• That will provide End to end governance;
• That will offer both business and technical monitoring (design and run time);
• Reduced maintenance and operational costs.

mySOA Reference Platform

The main services to be offered by the Enterprise Service Platform are presented below (and depicted in Figure 8:

• Enterprise Messaging Services. Offer secure, managed and reliable messaging services (for example you can find a list of open source java jms servers here, or use EMS in the cloud from Amazon SQS or Microsoft dotnet services).
• Master Data Management Services. As already stated, you can find documented process, best practices and commercial and open source tools.
• Semantic Data Integration Service. The objective is to create and manage Enterprise Information Models (mediations between applications and services with different structures and semantics). Then you can construct and centrally manage the rules and operations needed to validate and reconcile data. Semantic data integration automates and manages operations that are central to ensuring data quality like Transformations, Aggregation, Validation and enforcement of business rules. For more information on the semantic layer, please take à look at Sustainable IT Architecture documents.
• Data Distribution Services. Distribute and Ensure delivery of quality data to required end points, provides services for security, define canonical data formats and mappings when needed, collect and reports KPI information (ex: Tibco BusinessWorks and Informatica Platform)
• Integration and orchestration Services. Implements SOA standards, Business / Technical orchestration layer, Collects and reports KPI information (BPMS tools, SCA based tool, BPEL tool, Workflow oriented tool, etc.)
• Enterprise Service Management (ESM). Provides monitoring and policy description and enforcement. It controls and gives insight in the SOA exchanges dynamically. Few expensive commercial tools (Amberpoint, Progress, SOA Software) or one open source tool (Microsoft Managed Service Engine) are available on the market.
• Enterprise Service Monitoring. The service monitoring is generally offered in ESMP. Nevertheless some tools are now available to support this functionally at a more moderate pricing than ESMP (like JaxView) or even free (open source tools like Microsoft Managed Service Engine, or WS02 registry future version that will ship with a dashboard supporting Google gadgets).
• Enterprise Business Rules and Events Services. Sustainability comes at a price. Business rules and correlation of business events should be managed apart from the code. Business Rules should be defined, stored, reported, etc. separately from other entities. Business rules should also be defined declaratively and changing one rule definition should not requiring changes in another. Rules should be order-independent.
• Enterprise Repository and/or Registry services. All assets produced and governed should be stored and managed in configuration within a common repository. A registry could also be used during run time for discovering the services or to get some metadata to configure the service.
• Security and Policy management and enforcements services. Enabling security for the SOA platform should be as easy as possible. In order to avoid encapsulating the security inside the service call, it is recommended to use external security policy and dynamic enforcement points. All Identity and Access Management vendors have an offer in this area. Some pure players still exist.
• External Gateway Services. Used to protect the company at the network edge and to optimize message transfer and increase performance when dealing with suppliers. The most know vendors are layer 7 or IBM Websphere Data Power.

Figure  8: mySOA Service Platform

In Table 1, we provide for each mySOA reference platform services: indication of the tool landscape maturity, the maturity of the body of knowledge for implementing those services, if open source solutions exist and give examples of tools we know.

This first step can be done either in code or can be done with XML and XSD. As it turns out a number of enterprise scale projects prefer to take the second option. And for many integration and application development scenarios (not only at the enterprise level) it is customary to negotiate a WSDL/XSD based specification for the web services and then to embark on the actual development of the code that implements that specification. However, dealing with raw XML and WSDL can be very error-prone. WSDL in particular is non-trivial to handle as the original WSDL specification allows room for some complicated constructs and contracts to be defined. We need tools that can enable us to work at this level consistently and reliably and WSCF aims to address this need.

mySOA Platform Services	Tools Maturity	Body of knowledge	Open Source	Examples of tools
Enterprise Messaging Service	High	High	Yes	IBM, Microsoft, Mulesoft, Progress, Sun, Tibco etc.
Master Data Management Service	Medium	High	Yes	IBM, Kalido, Orchestra Networks, SAP, Siperian, Sun, Talend, etc.
Service Configuration Master Data Service	Low	Low	No	You need to build your own!
Semantic Data Integration Service	Medium	Low	No	Progress DataExtend SI, collibra, expressor
Data Distribution Service	High	High	Yes	Informatica, Tibco, Pervasive DataCloud 2 , IBM
Integration and orchestration Service	Medium	Low	Yes	Intallio, Tibco, ActiveBPEL, Oracle, Informatica
Design Time Governance	Low	Medium	Yes	In general bundled with repository. We are doing also quality testing with the Enterprise Service Testing tools.
Enterprise Service Management	Low	Low	Yes	Amberpoint SMS, Microsoft MSE, Progress Actional, SOA Software, etc.
Enterprise Service Monitoring	Medium	Low	Yes	JaxView, Amberpoint SMS,, CA wily, WSO2, etc.
Enterprise Business Rules and Events Service	Low	Low	Yes	Esker, Drools, Tibco Business Event, IBM Ilog, sci-flex
Enterprise Repository and/or Registry service.	Low	Low	yes	Software AG Centrasite, WSO2, Mule Galaxy, IBM WSRR, Adjoovo, Dragon SOA
Security and Policy Mgt service	Medium	Medium	Yes	Amberpoint, Sun, IBM, CA, Oracle, Vordel, layer 7, F5
External Gateway Service	High	High	No	XML appliance (Vordel, layer 7, F5, IBM) or Amberpoint
Enterprise Service Testing	High	High	Yes	SOASta, Parasoft SoaTest, Eviware SoapUI, ITKO Lisa, Actional Diagnostic and SOA Cleaner.

Table  1: mySOA reference Platform Services analysis

Lessons learned on Implementing mySOA Reference platform

Going from slides to real tools working in production is always a challenge. This is true, especially, when the SOA tool market is under consolidation and when standards are numerous and not completely finalized or interoperable.

Lack of interoperability standard

We consider that the major inhibitor to SOA adoption and realization is the lack of appropriate tooling and the lack of interoperability. Today SOA is used by most of the software vendors to push their full stack of software and even sometimes hardware.

On the modeling side, SOAML from OMG has not gained wide acceptance, even if already implemented in major UML tools. Michael Poulin states here, that “SoaML is not about orientation on service and is not fully architecture modeling language because it does not model the architecture entities in full but concentrates on the relationships between them (which is important but not enough)”. Its preference goes to OASIS SOA Reference Model standard and OASIS SOA Reference Architecture standard-draft that defines what SOA is, what Service is and how Service may be used in service-oriented environment.

Nevertheless, we do think that SOAML could be used for interoperability purpose, instead of working with a full stack of WS-* standards. Another interesting approach is to use semantic to the better defined services and their interactions. Current studies on those subjects are still in their infancy (like Semeuse, soa4all EU funded project, etc.).

Lack of tools interoperability

Except SOA Software, Progress, and Software AG (with Centrasite plug-ins), all software vendors tend to focus on their stack first (that’s true for IBM, Tibco, Oracle, Sun, Microsoft). Open source companies are more tempted to be open, but more and more tend to favor the integration with their own tools (OW2, SOAPera, WSO2)

They will all claimed that they support the WS-* stack and WS-I. But this is not enough, and everybody knows it. WS-I use cases are not following quickly enough the evolution of SOA de facto standard stacks. Users are then obliged to test all possible configurations in their own environment.

Microsoft/Sun agreement on interoperability is a good move but is not enough. Within Sun Metro project, WSIT is an extension to JAX-WS and provides interoperability between Java Web services and Microsoft's Windows Communication Foundation. It focuses on enhanced Web service features such as security, reliable messaging, and atomic transactions.

Some new entrants like proxisoft, are proposing an interesting, and for me quite intriguing, approach. They let you develop your java code and from Java Jar files are able to create the services you need. In that case, your services are completely managed independently from the initial code, providing thus a gateway to functionalities.

Invest in semantic data mapping

What if you could create your own Enterprise Data Model with a DSL or a high level modeling language (like UML) that reflects the business objects and services you need? What if each business objects lifecycle and relationships could be described if automatically the data service contract could be generated?

What if then you can deep dive in your current data and map them to this new model from real source. What if you could reuse business rules to reuse business logic and express routing logic as metadata in the database? Don’t you think your life will be easier? Some months ago, there was no solution on the market. Now, several exist and can be used (Progress DataExtend SI,, collibra, expressor).

Leverage EAI and ETL to create the Data Distribution Service

As we already stated, we began by launching initiatives to free data from their silos. We also needed to be flexible enough to be able to accommodate custom logic without a bolt on solution for each need.

This could be done in three complementary ways (see Figure 9): use a centralized MDM, create data services from application, create canonical format to ease data distribution inside and across the enterprise.

Most of companies today already have EAI, ETL in place and the knowledge to use them. Open source suppliers in those domains are plethoric (except for MDM, you have only two – Sun project Mural and Talend MDM), and some or even being offered as SaaS (Duns & Bradstreet Purisma).

When creating access to encapsulated data in an application you need to define at least three interfaces: CRUD, Search and Administration (start, stop, status of data connector). We called those services basic services. This approach could benefit from “Canonical Schema Model” (CSM) between service providers and service consumers. A CSM, by default, is forcing two transformations in the path of any message (both request and response).

Figure  9: ICC Data Services

The combination of model driven MDM and semantic integration tool is clearly a win win situation. For example, if you use Progress software DataExtend SI (data semantic integration) with Orchestra networks Ebx.Platform (master data lifecycle management) and with Informatica Platform (data quality, transport) you can build quickly a powerful solution, partly model driven based.

Do not forget to manage Service Configuration Master Data

Very often services have to be configured based on the client. This configuration has to be managed at the same time as the service. For example, a travel itinerary can be sent via email in HTML or PDF. For each client this information has to be stored somewhere. It is even more critical when external services are used (some capabilities can be different based on their contract). We recommend adding those data in the MDM tool or in a dedicated LDAP directory.

Do not confuse SOA and Integration

SOA is not Integration, though it is sharing technologies with Integration. SOA is about creating services that encapsulate existing systems of record such that new solutions can be developed by consuming these services without creating the need to duplicate information from other systems of record. When information is not duplicated it does not need to be synchronized and replicated. Information management is a wide topic, but in SOA it generally means, aggregating data, adding some semantics, applying business rules and providing rich interfaces.

In a Service Oriented Architecture, the service interface is “the” canonical model (Figure 10). It isolates the service consumers from the systems of records. When a service is well designed, all consumers invoke that particular service, and this service, in turn, invokes all the necessary back-end systems.

This is where SOA is tricky to implement. Tools are not today adapted to manage the necessary flexibility well defined in Jean Jacques Dubray’s article on Message type architecture for SOA. We are still in using integration tools to make SOA, by necessity! So let’s try to build the foundation for the most appropriate mySOA platform.

Figure 10: The service interface is “the” canonical model

Build the mySOA governance workflow

The SOA governance workflow needs to be implemented both in design time and run-time. That’s the first issue, since in general tools do not support both. All SOA software vendors are trying to extend their offer to integrate both. In the meantime, you can decide to implement your validation workflow with BPMS tool or to use the one that is generally integrated within a repository.

Design-time Governance

Our design time governance is based on static checking of:

• The WSDL. We implemented in SOAtest checking rules defined in xpath.
• The XML Schema. We implemented in SOAtest checking rules defined in xpath
• Web service security. We used standard SOAtest checking rules.
• WS-I compliance. We used standard WS-I test tool within SOATest.
• Impact analysis of the new services on existing one.

So our design process is very simple and consists of validating all standards we created and we implemented in SOATest. Results of the test are also stored and managed in configuration with the other assets

For example, one rule verifies that non-whitespace text has been specified for the element for each .

• Standard: All WSDL operations must include a element.
• Enforcement: SOA Test rule: WSDL\CWT.OperationNullDoc
• Error Message: Documentation missing, tags exists for operation with no text.

Rules are then validated within SOAtest using their WSDL Policy enforcer (see Figure 11).

Figure   11: SOATest WSDL Policy Enforcer

Design-time Governance – The Service Catalogue Holy Grail

The service catalogue is the Holy Grail of SOA governance. Most of the SOA software vendors on the market are promoting their own tool (Software AG, IBM, Oracle, Progress, HP, and recently Amberpoint) or are licensing it through OEM (less and less). Some UDDI repositories are client based, some are server based (but nobody will tell you the impact on synchronization), some implements the WSDL tModel, some not. Most of the interoperability testing we did, failed between products. It’s not just because of the software vendors’ implementation; it is also due to flaws in UDDI which is by the way a dead end standard.

SOA Software, HP Systinet and Software AG tried to make their products interoperable with others; with more or less success over time (what is working with one version is not with another one, coopetition is here to stay).

As a side note one can remark that the service catalogue is now part of a wider offer: the repository. A registry alone does not exist anymore in the market, except may be some remaining open source tool like Apache jUDDI. UDDI registry is dead, long live the multi-usage repository. You can create your own artifacts and store them in the repository. Of course, you will find a UDDI interface.

The conclusion: because of an over engineered standard (UDDI), today no solution is really interoperable on the market. So again, the vendor lock-in anti-pattern applies. SOA is not dead, but one should admit that one key part of its model is blown away! I still think that there is a market for company that will be able to provide and maintain repository/registry plug-ins, and sell it on a plug-in store!

Design-time Governance – Software AG Centrasite and Parasoft SOaTest

We did test and had discussions the last two years with Sofware AG. Centrasite version 8.x is the tool that covers nearly all our governance needs, especially concerning the integration with SOAtest (see Figure 13, Figure 14 and Figure 12). But we have to admit, that the business is still not convinced to invest money in a design time repository and our test license finally expired.

Figure   12: The service categorization

Figure   13: SOSTest Plug-in provided by Centrasite

Figure   14: Results of SOAtest design time governance tests are transferred in Centrasite

So, we had to take the poor man’s road and thanks to the open source movement, we had the opportunity to build a solution using Mulesoft Galaxy and Liferay portal (see Figure 15 and Figure 16) that fits at least the service catalogue needs.

Figure   15: ICC Service Catalogue – using the taxonomy of services

Figure   16: ICC Service Catalogue – TravelerProfileSearch.wsdl

Figure   17: ICC Service Catalogue – Documentation generated by WSDLdoc

We used WSDLdoc to document the WSDL and xsd. We also tested Intalio to create the governance workflow that does not exist in Mulesoft Galaxy and it worked well. But we do not think it is our job is to develop SOA tools or to spend time integrating SOA tools. That’s why we were interested by Centrasite and its “plug-ins” (not as easy as in firefox …).

So we will wait for the market to mature, for the price to drop and for the software vendors to make interoperability a reality. Open source tools can make software vendors react. Mulesoft, Microsoft Service Engine and WS02 SOA tools are already providing lots of features for a very good price. I would like to add, that, if all those tools could be offered as portlet (or widget) to be integrated easily in a WebTop (netvibes, Google IG) or in a portal, it will be great.

Run-time Governance – Service Consumer Management

The ICC, as part of its role in the enterprise is responsible for the management of policies and reporting of SLAs for the managed services and business blades.

The service consumer represents the user of a service. A service consumer is always an application GUI or another service. The service consumer can have confidence that any service managed by the ICC will conform to the standards set forth here. Specific responsibilities of the service consumer include:

• Be familiar with the technical integration standards;
• Follow the ICC governance Process when a service in found to be non-compliant or enhancement is desired.

In order to fulfill these responsibilities, the ICC must ensure all services consumers are declared and should report of usage regularly.

In Figure 18, we can see that three policies will be applied by the AmberPoint Proxy:

• Authentication – authenticates the service consumer with the identity stored in the AmberPoint LDAP;
• Credential Mapping – Applies static username and password to authenticate the request with Accenture.  Accenture Authentication is performed using HTTP Basic Authentication;
• Logging – logs the request and response and any faults.

Figure   18: ICC Service Catalogue – Documentation generated by WSDLdoc

In Figure 19, we can see how we can define performance agreement for the services.

 

Figure   19: ICC Runtime Governance – SLA definition

Run-time Governance – The last mile, the epic battle, convince IT-Operations

When you talk of real-time governance, you automatically imply hardware, network, load balancer, etc. You are now entering the IT operation zone. ITIL is everywhere and SOA is not a synonym of CMDB. The last mile is always the most difficult one in your SOA journey. It took us more than a year to deploy the run-time governance tool, and it was mainly due to human resistance. So, never neglect the last mile!

Run-time Governance – SOA virtualization

A very good post from John Michelsen is explaining why service virtualization is important. There are at least three distinct ways that you can use virtualization concepts in SOA:

1.      Hardware virtualization. “This is not a SOA specific thing. This is when you’re running many copies of the operating system within one physical hardware device so that you can get independence of those several virtual machines from each other”.

2.      Virtual service end points. Service virtualization architectures usually revolve around the idea of a service intermediary sitting between the service client and the service implementation. “In a sense what you’re doing is creating a virtual location for your consumers to access in order to invoke the service when in fact you’re completely shielded from the actual end point of the service itself.”

3.      Virtualized Services. “They are especially important to achieving the dream of agile SOA testing: shorter, iterative, requirement-driven test cycles, with testing happening every step of the way. Why? Because if you want to test earlier, you will need to test incomplete components, or “in progress”.

Service virtualization should be offered as a service by the tool you are using.

Run-time Governance –Mediation Service

Mediation services address two different purposes (taken from Combining SOA and EDA using an Enterprise Service Bus from Jean-Louis Maréchaux).

First, the mediation ensures the necessary protocol matching to integrate heterogeneous systems. As two different services do not have to use the same transport protocol, the mediation service takes care of the transformation from one protocol to the other, so that the communication is possible. The protocol switch is transparent for all the participating services of a business transaction.

Second, the mediation offers the possibility to transform the content of any message. This is a key service for business integration. It ensures that the data which transits through the bus is understandable by any process. Moreover, the mediation enables content augmentation to enrich a message with any additional information. The content transformation is managed by the bus: it is transparent for any participating service.

Conclusion

I hope you enjoyed this journey into mySOA. Creating SOA is difficult. Standards and tools are lacking, and their cost is still prohibitive for most small and medium companies. That’s why most of the time cosmetic SOA is done, and creates sometimes some frustrations and disappointments.

Open source suppliers are shaking the ecosystem by offering more and more functionalities for free (or reduced fees) with an on premise or on demand mode.

If open source products are not enough, you cannot wait, and you do not want to invest technically in interoperability test case then having most of the platform services coming from the same vendor stack is probably the best short to mid-term solution (IBM, Sun/Oracle, Microsoft, SAP).

SOA Software, Software AG, Progress, Tibco, Mulesoft, and WSO2 are also able to provide best of breeds module you could integrate in your “proprietary stack” if needed.

Real SOA for all will come when tools will be enable us to be agile, to govern at design and run time our managed services and will contribute to build sustainable IT services and systems.

Credits

I would like to thank my colleagues Bob Donley (St. Louis, USA) and Janusz Szyr (Warsaw, Poland) for their contributions to this work. Without their ideas, without their time, without their different cultural visions of SOA, nothing will have been done and formalized that way. I also want to thank Patrice Simon, our manager, for its inalterable trust and continuous support.

 And finally, I want to thank Claude Mariaccia (IBM IT software Architect), Ghyslaine Ferre Morel (sales engineer at Software AG France), Miko Matsumura (deputy CTO at Software AG), Paul Davidian (Account Executive at Mulesoft), and Thomas Been (Tibco Solutions Consultant) for their support and their help in answering quickly all our technical requests when evaluating their offer and tools.

[1] Sustainable IT Architecture. The Progressive Way of Overhauling Information Systems with SOA, BONNET Pierre,  DETAVERNIER Jean-Michel,  VAUQUIER Dominique,  BOYER Jérôme,  STEINHOLTZ Erik, March 2009, ISTE-Wiley.

[2] Eating the IT Elephant: Moving from Greenfield Development to Brownfield, Richard Hopkins and Kevin Jenkins, May 2008, IBM Press.

1. Back to top

   Pervasive SOA Standards

   Jan 18, 2010 10:56 AM by Bill Jacaruso

   Your article does a great job outlining the different service varieties as well as a process for evaluation, governance and management. The proliferation of “ready available” cloud based services may be the catalyst that SOA has needed to have hyper-growth. Imagine all the different variations of services that will be combined and leveraged. Pervasive software sees the power and widespread adoption of data services (not simply distribution) in the hybrid cloud model. We too look for emerging standards to evolve around interoperability, as this currently is a challenge to “mix” from many vendors. We believe that these standards are the responsibility of the service vendors and Pervasive (cloud.pervasive.com) is dedicated to assisting with their creation.

   Reply

2. Back to top

   more about the benefits ?

   Jan 19, 2010 2:06 AM by Bogdan Nedelcu

   Very detailed piece of work. Could detail a bit on the benefits of such an implementation ? Not IT, rather from the business perspective ?

   Reply

3. Back to top

   Re: more about the benefits ?

   Jan 19, 2010 12:06 PM by William El Kaim

   The first benefit from business is now that I moved to the business organization and left IT ;)
   More seriously, and again that's my own personal opinion not the one of my employer, from the business standpoint, SOA is:
   - enabling IT to answer more quickly and easily business needs, by reusing standardized and interoperable IT technologies available within a unique foundation platform. The upfront cost for SOA IT enablement is not negligible since you need to create architecture and B2B standards, support REST and SOAP and manage SLA. But current tools are offering most of what you need out of the box (Security, integration, monitoring, etc.). So instead of having the business worrying about technology, they could concentrate on business process, client support, advertising and billing.
   - a major facilitator to support specific clients requests' across the globe. The business or leisure travel market world is very fragmented and local niche players are numerous. Faclitating suppliers to join your business ecosystem with the lowest technical entry level is key. This is true both for being able to get more real time data (hotels especially) or services (a client can for example request all their travellers to pay a particular travel insurance for a particular companies when travelling on some regions)
   - is facilitating a business thinking shift from "product = application", to "product = set of features that are offered as services" available across different channels (mobile, portal, etc.). At the beginning, the granularity of services offered is high. Then, with maturity and based on client demands, we can increase the business portfolio by offering several suppliers for the same features. No client is the same.
   - creating new possible revenue streams by addressing niche market (the long tail!)
   - enabling social services and Web 2.0 approaches
   - it also enables what a French company clever Age CEO called reversed plugability. You can integrate your supplier services, but you can also integrate your services (or orchestrated or mashed up) in the suppliers offer and services.
   
   Hope this helps.
   
   - enabling a common business dialect to emerge within and across Business Unit. Master data and business rules are then "normalized" or "globalised"
   - providing a competitive advantage since clients are now asking more and more details about your IT and your IT vision before signing any contract.

   Reply

4. Back to top

   Future tense?

   Jan 20, 2010 5:53 AM by Dave Nicolette

   Excellent write-up! I'm keeping a link to it as a general reference.
   
   I notice most of the description is written in the future tense. Does the article describe a production environment? If so, it would be interesting to hear stories of positive and negative outcomes, with some observations about why the outcomes were either positive or negative. That sort of information can be very helpful in refining the model.
   
   I'm curious to know how the notion of "service blades" actually worked in practice. At a previous employer, we tried a similar approach although we didn't use the same terminology. What happened in that case was that ivory tower silos emerged in the enterprise architecture area, centered on each of the shared IT assets we supported. We built pretty good expertise within each silo pertaining to each product, but collaboration with business application groups was very poor. No business value was ever delivered as a result of the SOA initiative after several years' effort. How are you preventing the "ivory tower" effect in your organization?

   Reply

5. Back to top

   Re: Pervasive SOA Standards

   Jan 23, 2010 3:08 PM by Marney Hurst

   The problem with the vendors you've choses is the cost. the cost of governance for those products is outragous. A very comprehensive or cost effective SOA governance could include tools such as jUDDI, CentraSite for design time govnannce. SOAPUi for testing and JaxView for runtime governance.

   Reply

6. Back to top

   Re: Pervasive SOA Standards

   Jan 26, 2010 5:27 AM by William El Kaim

   We looked at those tools ... As I stated, we did not want to invest in low cost (free tools) and then spend money in extending or integrating those tools (as we intended to do with MulseSource at the beginning and then to offer the development as open source). This approach did not work in my company since it is not our core business. For some companies, it could be a better choice.
   I agree with SoapUI we use it more and more. We always loved this tool. But since we use it also for QA, it considered as a little bit too complex for non IT people coming from Mercury. The rule engine offered in SoaTest is also quite easy to use. SOATest was also enabling testing simply WS-security based service, offering security check on the XML Schema and WSDL and was supporting JMS testing. All that integrated in one place. This choice was done several years ago. I agree that SoaPUI is now a great great contender, that I also recommend.
   Centrasite was my personal preferred choice (even if not perfect ;) ), but even if not really expensive, I did not get the budget for it. WSO2 and MulseSource are gaining traction today. But the ecosystem around centrasite (not the free edition, which is the old platform) was very appealing for me. If only they could have built a kind of app store for all other tools or developers to build services around!
   I also recognize that I discovered JaxView quite recently, thanks to Forrester consulting, and had no opportunity to test it. WSO2 is also proposing tools that can fit this need. But agin, I did not test them.
   For me the major objective was to reuse the technology we had in town (Tibco, BEA) and to try to find the best modules to enable building the SOA foundation platform without doing myself the integration. One of the key reason was to avoid maintenance cost of the interfaces of tools connected to the repository.
   My main message was consider the foundation platform as a whole not as a set of tools to assemble yourselves. If you do it yourselves, it is becoming a new application ..

   Reply

7. Back to top

   Re: Future tense?

   Jan 26, 2010 9:33 AM by William El Kaim

   I can not disclose the details of our implementation state. So I used the future tense in order to not be to precise. Sorry for that.
   
   Service blades are already in place. Service blades are necessary to ensure alignment between business and IT.
   
   The best example is the traveler profile. In our business, the traveler profile is key for enabling booking and reporting:
   - profiles are defined using a unique canonical format, everybody should be able to use it or should go though the sync service.
   - profiles are synchronized with several other external (Global Distribution System like Amadeus or sabre, Central Reservation System like tour operator) and internal systems dynamically in different regions. So, here, it is greatly appreciated by everybody.
   - profile are key for making any bookings and generating revenues, so everybody can contribute to enhance its quality and avoid replication of data.
   - PCI compliance and data privacy prevent people to store or extract those data in an uncontrolled way.
   - we also consider that in the future application will be built using Web 2.0 techniques, and using business blades forces IT to not embed to much in he application code.
   
   Profile Data and their quality are our basic assets (like Oil). They are fueled through secured and agnostic (REST-SOAP) OPEN API business Services (refined oil). Other BU will use them just because services exist, are managed and work 24/24 7/7.
   
   For business support processes, mainly running in silo, like Finance, and HR it is a natural approach but with a shorter scope. For example, HR could fuel the Identity and Access Mgt services.
   
   Creating corporate standards and guidelines to force people to use the interfaces are also needed. Most of the WS governance tool are able to listen the network and show you stealth Web Service.
   
   Give a solution to the business, create standard and governance rules, monitor that people are follwoing the rules. And always be ready to accept exceptions when needed.

   Reply

8. Back to top

   Re: Business Benefits

   Mar 6, 2010 9:53 PM by p k

   There are a lot of business benefits depending on the perspective you are looking at i.e. Reducing Cost, Increasing Revenue, Improving customer satisfaction etc. I have started to detail out some of that in my blog soanami.blogspot.com/2010/02/holistic-soa.html
   
   I am trying to define this from a holistic picture so depending on the specific business benefit you are moving into, the appropriate boxes can be followed and the indicators judged to realize the same. Your feedback is most appreciated on the blog as well

   Reply

• Older >