MuleSoft Announces Next Generation Integration Platform-as-a-Service

Enterprise deployments had SQL backend & RPC/RMI access. Cloud applications do not have that access, because of shared data & NoSQL backends.

Naturally, integration is the "only" standard way to get data in & out of the applications. So this segment is bound to grow. But it is also important for the cloud applications to adopt REST as a first class citizen for even for its own user interface to counter the data lock-in threat & be open to iPaaS: manidoraisamy.blogspot.com/2008/06/jsr-286-now-...

Mani:

could you explain how REST helps integration? Integration is about synchronization and replication based on pub/sub and orchestration patterns. I have never seen REST being strong in these kinds of scenarios. If you are talking about HTTP and JSON as a protocol and neutral data format, why not. Other than that, I don't see how REST principles help in any way.

Orchestration patterns are iPaaS problems. I am talking about endpoints (i.e. cloud applications) to enable integration:

Significant number of enterprise deployments use database-to-database integration. Not SOA. The reason is simple: SQL is a common minimum denominator. Every access that you have done in your application is available to the integration/ETL tools.

In the absence of database access on cloud, i am suggesting that the same be repeated in the middle tier: Dont spit a single line of HTML code from the server. Let the server's standard communication mechanism be REST (JSON or XML), irrespective of its own user interface or iPaaS.

Some operations might be too fine fine-grained for iPaaS. But thats how SQL is to ETL tools. It doesnt need to be perfect. It needs to be practical. Integration is futuristic problem - Not a problem that you can design it once, provide API and be done with it.

This way, every feature that you add in your application will be available for integration automatically. Thats the only way to be a truly open "aaS" to counter the lock-in threat, IMO.

I see:

a) REST = Data Access Layer (as opposed to Service Access Layer) I think this is true
b) we should be porting the mistakes and inefficiencies of the past to the Cloud. Is that what you call "practical"?
c) "Integration is futuristic problem - Not a problem that you can design it once, provide API and be done with it. " do you understand how to manage the future in the present? do you understand that without a versioning strategy the future is doomed. Is that enough of a reason to be "practical"?

REST has no contract. Is that a way, your way to manage integration?

d) "every feature that you add in your application will be available for integration automatically." Could you elaborate? I am not quite sure I see how things a "automatically available"

b) we should be porting the mistakes and inefficiencies of the past to the Cloud. Is that what you call "practical"?

c) "Integration is futuristic problem - Not a problem that you can design it once, provide API and be done with it. " do you understand how to manage the future in the present? do you understand that without a versioning strategy the future is doomed. Is that enough of a reason to be "practical"?

REST has no contract. Is that a way, your way to manage integration?

d) "every feature that you add in your application will be available for integration automatically." Could you elaborate? I am not quite sure I see how things a "automatically available"

Mani:

again, I am not sure why architecturally a UI bound service interface would be "automatically" reused for integration scenarios. The patterns are very different. In integrations things are more event driven, or plain batch load. I don't see the benefit of using these UI bound interfaces, they are very inefficient for integration and may not contain the information necessary for synchronizing with another system. Integration is not just about "emulating" a user typing at a keyboard.

I think in general you are not talking about "integration". Not surprising coming from somebody who things REST is such a great thing.

An again, if it is true that you cannot predict the future, you can prepare yourself today such that some class of changes in the interfaces will not disrupt existing integration scenarios if they don't have to. Having a non breaking versioning strategy is paramount to a successful integration strategy, in the cloud or within the enterprise. Something that the RESTafarians never quite understood.

I agree that UI bound services tend to be fine-grained (although not as fine-grained as "user typing at a keyboard", especially with rich clients). But, it is a good fallback, when nothing is available.

I am not a RESTafarian. But i am disappointed with SOA specs that have not helped enterprises to move beyond CSV uploads & database updates, due to its sheer masochistic attitude.

1) I do think it's even easier to support service versioning with schemaless REST than SOAP (you version URIs). Strict XML Schemas make WSDL + SOAP a fragile combination for integration, because XML messages are rigid (so fragile) and you need to "change the world" only to adapt to a new XML element in your messages

On the other hand...

2) I do think neither SOAP (call it RPC) nor REST (call it resource-oriented) styles have much to do with actual integration problems. Integration happens at the technical services layer, while architectural styles applies at the business services layer.

I still think integration is about asynchronous event detection and processing, That's why fine-grained, flexible solutions like Mule or Apache Camel are growing in their adoption. IMHO the REST thing is very interesting but its about application development than integration of legacy systems, while SOAP is a competing paradigm that almost failed because of the unmanageable complexity of the WS-* babel.

Having worked on EAI projects for a couple of years, I'd say that any solid integration infrastructure needs asynchronous queuing. Maybe not for all cases, but for sensitive services the integration becomes fragile. Synchronous integration (like REST and SOAP with no extra measures) works nicely as long as the sun is shining. Once some central service goes down (including ESBs), things get hairy.

I've seen too many naive synchronous WS integrations without timeout handling. REST doesn't come with a solution right out of the box. A naive programmer can screw that up too.

I tried to get my head around the Atom protocols as described in Webber et al "REST in practice", which define pub-sub mechanism The amount of plumbing code and the complexity of the protocols is appalling IMO.

Just because integration infrastructure needs asynchronous queuing, endpoints should not & can not change. Integration middlewares are here to adapt to application data format & protocol. Not to create more headaches.

It is not the infrastructure, which requires asynchronous communication, it is the integration itself. Endpoints cannot be agnostic of this, because it implies a different data processing strategy. No abstraction possible. Asynchronous integration is a means of decoupling, which avoids headaches by the way.

Coupling all services synchronously in an enterprise is total madness. Decoupling the creation time of data from its usage time, through buffering in queues for instance, makes a service/application network a lot more robust.

So, how you are going to implement transaction rollback, when this asynchronous "robust" endpoint communicates back an error?

I avoid distributed transactions, because they tighten the coupling even more, and use a compensation strategy instead.

I consider synchronous service collaboration with distributed transactions across the web as a serious anti pattern.

So, who will implement compensating transaction in the application? And how are you planning to implement it?

Well, I am not going into "how do I implement a compensation mechanism" question here. What kind of answer would you expect from me? Such discussions takes weeks and doesn't make sense on an abstract level.

My point was to emphasize that I see no stable large scale integration without asynchronous communication and that async requires a different kind of processing on the service and the client side. Feel free to disagree with me and try using two phase commit instead. That's up to you. Good luck with that.

That's because, your "robust" implementation's devil is in that detail:
You should be changing your application datamodel to store ChangeSummary of data. Assuming that your integration endpoint is coarse grained, the changesummary will span across multiple entities. Reapplying changesummary into the transactional data, that is modified concurrently by users in the meantime, is what database has been trying to implement for years with transaction logs.

But, if you are one of those super SOA architects, who feel that you will get this right inside your application & this is less headache, pls go ahead. But, pls dont make generic statements like "synchronous communications are fragile" for mortals like me. I would rather be happy implementing idempotent REST calls.