Should developers write their own transaction coordination logic?

Mark Little is the Director of Standards and Development Manager for the JBoss division of Red Hat. He spent most of his career working on distributed computing and distributed transaction protocols. He has been involved in several working groups charted to define a Transaction Protocol for Web Services including BTP, WS-CAF and WS-TX.

In December, Mark blogged on "Large-Scale Distributed Transactions". He explains that as he worked in the past on replication protocols, he came to the conclusion that you need to trade consistency for performance and availability. This is in line with what Werner Vogels explained in his talk at QCon London 2007.

Mark argues that:

It turns out that the same is true for transactions: in fact, it's necessary in Web Services if you want to glue together disparate services and domains, some of which may not be using the same transaction implementation behind the service boundary.

He points out that other people such as Pat Helland are also thinking around the same line of relaxing transactionality. Mark is using the Heisenberg's Uncertainty Principle to illustrate his point:

you can either know the state that all participants will have, but not when; or vice versa.

He also points us to the WS-BP specification as part of the WS-CAF (Composite Application Framework) set of specifications:

One key difference between the business process transaction model and that of traditional 2PC is that it assumes success, that is the BP model is optimistic and assumes the failure case is the minority and can be handled or resolved offline if necessary, or through replay/void/compensation, but not always automatically, often requiring human interaction.

This approach is echoed in the example that Pat uses to illustrate his paper:

Consider a house purchase and the relationships with the escrow company. The buyer enters into an agreement of trust with the escrow company. So does the seller, the mortgage company, and all the other parties involved in the transaction.

When you go to sign papers to buy a house, you do not know the outcome of the deal. You accept that, until escrow closes, you are uncertain. The only party with control over the decision-making is the escrow company. This is a hub-and-spoke collection of two-party relationships that are used to get a large set of parties to agree without use of distributed transactions.

Pat adds:

When you consider almost-infinite scaling, it is interesting to think about two-party relationships. By building up from two-party tentative/cancel/confirm (just like traditional workflow).

Greg Pavlik, also co-author of the many Web Services Transaction specifications, is starting to think differently:

the application may subsume the role of the coordinator at which point it becomes less and less clear why the protocol is needed at all above and beyond the business logic itself.

There are patterns here that need to be understood by application developers. Perhaps supported by frameworks. But it may simply be that transaction management won't play a broad role in that context.

Mark responded, that yes,

[He doesn't] see distributed ACID transactions having much of a future in large scale systems

but,

There's still a reliable coordinator in there that controls the state transitions and can "do the right thing" on failure and recovery.

Last summer, Juval Löwy –Chief Architect of IDesign.net- suggested a similar conclusion in a podcast:

when several software agents are involved in performing a common unit of work, reinventing ad hoc transaction protocols to achieve state alignment is not necessarily a good idea.