Mark Little on Transactions, Web Services and REST

1. This is Stefan Tilkov at QCon 2008, and I am interviewing Mark Little. Welcome Mark! Can you tell us a little bit about yourself and what you do?

I am a Technical Development Manager for Red Hat's SOA platform, which basically means I am involved in all our SOA strategy. I have various groups reporting to me like workflow, transactions, ESB. I am also a Director of Standards, so I am also responsible for participation in W3C groups, OASIS and JCP.

2. Ok, so if I read your name somewhere the one thing that pops up in my mind, the one thing I associate you with is transactions. I think you have a long history in being involved with standards around transactions. Can you give us a little background; can you actually define the term transaction for us, give us a little refresher there?

So "Transaction" is a term that is misused probably a hundred and one different ways in our industry. It's probably better to be a bit more explicit and say it's atomic transactions. An atomic transaction has its history back in the '60s, and it's hard, it's like it's a fault tolerance mechanism. It's based on work that was done around that time in the '60s on spheres of control. Basically an atomic transaction is a sphere of control, it's an activity that guarantees that work done within the scope of that transaction is either all done, or it's not done at all, you get no partial failures.

A typical example would be a bank account system. Suppose you are transferring money from a current account to a high interest account, and one way of doing that without using transactions would be to remove money from the current account, so you are holding the money and then deposit it into the savings account. If you have a crash of the system at doing this, then depending on where the crash happens, you may lose your money. If you take the money out of the current account but it hasn't quite been put into the high interest account yet, where is it? Obviously the bank hopefully would have information about where it is, and they can do some kind of money resolution, but that could take days to actually sort out and meanwhile you haven't got your money, you can't pay your bills. If you are to do that transfer within an atomic transaction, then the atomic transaction would guarantee that if there was a crash the money goes back into the current account, the transaction system would do that, or depending on where the crash is, it would guarantee that it would eventually turn up in the high interest account, hopefully within a matter of seconds or minutes at most. And it would do that itself automatically, there will be no requirement for manual intervention.

3. One of the topics that come up often in discussions about web services is whether or not they need transactions. Maybe we can start by a quick description of what is actually available for web services in the transactions space.

Web services transactions development has been going on for almost as long as web services has been developed. So I started doing work around that in 1999, which is pretty close after SOAP was first released. What we were looking at then and what we have continued to look at over the intervening eight or nine years is actually a number of different approaches for transactions in the web. So, traditional atomic transactions that I described earlier have some in-built assumptions about how they will work and the environment in which they will work, so pretty much they assume they will work in a closely coupled environment that can be trusted and they last for seconds, milliseconds hopefully, but seconds, maybe at the utmost minutes. On the web those kinds of interactions typically don't happen, you know you might be booking a night out, or buying a book from Amazon and you might be doing that over the course of hours or days. And to do all of that within the scope at the top level of an atomic transaction, just doesn't work. So we were starting with how do we do transactions that are specific for web services or for these long duration interactions. And there was a lot of work that was done back in the '80s and early '90s on what is known as extended transactions.

There is a range of extended transactions. Basically the principle about extended transactions is to relax the very properties that are inherent within an atomic transaction, so if you go and look at the literature then you'll find that another acronym that is put around atomic transactions is also known as ACID transactions. That is ACID - A for atomic, everything happens or nothing happens, C for consistent, the state of the system moves from one consistent state to another, I for isolation, so you can't see dirty data and D for durable, so that if the work happens it is made persistent even if there is a crash, you'll eventually get the same state. Extended transactions relax those properties, so you might relax atomicity, so when an extended transaction or a certain type of extended transaction terminates, you may say "I want to commit but I don't want to commit two out of three of these participants, I want to commit that one, but the other two I actually want to undo". Another extended transaction model might relax isolation. And the reason for relaxing the different properties is to cater for the type of use cases that you want, and that's why there is a lot of different extended transactions models. There is no one model that actually fits everything you could ever want to do.

So that's what we have been doing over the last eight years, we have been looking at extended transaction work that has been done and trying to come up with a way of allowing people to develop extended transaction models that are good for their particular use case, rather than try as a transaction industry has done twenty years prior to this, shoehorn the ACID transaction into absolutely everything, let's have targeted models, targeted implementations, and we have got there. So it has taken eight or nine years to get there but finally in OASIS there's the WSTX technical committee, which has defined a framework, WS-Coordination, which allows you to plug in different intelligences, so this would be the different types of extended transaction models.

Out of the box, the standard provides two extended transaction models, because of the use cases that we currently have that we need to adopt. One is Business Activity, which is for these long running units of work, the other is Atomic Transaction, so despite what I said earlier about atomic transactions not being good for web services, if you can recall that back when web services where first starting and even through to today, people are using them for interoperability, as much, if not actually more than for Internet scale computing.

So, atomic transaction in the WSTX spec is really there for interoperability between heterogonous systems running on closely coupled networks. You could use it across the Internet, there's absolutely nothing to prevent you from doing that, but there are really good reasons why you shouldn't.

The Atomic Transactions spec in WSTX has given us transaction industry interoperability between obviously Red Hat, IBM, Microsoft, and a couple of other companies. All heterogeneous transaction protocols within about a year and a half of the spec's being finalized, probably less actually, whereas if you are looking when we last tried to do this, which was in the OMG within the Object Transaction Service work, that really took us about ten years. So there were definitely benefits for doing it in web services.

4. So you have some practice now, so maybe it's no wonder it took you less than ten years to start.

Yes, you are right, we did learn from our previous mistakes.

5. So you said that on the Internet you would never use atomic transaction, which is pretty obvious, but I believe some people would claim that even if you have the ideals of building a service oriented architecture then loose coupling becomes a design principle even if you stay within the company's boundaries. Would you also say that if loose coupling is one of your goals, atomic transactions is not a good match for that?

Yes, I would, but some people still want to do it. So you can make recommendations but ultimately if they want to do it then nothing in WS-Atomic Transaction will prevent them.

6. Many people say transactions and SOA, transactions and loose coupling don't mix at all. What are the benefits, how would you actually advocate the usage of those pretty complicated standards to people who use them?

I think some of this comes back to what I said at the start about the use of the word "transaction", a lot of people when they see transaction they immediately assume ACID transactions, two phase commit, database transactions, however they've run into transactions before. And in that case they are right, I would not recommend to customers to use atomic transactions across the Internet or within the corporate firewall if what they are trying to achieve is a service oriented architecture based system.

But if you look at extended transactions, like I said about the relaxation of different properties, you are going to actually see that there are certain ones that are actually good for SOA based applications, they provide you the guarantees that you might want but they don't provide the restrictions that ACID transactions require. So if you actually look at some of the work that we did in WS-CAF, the web services composite application framework, which predated WS-TX, there's one of the transaction models there, that unfortunately we didn't adopt into WS-TX, which is actually much more relevant to SOA based implementations.

It's the WS Business Process Model, and I think we actually started that back in 2003, and since then companies like Microsoft and obviously Red Hat, and other companies, are talking about very similar things where you no longer have this notion of global consistency, there's no notion of "everybody has the same state", because in a large scale system you can't guarantee that. Well, you can guarantee it, but it might take you until the heat death of the universe to make sure that it is the case. And that is exactly what WS-BP did, assumed that there were these domains of consistency and in between them there might be domains of inconsistency, up to a certain level or even fuzzier than that. So for people looking to use transactions in a SOA based environment I would suggest that they don't come into it with the preconceived notion that transaction equals ACID transactions or atomic transaction. "Transaction" is too over-used, there are extended transaction models out there that can be of benefit to your application.

7. One of the things you have briefly touched upon is that it is actually WS-Coordination, and WS-Atomic Transaction and WS-Business Activity, which are essentially the Microsoft-driven standards, that have been incorporated. Is that a correct view of this? Because I actually remember that there was a sort of as usual in the web services, or what used to be in the web services space there was a wall between two different fractions, maybe you can give us some background on that?

So, the work actually on extended transaction being standardized, started back in 1997, in the OMG with Arjuna, the company that was with us at that point, and IBM and a few other companies working on something called the Additional Structuring Mechanisms for the OTS, rolls of the tongue, which is short hand to the CORBA Activity Service. That was developing a framework essentially a pluggable coordinator, where you could add the intelligence for your specific transaction models. If you map that to what we have in WSTX you'll see that there is almost a one to one: the pluggable framework and the OMG spec is essentially WS-Coordination. And the intelligences were the different protocols. So for WS-Transactions there is an equivalent mapping in the OMG spec.

When we actually started to do work on the web services transactions back in the 1999 we were working with IBM on essentially taking this model and adapting it to web services. But you are right, at that point the web services wars between Microsoft and IBM fighting Sun, Oracle and pretty much everybody else, they kicked off, and IBM and Microsoft went their own way, building on this work, and we went our own way again building on the same kind of framework, and what came out of it was WS-T from IBM and Microsoft in 2001, and WS-CAF which came out in 2002. Eventually everybody kind of kissed and made up, and we had the OASIS WSTX TC that formed, but the principle input to that was still WS-T and WS-CAF, despite the fact that I am a co-author on both of those original specs and obviously the standards as well, WS-CAF was still better I think. It was much more SOA based and much less "CORBA with angle brackets", if you like. And it's died a death now, but yes the TX one is the one that we are stuck with.

8. Sounds like a VHS/BetaMax story.

It is and the BetaMax was the better one. Don't even mention Blue Ray!

9. Given that web services support - for some level of support - transactions, would you see this as one of the benefits over REST? You know there is one continuous topic that comes up.

So we did add transactions to REST when I was working at HP in 2000, we actually did some work on trying to standardize a transaction-based, REST-based protocol. And we did it and looking back I am not sure if it was a hundred percent REST, I actually think it's probably ninety five percent REST, but anyway. The reason we did that is because we actually had customers who were coming to us and say "This web services stuff is a little bit too new for us at the moment. We are not too sure"; it was Axis 1.0 days, Axis didn't perform very well, I'm not even sure if it does these days.

"We would like to do something about coordinating multiple updates to web servers, we are using HTTP, sorry multiple web servers, we are using HTTP, can you do something for us?" And we did it, we did it for these customers, but it never got progressed and when I left HP it pretty much folded, went on the shelf, because web services were big or at least were getting big, and it hasn't really been any push at transactions back into vanilla web if you want, or REST or HTTP. I think that's because people are diverted towards web services; it's not because I don't think the requirement is no longer there, I have come across a few companies over the last five or six years who have asked similar questions, but they've gone eventually either for not using transactions at all and chancing to luck, or they have gone with web services. So I think the need is there, I don't think it's a huge need, but then there isn't a huge need for transactions anywhere.

I think that it would be nice if there was a standard, I think this is one of the problems with REST over HTTP at the moment, in that although it's a standard there is no generally agreed upon standardization of protocols that might sit on top, like transactions, like group communications, that sort of things. And hopefully if we can put these REST and web services wars behind us and kiss and make up then maybe we can actually take some of the benefits of both systems and standardize things in REST over HTTP. That's some customers are actually crying out for.

10. What is your opinion about the REST vs. web services war? While we are at it we might as well address it? What is your opinion on that?

I think it's going way too long and I think it's become very polarized in some sectors when it shouldn't have. There are certainly good reasons for using REST over HTTP, so obviously there is a distinction between REST and what I would like to call REST over HTTP, which is one way of doing REST.

There are also good reasons why you might want to use web services. And I think for web services it really is the interoperability and the fact that everybody has got together and we have standardize these high level application protocols that sit on top. I don't think it's an either/or situation, it might be in certain cases, it might be that it really does make sense to use REST everywhere in a particular deployment, but I think in general and if you look back over like forty-odd years of distributive systems development, there has never been a global panacea for distributed systems. One thing does not work well for absolutely all distributed system or component within distributed system like you might come up with.

RPC has worked very, very well since the'70s, and we are revisiting that over the last ten years or so, but RPC isn't dead, people are saying RPC is dead, it is not, if you actually look around RPC is running a lot of backend infrastructural systems that are always going to be RPC-based. So when people come and say "You need to change this to message oriented" that is not going to buy those companies anything, they are happy with what they have got. And I think the REST and web services wars should pretty much just end; let's agree that there are good things and bad things about both, and let's try and use them together if we can. I think they can actually be used well together, the work to merge them together and make them work efficiently together hasn't been done, I think it could be done, and let's just get on with it, let's just stop too much fighting.

11. Could you become a little more specific, so what is good in web services that should be adopted in the RESTful word and what is good in the RESTful world that should be adopted in the web services world? Is there such a list?

Transactions.

12. Which nobody needs.

Yeah, there isn't a huge need for transactions, but there is a need for transactions. Security, high availability, WSRX for instance, you can build on that to do high availability services. From a REST perspective, the uniform interface does make a lot of sense in many cases. I think one of the problems that we have with web services is WSDL, to be perfectly honest. Certainly when I started doing work around web services back in 1999/2000 WSDL was still in its infancy and when we were developing specs and actually doing implementation in HP, we were developing on SOAP over HTTP and back porting the WSDL afterwards because the WSDL really got in the way, and I think it still does today. So getting away from WSDL and looking at the benefits that a uniform interface can provide rather than a specific interface, and also try not to abuse transports, so saying that SOAP over HTTP is the same as SOAP over TCP/IP because HTTP is most obviously a transport, it's most obviously not a transport.

13. You mentioned that it is now time to go beyond the wars between different facts. I mean that's probably something that has been going on for years in different areas, it's been COM vs. CORBA, and it's been these kind of web services versus that kind of web services and now it's REST versus web services. Given the last debate what would be your suggestion, what do you think should be worked upon, what are the things that we should address to get those two to unite, kiss and make up?

I think that the web services guys, and I kind of include myself in this group, need to realize that certainly interactions across the Internet that are based on HTTP are more likely to be REST based than not. And bridging between web services and HTTP or REST over HTTP I should say, should be addressed in a more efficient manner. We shouldn't try to bastardize HTTP anymore as a transport, we should try and work with it rather than against it, and I don't think we are really doing that at the moment. Web services uses HTTP for a really good reason, and it's so that you can tunnel through firewalls.

And that's the real reason. I was at the first OMG meeting where SOAP was brought to life, and it was one of the debates about why it was there. I think we kind of progressed that we are doing it for the same reasons over the last seven or eight years. But I think that as a web services community we can do more to embrace REST than we are currently doing. I think we should. I don't think there is anything technically that would prevent us from doing that. If you actually look at the way the web works, there is nothing that should prevent us from being able to use true web protocols with nothing else laid on top of them, to actually talk between web servers across different continents.

Like I said before about the extended transactions stuff that's the WS-BP spec that I mentioned which has this notion of loosely coupled domains of consistency with inconsistencies between them, that's kind of very similar in that what happens within the corporate firewall might well be a combination of CORBA, REST, web services, DCOM, Java RMI ... whatever you want. But between the corporate firewalls, I think it should more likely try to be not REST we should actually work with that rather than try and fight it.

14. Would you say that a lot of the arguments that were dismissed within sixty seconds five or six years ago that the REST people made are now being accepted pretty much by everybody in the web services world? I mean I personally see lots of people like you, people who write the standards, who are involved at least conceding that REST is a good solution for many cases? I distinctly remember five years ago when it was said that it was only usable for browsers, for human to web server interaction, no machine to machine could ever possibly work within HTTP. That seems to have changed?

It has changed.

15. Could it be that they were just right?

To speak personally I did transactions over REST back in 2000, it's not like I certainly had a epiphany moment six months ago. I think a few other people have been kind of coming around to this, and yes whether you want to say that they suddenly realized that what other people were saying was right or whether they always knew it was right, and they just had different masters at that time who wouldn't let them say what was really going through their mind, I obviously can't speak for everybody.

16. One of the basic principles of web services is this protocol independence, is the independence of particular transport protocol, which is one of the big elements in the sales pitch for web services. Doesn't that make the option of consolidating HTTP the way it was supposed to be used impossible. Is there really a way to consolidate web services in Restful HTTP?

I think there is. I am not necessarily sure that it's SOAP over REST. I think that again going back to what we were doing with transactions on REST back in HP, we were actually working to bridge web services transactions to RESTful transactions. And you could do it. I like to think that if we been allowed to finish it maybe it would have been one of those little gems that would have grown and maybe unified people around that fact that web services and REST can be used together. I do believe that we can do it, I'm not suggesting that it is easy, but if you look back at the amount of time and effort that has been wasted in these fights that we've had from individuals to big corporations, I would like to think that if we'd actually spent that time actually talking and trying to get these things resolved in a reasonable manner we could have been there by now.

17. There are lots of rumors going on at the moment about those two companies that happen to have merged recently, which happen to be JBoss and Red Hat. Can you give us a little bit of background on that? Tell us a little bit about whether everything is just nice and all those rumors are just crazy little things that we shouldn't believe? Are you willing to talk about that?

Red Hat acquired JBoss in July 2006. It definitely wasn't a smooth transition but I certainly didn't expect a smooth transition. But that wasn't because it was JBoss and it was Red Hat, I have been involved in more acquisitions in my career that I care to remember, and none of them have been particularly smooth. So it didn't come as a big surprise to me, I think one of the big problems though which is probably specific to the JBoss/Red Hat acquisition was the culture within JBoss because of its history of having to fight against the man, who was IBM or who was Oracle, it was much more combative than Red Hat. Being assimilated into a company like Red Hat, that had a different kind of culture like that, did cause friction.

I won't go into specifics but things like being a very private company as well, various mailing lists that people would talk within JBoss to each other throughout the whole company and use very colorful language, for a start you probably wouldn't want to do that in a public company anyway, but also it can offend ... the larger the size of the company the more chance it is it would offend somebody. There were those kind of things you have to be careful about what you say, your level of freedom has gone down a bit as a result, but obviously it's a bigger company, it has more money, has a bigger reach, so there are tradeoffs.

I think overall it has been a good thing, I think. Some people have left for one reason or another, sometimes because they didn't like the culture change, others because they wanted to stay with the startup mentality and not go to a big company, so they gone to other startups. There are still the odd culture clash within Red Hat and I think quite a few people, old time Red Hat people see JBoss guys as upstarts and brash and trouble makers, and treat them a bit like that as well and that doesn't go down well with some individuals who are brash and upstarts. They push back. There is a lot of to and throw but I think the analogy is probably teenage son versus forty years old father, JBoss is going through puberty and the father is the one who has been there and done it before.