00:31:45 video length
Bio Dr Mark Little is Technical Development Manager for the JBoss SOA Platform, Red Hat's Director of Standards and representative on the Java Executive Committee. He has over 20 years of experience working in the area of reliable distributed systems. While at Red Hat/JBoss Mark has been the lead of the JBoss ESB and JBoss Transactions products as well as working from the office of the CTO.
I am a Technical Development Manager for Red Hat's SOA platform, which basically means I am involved in all our SOA strategy. I have various groups reporting to me like workflow, transactions, ESB. I am also a Director of Standards, so I am also responsible for participation in W3C groups, OASIS and JCP.
2. Ok, so if I read your name somewhere the one thing that pops up in my mind, the one thing I associate you with is transactions. I think you have a long history in being involved with standards around transactions. Can you give us a little background; can you actually define the term transaction for us, give us a little refresher there?
So "Transaction" is a term that is misused probably a hundred and one different ways in our industry. It's probably better to be a bit more explicit and say it's atomic transactions. An atomic transaction has its history back in the '60s, and it's hard, it's like it's a fault tolerance mechanism. It's based on work that was done around that time in the '60s on spheres of control. Basically an atomic transaction is a sphere of control, it's an activity that guarantees that work done within the scope of that transaction is either all done, or it's not done at all, you get no partial failures.
A typical example would be a bank account system. Suppose you are transferring money from a current account to a high interest account, and one way of doing that without using transactions would be to remove money from the current account, so you are holding the money and then deposit it into the savings account. If you have a crash of the system at doing this, then depending on where the crash happens, you may lose your money. If you take the money out of the current account but it hasn't quite been put into the high interest account yet, where is it? Obviously the bank hopefully would have information about where it is, and they can do some kind of money resolution, but that could take days to actually sort out and meanwhile you haven't got your money, you can't pay your bills. If you are to do that transfer within an atomic transaction, then the atomic transaction would guarantee that if there was a crash the money goes back into the current account, the transaction system would do that, or depending on where the crash is, it would guarantee that it would eventually turn up in the high interest account, hopefully within a matter of seconds or minutes at most. And it would do that itself automatically, there will be no requirement for manual intervention.
3. One of the topics that come up often in discussions about web services is whether or not they need transactions. Maybe we can start by a quick description of what is actually available for web services in the transactions space.
Web services transactions development has been going on for almost as long as web services has been developed. So I started doing work around that in 1999, which is pretty close after SOAP was first released. What we were looking at then and what we have continued to look at over the intervening eight or nine years is actually a number of different approaches for transactions in the web. So, traditional atomic transactions that I described earlier have some in-built assumptions about how they will work and the environment in which they will work, so pretty much they assume they will work in a closely coupled environment that can be trusted and they last for seconds, milliseconds hopefully, but seconds, maybe at the utmost minutes.
On the web those kinds of interactions typically don't happen, you know you might be booking a night out, or buying a book from Amazon and you might be doing that over the course of hours or days. And to do all of that within the scope at the top level of an atomic transaction, just doesn't work. So we were starting with how do we do transactions that are specific for web services or for these long duration interactions. And there was a lot of work that was done back in the '80s and early '90s on what is known as extended transactions.
There is a range of extended transactions. Basically the principle about extended transactions is to relax the very properties that are inherent within an atomic transaction, so if you go and look at the literature then you'll find that another acronym that is put around atomic transactions is also known as ACID transactions. That is ACID - A for atomic, everything happens or nothing happens, C for consistent, the state of the system moves from one consistent state to another, I for isolation, so you can't see dirty data and D for durable, so that if the work happens it is made persistent even if there is a crash, you'll eventually get the same state. Extended transactions relax those properties, so you might relax atomicity, so when an extended transaction or a certain type of extended transaction terminates, you may say "I want to commit but I don't want to commit two out of three of these participants, I want to commit that one, but the other two I actually want to undo". Another extended transaction model might relax isolation. And the reason for relaxing the different properties is to cater for the type of use cases that you want, and that's why there is a lot of different extended transactions models. There is no one model that actually fits everything you could ever want to do.
So that's what we have been doing over the last eight years, we have been looking at extended transaction work that has been done and trying to come up with a way of allowing people to develop extended transaction models that are good for their particular use case, rather than try as a transaction industry has done twenty years prior to this, shoehorn the ACID transaction into absolutely everything, let's have targeted models, targeted implementations, and we have got there. So it has taken eight or nine years to get there but finally in OASIS there's the WSTX technical committee, which has defined a framework, WS-Coordination, which allows you to plug in different intelligences, so this would be the different types of extended transaction models.
Out of the box, the standard provides two extended transaction models, because of the use cases that we currently have that we need to adopt. One is Business Activity, which is for these long running units of work, the other is Atomic Transaction, so despite what I said earlier about atomic transactions not being good for web services, if you can recall that back when web services where first starting and even through to today, people are using them for interoperability, as much, if not actually more than for Internet scale computing.
So, atomic transaction in the WSTX spec is really there for interoperability between heterogonous systems running on closely coupled networks. You could use it across the Internet, there's absolutely nothing to prevent you from doing that, but there are really good reasons why you shouldn't.
The Atomic Transactions spec in WSTX has given us transaction industry interoperability between obviously Red Hat, IBM, Microsoft, and a couple of other companies. All heterogeneous transaction protocols within about a year and a half of the spec's being finalized, probably less actually, whereas if you are looking when we last tried to do this, which was in the OMG within the Object Transaction Service work, that really took us about ten years. So there were definitely benefits for doing it in web services.
Yes, you are right, we did learn from our previous mistakes.
5. So you said that on the Internet you would never use atomic transaction, which is pretty obvious, but I believe some people would claim that even if you have the ideals of building a service oriented architecture then loose coupling becomes a design principle even if you stay within the company's boundaries. Would you also say that if loose coupling is one of your goals, atomic transactions is not a good match for that?
Yes, I would, but some people still want to do it. So you can make recommendations but ultimately if they want to do it then nothing in WS-Atomic Transaction will prevent them.
6. Many people say transactions and SOA, transactions and loose coupling don't mix at all. What are the benefits, how would you actually advocate the usage of those pretty complicated standards to people who use them?
I think some of this comes back to what I said at the start about the use of the word "transaction", a lot of people when they see transaction they immediately assume ACID transactions, two phase commit, database transactions, however they've run into transactions before. And in that case they are right, I would not recommend to customers to use atomic transactions across the Internet or within the corporate firewall if what they are trying to achieve is a service oriented architecture based system.
But if you look at extended transactions, like I said about the relaxation of different properties, you are going to actually see that there are certain ones that are actually good for SOA based applications, they provide you the guarantees that you might want but they don't provide the restrictions that ACID transactions require. So if you actually look at some of the work that we did in WS-CAF, the web services composite application framework, which predated WS-TX, there's one of the transaction models there, that unfortunately we didn't adopt into WS-TX, which is actually much more relevant to SOA based implementations.
It's the WS Business Process Model, and I think we actually started that back in 2003, and since then companies like Microsoft and obviously Red Hat, and other companies, are talking about very similar things where you no longer have this notion of global consistency, there's no notion of "everybody has the same state", because in a large scale system you can't guarantee that. Well, you can guarantee it, but it might take you until the heat death of the universe to make sure that it is the case. And that is exactly what WS-BP did, assumed that there were these domains of consistency and in between them there might be domains of inconsistency, up to a certain level or even fuzzier than that. So for people looking to use transactions in a SOA based environment I would suggest that they don't come into it with the preconceived notion that transaction equals ACID transactions or atomic transaction. "Transaction" is too over-used, there are extended transaction models out there that can be of benefit to your application.
7. One of the things you have briefly touched upon is that it is actually WS-Coordination, and WS-Atomic Transaction and WS-Business Activity, which are essentially the Microsoft-driven standards, that have been incorporated. Is that a correct view of this? Because I actually remember that there was a sort of as usual in the web services, or what used to be in the web services space there was a wall between two different fractions, maybe you can give us some background on that?
So, the work actually on extended transaction being standardized, started back in 1997, in the OMG with Arjuna, the company that was with us at that point, and IBM and a few other companies working on something called the Additional Structuring Mechanisms for the OTS, rolls of the tongue, which is short hand to the CORBA Activity Service. That was developing a framework essentially a pluggable coordinator, where you could add the intelligence for your specific transaction models. If you map that to what we have in WSTX you'll see that there is almost a one to one: the pluggable framework and the OMG spec is essentially WS-Coordination. And the intelligences were the different protocols. So for WS-Transactions there is an equivalent mapping in the OMG spec.
When we actually started to do work on the web services transactions back in the 1999 we were working with IBM on essentially taking this model and adapting it to web services. But you are right, at that point the web services wars between Microsoft and IBM fighting Sun, Oracle and pretty much everybody else, they kicked off, and IBM and Microsoft went their own way, building on this work, and we went our own way again building on the same kind of framework, and what came out of it was WS-T from IBM and Microsoft in 2001, and WS-CAF which came out in 2002. Eventually everybody kind of kissed and made up, and we had the OASIS WSTX TC that formed, but the principle input to that was still WS-T and WS-CAF, despite the fact that I am a co-author on both of those original specs and obviously the standards as well, WS-CAF was still better I think. It was much more SOA based and much less "CORBA with angle brackets", if you like. And it's died a death now, but yes the TX one is the one that we are stuck with.
It is and the BetaMax was the better one. Don't even mention Blue Ray!
So we did add transactions to REST when I was working at HP in 2000, we actually did some work on trying to standardize a transaction-based, REST-based protocol. And we did it and looking back I am not sure if it was a hundred percent REST, I actually think it's probably ninety five percent REST, but anyway. The reason we did that is because we actually had customers who were coming to us and say "This web services stuff is a little bit too new for us at the moment. We are not too sure"; it was Axis 1.0 days, Axis didn't perform very well, I'm not even sure if it does these days.
"We would like to do something about coordinating multiple updates to web servers, we are using HTTP, sorry multiple web servers, we are using HTTP, can you do something for us?" And we did it, we did it for these customers, but it never got progressed and when I left HP it pretty much folded, went on the shelf, because web services were big or at least were getting big, and it hasn't really been any push at transactions back into vanilla web if you want, or REST or HTTP. I think that's because people are diverted towards web services; it's not because I don't think the requirement is no longer there, I have come across a few companies over the last five or six years who have asked similar questions, but they've gone eventually either for not using transactions at all and chancing to luck, or they have gone with web services. So I think the need is there, I don't think it's a huge need, but then there isn't a huge need for transactions anywhere.
I think that it would be nice if there was a standard, I think this is one of the problems with REST over HTTP at the moment, in that although it's a standard there is no generally agreed upon standardization of protocols that might sit on top, like transactions, like group communications, that sort of things. And hopefully if we can put these REST and web services wars behind us and kiss and make up then maybe we can actually take some of the benefits of both systems and standardize things in REST over HTTP. That's some customers are actually crying out for.
I think it's going way too long and I think it's become very polarized in some sectors when it shouldn't have. There are certainly good reasons for using REST over HTTP, so obviously there is a distinction between REST and what I would like to call REST over HTTP, which is one way of doing REST.
There are also good reasons why you might want to use web services. And I think for web services it really is the interoperability and the fact that everybody has got together and we have standardize these high level application protocols that sit on top. I don't think it's an either/or situation, it might be in certain cases, it might be that it really does make sense to use REST everywhere in a particular deployment, but I think in general and if you look back over like forty-odd years of distributive systems development, there has never been a global panacea for distributed systems. One thing does not work well for absolutely all distributed system or component within distributed system like you might come up with.
RPC has worked very, very well since the'70s, and we are revisiting that over the last ten years or so, but RPC isn't dead, people are saying RPC is dead, it is not, if you actually look around RPC is running a lot of backend infrastructural systems that are always going to be RPC-based. So when people come and say "You need to change this to message oriented" that is not going to buy those companies anything, they are happy with what they have got. And I think the REST and web services wars should pretty much just end; let's agree that there are good things and bad things about both, and let's try and use them together if we can. I think they can actually be used well together, the work to merge them together and make them work efficiently together hasn't been done, I think it could be done, and let's just get on with it, let's just stop too much fighting.
11. Could you become a little more specific, so what is good in web services that should be adopted in the RESTful word and what is good in the RESTful world that should be adopted in the web services world? Is there such a list?
Yeah, there isn't a huge need for transactions, but there is a need for transactions. Security, high availability, WSRX for instance, you can build on that to do high availability services. From a REST perspective, the uniform interface does make a lot of sense in many cases. I think one of the problems that we have with web services is WSDL, to be perfectly honest. Certainly when I started doing work around web services back in 1999/2000 WSDL was still in its infancy and when we were developing specs and actually doing implementation in HP, we were developing on SOAP over HTTP and back porting the WSDL afterwards because the WSDL really got in the way, and I think it still does today. So getting away from WSDL and looking at the benefits that a uniform interface can provide rather than a specific interface, and also try not to abuse transports, so saying that SOAP over HTTP is the same as SOAP over TCP/IP because HTTP is most obviously a transport, it's most obviously not a transport.
13. You mentioned that it is now time to go beyond the wars between different facts. I mean that's probably something that has been going on for years in different areas, it's been COM vs. CORBA, and it's been these kind of web services versus that kind of web services and now it's REST versus web services. Given the last debate what would be your suggestion, what do you think should be worked upon, what are the things that we should address to get those two to unite, kiss and make up?
I think that the web services guys, and I kind of include myself in this group, need to realize that certainly interactions across the Internet that are based on HTTP are more likely to be REST based than not. And bridging between web services and HTTP or REST over HTTP I should say, should be addressed in a more efficient manner. We shouldn't try to bastardize HTTP anymore as a transport, we should try and work with it rather than against it, and I don't think we are really doing that at the moment. Web services uses HTTP for a really good reason, and it's so that you can tunnel through firewalls.
And that's the real reason. I was at the first OMG meeting where SOAP was brought to life, and it was one of the debates about why it was there. I think we kind of progressed that we are doing it for the same reasons over the last seven or eight years. But I think that as a web services community we can do more to embrace REST than we are currently doing. I think we should. I don't think there is anything technically that would prevent us from doing that. If you actually look at the way the web works, there is nothing that should prevent us from being able to use true web protocols with nothing else laid on top of them, to actually talk between web servers across different continents.
Like I said before about the extended transactions stuff that's the WS-BP spec that I mentioned which has this notion of loosely coupled domains of consistency with inconsistencies between them, that's kind of very similar in that what happens within the corporate firewall might well be a combination of CORBA, REST, web services, DCOM, Java RMI ... whatever you want. But between the corporate firewalls, I think it should more likely try to be not REST we should actually work with that rather than try and fight it.
14. Would you say that a lot of the arguments that were dismissed within sixty seconds five or six years ago that the REST people made are now being accepted pretty much by everybody in the web services world? I mean I personally see lots of people like you, people who write the standards, who are involved at least conceding that REST is a good solution for many cases? I distinctly remember five years ago when it was said that it was only usable for browsers, for human to web server interaction, no machine to machine could ever possibly work within HTTP. That seems to have changed?
It has changed.
To speak personally I did transactions over REST back in 2000, it's not like I certainly had a epiphany moment six months ago. I think a few other people have been kind of coming around to this, and yes whether you want to say that they suddenly realized that what other people were saying was right or whether they always knew it was right, and they just had different masters at that time who wouldn't let them say what was really going through their mind, I obviously can't speak for everybody.
16. One of the basic principles of web services is this protocol independence, is the independence of particular transport protocol, which is one of the big elements in the sales pitch for web services. Doesn't that make the option of consolidating HTTP the way it was supposed to be used impossible. Is there really a way to consolidate web services in Restful HTTP?
I think there is. I am not necessarily sure that it's SOAP over REST. I think that again going back to what we were doing with transactions on REST back in HP, we were actually working to bridge web services transactions to RESTful transactions. And you could do it. I like to think that if we been allowed to finish it maybe it would have been one of those little gems that would have grown and maybe unified people around that fact that web services and REST can be used together. I do believe that we can do it, I'm not suggesting that it is easy, but if you look back at the amount of time and effort that has been wasted in these fights that we've had from individuals to big corporations, I would like to think that if we'd actually spent that time actually talking and trying to get these things resolved in a reasonable manner we could have been there by now.
17. There are lots of rumors going on at the moment about those two companies that happen to have merged recently, which happen to be JBoss and Red Hat. Can you give us a little bit of background on that? Tell us a little bit about whether everything is just nice and all those rumors are just crazy little things that we shouldn't believe? Are you willing to talk about that?
Red Hat acquired JBoss in July 2006. It definitely wasn't a smooth transition but I certainly didn't expect a smooth transition. But that wasn't because it was JBoss and it was Red Hat, I have been involved in more acquisitions in my career that I care to remember, and none of them have been particularly smooth. So it didn't come as a big surprise to me, I think one of the big problems though which is probably specific to the JBoss/Red Hat acquisition was the culture within JBoss because of its history of having to fight against the man, who was IBM or who was Oracle, it was much more combative than Red Hat. Being assimilated into a company like Red Hat, that had a different kind of culture like that, did cause friction.
I won't go into specifics but things like being a very private company as well, various mailing lists that people would talk within JBoss to each other throughout the whole company and use very colorful language, for a start you probably wouldn't want to do that in a public company anyway, but also it can offend ... the larger the size of the company the more chance it is it would offend somebody. There were those kind of things you have to be careful about what you say, your level of freedom has gone down a bit as a result, but obviously it's a bigger company, it has more money, has a bigger reach, so there are tradeoffs.
I think overall it has been a good thing, I think. Some people have left for one reason or another, sometimes because they didn't like the culture change, others because they wanted to stay with the startup mentality and not go to a big company, so they gone to other startups. There are still the odd culture clash within Red Hat and I think quite a few people, old time Red Hat people see JBoss guys as upstarts and brash and trouble makers, and treat them a bit like that as well and that doesn't go down well with some individuals who are brash and upstarts. They push back. There is a lot of to and throw but I think the analogy is probably teenage son versus forty years old father, JBoss is going through puberty and the father is the one who has been there and done it before.
Very good presentation about Transcations.
siva prasanna kumar P
I think they (BIG Companies) probably wanted to make WS-* some kind of Rocket Science, especially looking at the number of specifications and two different major standardizing agencies (W3C & OASIS in WS-* Space) causes further more problems for standardization itself.
Best part of presentation was Mark mentioning a hand in hand approach of REST and Web Services in general, it really make sense to say that. Enough of REST vs (ALL of the rest), we all know neither REST solves all the problems nor WS-*. Mark was very clear about the fact that both have their goods and bads.
BTW I will be more interested in an interview explaining best things of REST, particularly implementing a real life application using RESTful principles and situations under which RESTful approach is appropriate.
On REST: stop calling it "REST over HTTP". REST without HTTP is just RPC with POX, no? The whole point of REST is to use the "fabric" of the internet -- HTTP (yes and the underlying protocols). Content negotiation, HTTP verbs, response codes, URIs, etc. You can't take the HTTP out of REST.
And REST is an architectural approach, with the Web as the best example of what's possible. But REST does not mandate HTTP. Have you not heard of REST-over-JMS, for instance? Or didn't you catch our WOA article the other week?
"REST does not restrict communication to a particular protocol, but it does constrain the interface between components, and hence the scope of interaction and implementation assumptions that might otherwise be made between components [...]"
And if something has to be supported by Spring and CXF to be meaningful then I think I'll hang up my hat and retire. No disrespect to either of those communities, but for good reasons they can't always be at the forefront of everything.
I did not mean to imply that simply because Spring WS or CXF don't have it, then it's not good enough. But if this has been in development for some years now and major open source projects aren't supporting it, it certainly makes you wonder, doesn't it? You probably feel differently since you have been intimiately involved with it, but for an outsider, it's a bit different.
Re WOA article, you mean the one that quotes, "Resources are manipulated by HTTP verbs (GET, PUT, POST, DELETE)"? Hopefully I'm not taking that out of context. I would love to see what REST-over-JMS looks like too, since I have in fact not heard of it (SOAP-over-JMS I have).
I'm not at all interested in starting a "flame war", I do appreciate your point of view.
Re: Very good presentation about Transcations.
I'd love to see more implementations of WS-TX out there, just as I'd love to see more use of transactions across the board. But transactions are a very hard sell; take it from someone who has been trying to do it for 20+ years: persuading someone that they need something that only shows its benefits when there are failures that don't happen often, is really difficult to do. I think that has affected the take-up of transactions in lots of sections of middleware, not just Web Services. As a die-hard transaction guy, it scares me that there are systems running that really should have transactions (even "just" normal ACID transactions) and don't.
There's also an aspect of complexity here that doesn't help matters: transactions are hard to do right and this often puts people off. This isn't an open-source vs closed-source issue either: there are (were) closed source transaction managers that didn't really cut it because the developers didn't bother working on the "edge cases" (e.g., recovery). But it's the "edge cases" where transactions really give benefit.
As for REST support for something other than HTTP, I know a couple of groups have been talking about JMS (JBoss definitely, I think Jersey). I believe Roy is also still working on waka (en.wikipedia.org/wiki/Waka_(protocol)).
I would agree that the "REST style" can be acheived without HTTP. However again, in practical terms, I don't see it (for now at least) living without HTTP. Waka looks fun though!
Thanks for the replies...
Interesting blue screen behind Mark
Re: Other standards
Commit protocol control
REST Vs. SOAP
REST does not support a contract based approach, so it does not support this kind of interoperability. All I hear is that its lightweight. Well, it lacks a lot of functionality, which is why its lightweight.
I don't mind using REST where it can be beneficial as a lightweight framework.
Transactions - which nobody needs? Most of us not yet, maybe...
It more or less went like this:
-reserve the number for the customer (in one backend service)
-do workflow stuff
-bill the customer
One of the particularities that I remember: they needed to be able to reverse a composite process when it failed at the end, i.e. when billing the customer failed. The number from step 1 had to be freed up again. And: they needed to be able to change the process (business agility) within every x weeks.
Now, I remember my friend complaining that this was nearly impossible, especially the reverse logic. Had they used transactions, they would have been focusing on the happy path instead.
My point: IMHO we are still at the dawn of a new age where service composition will happen (some day). When that day comes, I bet a lot of us will looking for some transactional guarantees.
However, today, for your facebook page all this won't matter I guess; not yet at least...
Transactions for SOA and XTP