Advanced Message Queue Protocol to Commoditize Messaging

For a while there I wasn't sure if this was ever gonna make it out of JPMChase - so great news guys!

Due to the uncertainty behind AMQP we ended up building two alternatives at Apache to AMQP which are working nicely. I'm sure one day we'll add AMQP support to Apache ActiveMQ too then in addition to REST, Ajax and WS-Notification users can choose the wire protocol that suits their requirements the best.

James
LogicBlaze
Fuse: Open Source SOA

James,
Having multiple standards is not such a good idea and rather defeats the point. I would rather see ActiveMQ implement AMQ than have a whole choice of standards to choose from. Go on, you can do it, I expect to see something tomorrow morning after we've drunk Barcelona dry.

-John- (in Barcelona for TSSJS-E)

I am no messaging expert, but somehow I doubt that AMQP will be the end all be all of messaging protocols. One of the cool things about ActiveMQ is you can use which ever one may suit your needs. This helps keep standards honest and gives a chance for innovation.

Dan Diephouse
Envoi Solutions

The article says that "Implementations of AMQP will also be required to be open and royalty free". I checked the specs and didn't find any obvious (note: I am not fluent in Legalese) language to that effect (line 14 to 25 maybe? but these are among the most convoluted sentences I've ever read).

Could someone comment on this?
Any home (www) identified for that project?

Glad to see the tech part is easier to read ;-)

John,

Stomp actually has gained a fair amount of traction, and I am willing to be sees wider usage than AMQP for a lot of things just because of its extreme simplicity. AMQP has the potential to be more performant, but until we have a chance to really dig into it, who knows.

I;m excited ot have both, to be honest.

-Brian

Wow, I need more coffee before I (attempt to) type.

-Brian

Wow, I need more coffee before I (attempt to) type.

-Brian

It is not the case that implementations have to be open source or royalty free.

The specification itself is free to use - no licencing is required. You can download it and start implementing it.

--Robert (JP Morgan Chase)

I wholeheartly support John's view here.

There needs to be confidence in each implementation of a common standard. Like implementations of TCP/IP, you try to avoid variety in the interpretation of the standard so interop works well. Developers can take pride in their implementations, but don't actually change TCP, which is not perfect but does the job.

A choice of transports for a middleware product just dilutes testing efforts and leaves everyone wondering "which is right", which lowers the end users' confidence.

Confidence in the stability, longevity and reliability of middleware is very important for users -- right up there with Databases and Operating Systems. That's why all three of these need to adhere to standards religiously.

The specification itself is open and royalty free.

Anyone is free to make free or commercial (or even hardware) implementations of AMQP. That's the point.

There will be a protocol community site, but that's still being put together. Please be patient; it should appear on www.amqprotocol.org one day.

Recently there was an article somewhere about banks consuming open products and not giving back. Well JPM has given back to the community - no strings attached (but do read the license). AMQP is the result of many man-years of effort driven by real requirements and involving several companies. Enjoy.

Why did we do it? When a firm has thousands of systems to integrate it just makes sense.

Look out for open project and commercial product announcements from other sources in the near future...

Stomp is a nice conversational command language to front a middleware server.
But I can't write a program in Stomp protocol and *know* that it will work the same when pointed at a different middleware server.

Why? Stomp doesn't specify all the semantics of the middleware; there are a million little questions to answer; how does prefetch work, how is flow control handled, how are async events in the middle of a transaction presented, when something goes wrong what happens to the connection and what state is it left in, if I sent a message and the server acks it did it get to the server or another client, was it guaranteed to be safe stored or was that 'optimised' away, how does message re-delivery work, when a message is allocated across a cluster what are the ordering guarantees, how does a firewall recognise this protocol, if I write an MT program can I still use network traffic shaping, if the protocol resistant to buffer overflow exploits ... this list goes on and on.

AMQP documents the semantics you can depend on (or tries to, we must have missed something). If we have missed something, we have a bug in the protocol - please tell us and we'll correct it.

AMQP was almost a text protocol at one stage; but we wanted to make life easier for hardware acceleration.

Standards often end up benefiting all involved (as long as they cover enough functionality) - so I don't know why anyone would object to this effort. Sounds good.

John,

Yep, I agree, and adding a JMS behavioral spec is something I have actively resisted -- Stomp is designed as a very simple edge protocol. The behavior inside the messaging system is up to the messaging system.

I am looking forward to having some down time to do an AMQP implementation and see how it feels. I really am looking forward to it a lot, and if it turns out to be on about the same order of complexity to implement I'll do a backflip in excitement.

Question about it though, AMQP was developed behind closed doors, once released, what is the specification development model, and who will own it? The copyright seems to be jointly assigned to all the folks who worked on it, in the spec doc.

-Brian

Pass one through the spec -- I like it a lot at first glance!

James,
Having multiple standards is not such a good idea and rather defeats the point.

I would rather see ActiveMQ implement AMQ than have a whole choice of standards to choose from.

Go on, you can do it, I expect to see something tomorrow morning after we've drunk Barcelona dry.

-John- (in Barcelona for TSSJS-E)

As a JMS vendor (SwiftMQ) I really appreciate this effort. A wire protocol for interop is what we need. However, what I'm missing is a standard for destination names. They highly differ between the different messaging vendors and sometimes have semantics attached (like hierachical topic names etc). How will you handle that?

There will be a protocol community site, but that's still being put together. Please be patient; it should appear on www.amqprotocol.org one day.

However, what I'm missing is a standard for destination names. They highly differ between the different messaging vendors and sometimes have semantics attached (like hierachical topic names etc). How will you handle that?

OpenWire is a protocol designed with performance, flexibility and features in mind in the ActiveMQ project rather than interoperability between any client and server from different vendors.

This is really good news but I would like to see some kind of independent interoperability testing group set up to *prove* compatibility. Something like what the Drummond Group does for ASx.

Then I can check a matrix and see that, eg, Product X is known to work with Product Y. Standards adherence is not enough. It is too woolly and open to interpretation.

Kit

Testing compliance is one of the tough areas that the WG is discussing. The whole project is very aware of things that make these efforts fail (the article about the problems with CORBA over on slashdot is very relevant, and what we're trying to avoid). In the end, interop is the ultimate test. You either work with the other implementations that exist, or you don't. Those other implementations which interoperate are controlled by different entities, so that seems like a fair and pragmatic starting point.

AMQP wants to be a reliable, stable, neutral, long term protocol. These are the characteristics users want - but not necessarily the developers working on the middleware.

On Destination Names: it became obvious during the 3 major revision of the protocol and the implementations by the group that the concept of "destination", while convenient is fundamentally broken.

Draw parallels to FedEx; to send a message you get a box (message) apply a stamp (QoS), add an address (routing information), doodle on the box (header based routing) and *give it to FedEx* (or another service - your AMQP implementation). You don't send it to the destination, per se.

Once we had this "ah ha" moment, the whole thing became a lot more elegant and efficient. Its also how IP works and how email works (ish). Read the spec, its interesting.

I do agree that having a nice naming convention for routing information (like peoples home addresses, or email addresses) would be great. Feel free to propose one, but make sure you've implemented it at least once and put it in production first ;-)

To correct James Strachan's point; AMQP is a client-to-broker protocol (but it has also been successfully used for cluster interconnect and WAN bridging). It's also very efficient (not insanely efficient :-)

I should be able to take a PERL AMQP client from my Linux supplier and point it at an AMQP compliant server on Java from my favourite middleware vendor and it should work. Perhaps in the future I can buy a hardware device that implements the AMQP and up-size to that just by changing the connection parameters. Hopefully we can get to full plug and play. History has shown that the industry benefits from that kind of convergence.

Perhaps I might the first person to coin the term "Middleware Enabled Network (MEN)" (like SAN, or MAN), or "Network Messenger".

It should be noted, that according to Gartner, all the small middleware vendors combined have less market share than the largest supplier.

A well implemented standard levels the playing field for smaller suppliers, grows the market and increases the confidence of your potential customers. It also enables everyone to move up the stack and do more interesting things.

AMQP is jointly owned by everyone who contributes to it. If you make a big enough contribution to AMQP that you are invited to sign the copyright and patent licensing contract for AMQP, you get your name on the Copyright statement. The Working Group is careful to manage the IP in AMQP to keep it open.

AMQP, once finalised, will go to a standards organisation, which will be licensed to maintain it and evolve it from then on.

There will be a web portal for open development coming in a couple of weeks at www.amqprotocol.org (and www.amqp.org)

Testing compliance is one of the tough areas that the WG is discussing. The whole project is very aware of things that make these efforts fail (the article about the problems with CORBA over on slashdot is very relevant, and what we're trying to avoid). In the end, interop is the ultimate test. You either work with the other implementations that exist, or you don't. Those other implementations which interoperate are controlled by different entities, so that seems like a fair and pragmatic starting point.

It should be noted, that according to Gartner, all the small middleware vendors combined have less market share than the largest supplier.

Andreas:
Right. And even the JMS standard did not change that dramatically. Before JMS came up, MQ had ~75% market share. I think they are still around 70%. A standard middleware protocol will not change it either.

Floyd,

This was confirmed by various folks in the comments. The statement on licensing requirements in the article ("Implementations of AMQP will also be required to be open and royalty free") is not quite correct. Thanks.

Thanks for the confirmation. Makes more sense and in-line with my understanding of the license. Just suggested Floyd to correct the article.

Thanks for the confirmation. Makes more sense and in-line with my understanding of the license. Just suggested Floyd to correct the article.

Floyd, This was confirmed by various folks in the comments. The statement on licensing requirements in the article ("Implementations of AMQP will also be required to be open and royalty free") is not quite correct. Thanks.

Finally. :)

I should be able to take a PERL AMQP client from my Linux supplier and point it at an AMQP compliant server on Java from my favourite middleware vendor and it should work.

Stomp does, indeed, let different languages work together. For what it's worth, Stomp implements more or less the first version of AMQP, which we designed in early 2005, and rewrote twice more after that. The goal at that stage was to cover the needs of JMS, in a clear wire-level protocol.

But it soon became clear to us that this did not deliver interoperability. JMS has some real flaws, the most severe being its obsession with "destination" even though the semantics of queues and topics are quite different - queues are a message store, and topics are a distribution mechanism.

We threw away the concept of destination and dug deeper into the question, "what does it mean to do messaging?" and came up with a semantic model that seperates the entities that do distribution, and the entities that do storage. The result is the model of AMQP exchanges and queues.

This is the real innovation in AMQP, in my opinion. Wire level protocols are a dime a dozen, and APIs are no big deal. SWIG makes it easy to export an API to lots of languages. All this is fun, especially for vendors, because it captures lots of users and applications.

The goal of AMQP is the opposite - to free the users. And to do this, we needed to define a rule book for interoperability. The AMQP model does this. We've tested it - two teams, building their own brokers and clients, and plugging them together, and we know it works. We know exactly how long it takes a good developer to implement AMQP and we know exactly how to measure whether the results play by the book, or do not.

You'll find that the protocol specifications are extremely lucid in this regard. They define what is needed for interoperability but nothing more. There are some areas that need closure, obviously. And it's possible to make proprietary extensions to the protocol in many places. This is not a bad thing.

The cost of the clarity that AMQP delivers is that it's non-trivial. You can't just telnet to an AMQP broker and try it. It's unlikely that existing brokers can just implement the protocol because AMQP goes deep into the functionality of the broker, it rips open that black box and it says, "we want this piece, here, and that piece there", and the pieces it specifies are sophisticated, complete, nearly a full recipe for a messaging broker.

The upside is that we know this approach works. So the risk of implementing AMQP is almost zero.

We threw away the concept of destination and dug deeper into the question, "what does it mean to do messaging?" and came up with a semantic model that separates the entities that do distribution, and the entities that do storage. The result is the model of AMQP exchanges and queues.

This is the real innovation in AMQP, in my opinion.

Wire level protocols are a dime a dozen, and APIs are no big deal. SWIG makes it easy to export an API to lots of languages. All this is fun, especially for vendors, because it captures lots of users and applications.

The goal of AMQP is the opposite - to free the users. And to do this, we needed to define a rule book for interoperability. The AMQP model does this. We've tested it - two teams, building their own brokers and clients, and plugging them together, and we know it works. We know exactly how long it takes a good developer to implement AMQP and we know exactly how to measure whether the results play by the book, or do not.

You'll find that the protocol specifications are extremely lucid in this regard. They define what is needed for interoperability but nothing more. There are some areas that need closure, obviously. And it's possible to make proprietary extensions to the protocol in many places. This is not a bad thing.

The cost of the clarity that AMQP delivers is that it's non-trivial. You can't just telnet to an AMQP broker and try it. It's unlikely that existing brokers can just implement the protocol because AMQP goes deep into the functionality of the broker, it rips open that black box and it says, "we want this piece, here, and that piece there", and the pieces it specifies are sophisticated, complete, nearly a full recipe for a messaging broker.

If the standard is inadequate to ensure interoperability then it should be tightened up - that is its purpose. Test suites are really only a compromise (albeit a pragmatic one) and should not be used in place of a good standard