Apollo 1.0 Released, Next Generation ActiveMQ
- Share
-
- |
Read later
My Reading List
Apache Apollo 1.0, ActiveMQ subproject, was just released. Apollo's new threading model which is geared to multi-core microprocessors makes it faster, more scalable and more reliable than ActiveMQ and perhaps many other messaging projects.
Apollo 1.0 features:
- Stomp 1.0 wire protocol,
- Stomp 1.1 wire protocol,
- topics and queues,
- queue browsers,
- durable subscriptions for topics,
- reliable messaging,
- JMS API,
- and much more
To demonstrate the raw performance of Apollo, Hiram Chirino, lead developer on Apollo, created a set of STOMP benchmarks to build an equivalent comparison of major STOMP vendors (JBoss HornetQ, ActiveMQ and Apollo). There was some question about the validity of this since most users will use the JMS API.
Thus, Hiram created a new set of JMS benchmark to quell all doubts of Apollo's speed. The benchmarks speak for themselves. Apollo does quite well, and in most scenarios appears to be the clear winner against JBoss HornetQ and Apache ActiveMQ.
Hiram described in the Apache Apollo 1.0 release announcement that Apollo was done as a subproject because it is a radical departure from ActiveMQ, as Apollo was built to address shifts in the processor market to multi core. The current version of ActiveMQ employs complex thread locking that is a bottleneck if you want to take advantage of multi core machines. Since the Apollo subproject has shown that it is a reliable speed demon, it will be likely integrated in ActiveMQ 6.
InfoQ caught up with Hiram to cover the Apache Apollo 1.0 release:
InfoQ: Can you briefly describe what types of optimization were done in Apollo to address the multi-core microprocessors?
Apollo has switched to a fully asynchronous processing model based on the ideas behind libdispatch. This model avoids increased context switching since it uses a small fixed number of threads to process all operations.
Libdispatch powers Grand Central Dispatch from OS X. As part of the Apollo effort Hiram led up another project to come up with a Java like version of libdispatch and Grand Central called HawtDispatch. The HawtDispatch library endeavors to follow the same principles of libdispatch and Grand Central. HawtDispatch is used by Apollo so it can take advantage of multiple cores.
Hiram went on to mention that using HawtDispatch avoids the problems and performance bottlenecks associated with simultaneously locking multiple critical sections. With this multi-core friendly approach processing Apache Apollo doesn't use waits. This approach makes heavy use of atomic compare-and-swap (CAS) processor operations.
InfoQ: What did you learn about the JMS market by doing the benchmarks?
It's great to see that Java enjoys so many high quality open source messaging options. They all implement the same API (and some the same wire protocol) which makes it easier for end users to switch from vendor to vendor without impacting their applications.
InfoQ: What did you learn about the JMS market by doing the benchmarks? It appears the Apollo STOMP does better over all than Apollo OpenWire. Why is that? Many might assume since STOMP is a simpler string oriented protocol and OpenWire is a binary oriented protocol that OpenWire would be the clear winner.
Well, under different conditions like in a bandwidth constrained network, OpenWire might sill be better--I would have to benchmark that case. But it shows that with todays processors there's not a huge difference between parsing a few text headers or a few binary headers. Your time is better spent optimizing for keeping processor caches hot and avoiding processor stalls due to too much synchronization.
InfoQ: Apollo seems to do very well against HornetQ and ActiveMQ. Is it because Apache Apollo takes better advantage of multi-core processors?
Making efficient use the muli-core processors helps, but that can't be the whole story. Perhaps it's also because Apollo also does a really good job at managing the flow control and the size of the memory buffers used as messages move from producers to consumers. Apollo's memory management strategy is to use the smallest possible amount of memory buffering which still provides good throughput. If a message is not going to be consumed right away, it is stored to disk and evicted from memory. Apollo actively monitors the consumer consumption rates to know if it should flow control producers to match the consumer rates or if it should actively swap out the new producer messages. It also actively prefetches persisted messages from storage so that it's ready in memory by the time it's needed for delivery to a consumer. Furthermore, Apollo sends messages to consumer while concurrently attempting to store persistent messages. If the consumer acks the message before the storage operation completes, then the message is never stored and the subsequent message delete operation is not required either. This in turn reduces the disk IO workload.
InfoQ: What are your options for Apollo journaling? It appears from looking at the code that you are using LevelDB or BerkleyDB for the journaling (persistent queues and durable topics). Before Apache ActiveMQ used KahaDB or AMQ Message Store (which was based on Howl, which gets used by JOTM, the open source JTA). Why the different direction and how are you using key-value database as a journal? Also since Apollo depends on LevelDB and/or Berkley DB, it is not pure Java. Right? Must it have JNI libs compiled for each supported OS? Is it the case that Apollo will primarily support Linux (at least initially)? Will there be a pure Java Apollo version or does that even make sense anymore?
In the case of the BerkleyDB (BDB) store implement, we just use BDB APIs to implement the store. BerkleyDB itself employes journalling to provide durability for persistent operations executed against it.
LevelDB indexes is more efficient if you keep the key/values small since the subsequent index compactions will also be small. So, in this store's case, we use the same type of journaling strategy that is in place for ActiveMQ KahaDB store. We first sync all persistent operations to log files. The LevelDB index entries can then remain small as they typically are just pointers back into the journal files. This allows us to do async updates to the LevelDB index to further improve the index performance. Furthermore, the nice thing about LevelDB is that it performs much better at sequential reads and writes than a BTree based indexes and that is the main usage pattern of messaging queues.
(Regarding need for JNI) Apollo also ships with a pure Java implementation of LevelDB and uses the pure Java version when the native version is not available on your platform, for example, Windows. But you can also use the BerkleyDB based message store which is a pure Java implementation too. So while Linux might enjoy some optimizations, the other platforms are not being left out in the cold.
A good benchmark would be Apache Apollo on Linux versus Windows (native journaling versus pure Java).
LevelDB is a fast key-value storage library by Google. It provides ordered mapping from string keys to string values.
Warm welcome to the Apache Apollo project. Congratulations on the 1.0 release.
HornetQ results
by
clebert suconic
The JMS Benchmark is invalid because of two things:
- It's measuring specific use cases without doing any proper configurations on HornetQ. That's pointless! For instance the single-producer case without a consumer: We are optimized for the send-consume. If you don't have a consumer you have to proper configure paging or disable blocking
The single producer case is not syncing properly. ActiveMQ guys think they can optimize the persistence by not syncing or persisting at all if you send and consume. We have identified scenarios where a message loss could occur on cases like this. There are assumptions about clients resending eventually lost messages, what I don't think it's a valid assumption.
If you think you are that fast just run SpecJMS and stop looking for specific use cases that are using questionable optimizations.
Re: HornetQ results
by
Hiram Chirino
I hate to have to rehash our discussions we have been having on HornetQ forums here. But since you brought it up... Please submit a patch/pull request to the jms-benchmark project if you would rather see HornetQ benchmarked with a different configuration.
Secondly, our optimization of concurrently dispatching to consumers and the persistence store is valid, it does not cause message loss. It's not like we are breaking new ground here, there are commercial JMS servers that already use this kind of optimization.
Regards,
Hiram Chirino
FuseSource Corp.
Re: HornetQ results
by
Tim Fox
In case of server failure it can result in persistent messages being seen by consumers, then never seen again even though they were never acknowledged. This breaks the definition of persistent messages as per JMS. If a message is persistent it must be retained by the server until it is acknowledged.
We discussed implementing this "optimisation" for HornetQ and rejected it for the above reasons when I was at Red Hat. It was also discussed for RabbitMQ when I was working with the Rabbit team, and also rejected.
I know of only one other commercial vendor that implements this "optimisation", and that broker is well known to play fast and loose when it comes to spec compliance.
Re: HornetQ results
by
Hiram Chirino
Re: HornetQ results
by
Richard Hightower
Re: HornetQ results
by
Hiram Chirino
That option does not exist yet but it should be easy to add to the next release. Note that the optimization only applies to the persistent message cases which is only half the scenarios in the benchmark.
Re: HornetQ results
by
clebert suconic
These guys should run SpecJMS which is a benchmark meant to enterprise use cases, peer reviewed and not just pointing at specific cases.
Re: HornetQ results
by
Matt Giacomini
I agree with this.
Also, and I know I have said before, none of it matters to many of us until these systems start showing benchmark numbers while clustered.
Re: HornetQ results
by
Andreas Mueller
Oh, JBoss FUD against competitors? Well, nothing new really...
Yep.
I have a question for Hiram. What QoS does the consumer have to consume the message? For me it's valid to let an ack cancel a disk write/sync if the producer waits. However, I don't think that an ack which is send from the consumer after the message has been received is faster than a disk write. A pre-ack like DUPS_OK in JMS might do the trick, however.
Re: HornetQ results
by
Ronald Miura
If a competitor immediately comes saying that you may loss your data, without actually testing the product (so it's not a bug you found, it's a theoretical case), then it's FUD.
It's just like Gavin King blaming Spring for LazyInitializationException, or he saying how eternally 'locked' you are if you use it. FUD
Re: HornetQ results
by
Tim Fox
The problem is on the *consumer* side, not the producer side.
With Apollo, a *consumer* can get to see persistent messages delivered from the server, then, if the server crashes before they are unacknowledged, those persistent messages may disappear forever.
This is clearly in contravention of the JMS 1.1 specification, section 4.7:
"A JMS provider must deliver a PERSISTENT message once-and-only-once. This
means a JMS provider failure must not cause it to be lost..."
Note that it is talking about message "delivery", that's the part between the server and the consumer. Nothing to do with the part between the producer and the server.
Let me highlight the relevant part "**This means a JMS provider failure must not cause it to be lost**". With the Apollo optimisation, such delivered messages can indeed be lost. Here is the sequence of actions if any one is in doubt:
1) Producer sends message to server
2) Server forwards message to consumer immediately without persisting it first.
3) Consumer receives message, and notices it is persistent.
4) Server crashes before consumer acknowledges message
5) Server is restarted
6) Message has disappeared. Hmm, message was persistent, that shouldn't happen.
7) => Apollo is not JMS compliant.
Now, a caveat.
I am not saying the Apollo behaviour is *bad*. Some users may not care that persistent messages can be lost after delivery, and they might welcome the speed boost such an extension brings if implemented by any messaging broker. However it's not JMS compliant.
I suggest Apollo keeps the feature, but keeps it disabled in the default configuration, and documents it as an optimisation that relaxes strict JMS compliance.
Re: HornetQ results
by
Tim Fox
"... if the server crashes before they are acknowledged"
Not
"... if the server crashes before they are unacknowledged"
Re: HornetQ results
by
Samuel Tauil
I appreciate all benchmark posts, let's do this fairly so.
let's check it: www.spec.org/jms2007/results
Re: HornetQ results
by
Samuel Tauil
Re: HornetQ results
by
Andreas Mueller
If is legal if the producer waits until the ack has been done. The message needs to be resend if the broker crashes before the send method returns.
Re: HornetQ results
by
Andy Taylor
@Andreas could you explain your last point a little clearer
Re: HornetQ results
by
Andreas Mueller
In case of a broker crash before the send method returns it leads to the same outcome for both cases (disk write or disk write cancel due to ack) - the message is in doubt. This problem is usually solved by transparent reconnect and in/outbound message deduplication to ensure once-and-only-once delivery.
However, I'm not convinced if that is an optimization in a real world scenario where you have a network between consumer and broker. Except the use case of non-durable subscribers where you pre-ack and thus the ack will be of course faster than a disk write (but persistent messages don't make sense for non-durable subscribers), all other QoS require to send the ack after the message has been consumed so I doubt that this is faster than a disk write.
Re: HornetQ results
by
Tim Fox
I will re-iterate - the issue is not between producer and server, it is between server and consumer - it is about *delivery*.
If a consumer sees a message, that means, by any definition of "exists", that the message exists. It has been seen, it exists. Simple.
If it exists, and is persistent, which it is, then it must obey the rules for persistent messages, in particular.
"A JMS provider must deliver a PERSISTENT message once-and-only-once. This
means a JMS provider failure must not cause it to be lost...
Really, there is no wiggle room here.
Re: HornetQ results
by
Andreas Mueller
Re: HornetQ results
by
Tim Fox
* I have provided a detailed argument as to why the ActiveMQ optimisation is not spec compliant and cited the JMS spec. The argument is as clear as day.
* Everything is run on the same box. No-one would do this in real life. Performance characteristics in this case would be quite different to where a real network and separate machines are involved.
* Has disk write cache been disabled on the box? If not, then you're just writing into a cache, even if you sync. Since no mention has been made of this I will assume it is still enabled. Performance characteristics will be again quite different depending on whether cache is on or off.
* When benchmarking other systems always use the recommended configuration and tuning for performance for that system. Out of the box defaults are not usually tuned for performance.
JBoss FUD? I call bullshit on that one, and its pretty insulting for the JBoss guys. Calling FUD is the tech equivalent of playing the race card. It's divisive, insulting and devalues the importance of tackling true FUD when it occurs.
The truth is, it's just a poor benchmark.
BTW, I used to work for JBoss over a year ago, but now work for SpringSource/VMware. I also used to work for the RabbitMQ team. BTW, the RabbitMQ team also rejected the "optimisation" for RabbitMQ on the same basis, and that's not even a JMS broker!
Personally, I have defended ActiveMQ on previous occasions when other vendors (Fiorano) were spreading FUD against ActiveMQ. How ironic. I would have hoped the ActiveMQ would have learned from that experience. [See Bruce Snyder's blog post of 07/2011 - I can't post a link since infoq rejects the post as spam]
This is not about being partisan, it's about integrity and honesty. It's about engineers devaluing their profession by acting like spray tanned marketeers rather than scientists.
If you want a real, *independent* benchmark, that measures a wide selection of real world messaging use cases, use specJMS. But I already know most vendors won't have the balls to do that since in their view the primary purpose of a benchmark is to create some meaningless statistics for the marketecture pages of their web site, not to objectively measure the performance of their system against others.
Enough with the bad old days of the FUD wars. Take the moral high ground and if your system really is as good as you say it is you have nothing to hide from an independent, well run benchmark.
Re: HornetQ results
by
Tim Fox
Please stop harping on about what the producer sees, it's not relevant.
The inconsistency occurs in view of the *consumer*, they see a message which is persistent, which then disappears without being acked. That's clearly in contravention of the JMS spec, as cited.
We are going around in circles here, so we will have to agree to disagree.
Re: HornetQ results
by
Andreas Mueller
Re: HornetQ results
by
Tim Fox
1) Producer sends message to server
2) Server *immediately* forwards message to consumer without first persisting it.
3) Consumer sees messages and notes that it is persistent. "Ah I have seen persistent message 567, it therefore exists".
4) Server crashes, before consumer can send acknowledgement.
5) Server is restarted
6) Consumer consults section 4.7b of JMS spec "A JMS provider must deliver a PERSISTENT message once-and-only-once. This
means a JMS provider failure ***must not cause it to be lost...***"
7) Hmm, well I saw persistent message 567 before the crash, and I am the only consumer, ERGO, I MUST see it again on recovery.
8) Consumer does not receive message 567!
Unless you are somehow saying that message 567 did not really exist the first time the consumer saw it. (Perhaps it was a ghost? Or did the consumer hallucinate it?) then you MUST accept it was a persistent message and therefore obey the rules for persistent messages as specified in the JMS spec.
Re: HornetQ results
by
Andreas Mueller
...
5a) Producer disconnects due to the server crash while it was within the send call
5b) Message is in-doubt and is being resent
5c) Consumer receives the same unacked message
Now reiterate the loop but consider the message is always written to disk. Although this leads to a duplicate in 5b) which needs to be solve by a deduplication mechanism, the outcome is the same: consumer receives the very same message.
So in both cases you have the very same outcome and thus the optimization is valid.
What you do in your post is to ignore the producer. But just the producer fills the gap in this picture.
My time is limited, sorry. I give you the opportunity to spread one last FUD.
Re: HornetQ results
by
Tim Fox
But more importantly, even if the producer resends the message, it will have a different message id, and therefore will not be the same message as defined by JMS
I.e the consumer will never see message 567 again, it will see message, 876.
However you look at it, the consumer can see "ghost persistent messages" that disappear without being acked.
Re: HornetQ results
by
Andreas Mueller
hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/u...
cheers...
Re: HornetQ results
by
Tim Fox
Troll! Back in your cave. I really should stop feeding you ;)
Re: HornetQ results
by
Andreas Mueller
Re: HornetQ results
by
Samuel Tauil
just a little question that I think you can answer easily:
between your steps 5a and 5b where is the message?
thanks.
Re: HornetQ results
by
Samuel Tauil
It is important to note that to do reliable messaging, your system will be prone to receive duplicate messages. For example, if your sending application dies before it receives the RECEIPT frame, then when it starts up again will probably try to send the same message again which may cause a duplicate message to get placed on the broker.
It means that *probably* my producer have to be implemented this behavior of sending again the message and the broker have a deduplicate mechanism that knows (I don't know how) that I'm sending again the same message.
right?
Re: HornetQ results
by
Andreas Mueller
I'm not affiliated with Apollo. Please ask them directly how they do their things. I just tried to explain JMS behavior ...
Re: HornetQ results
by
Andy Taylor
What you do in your post is to ignore the producer. But just the producer fills the gap in this picture.
I will just reiterate what Tim has said, It is the Brokers responsibility to guarantee delivery not the producer, your main assumption here is that the producer will still be around when the server comes backup, it may have crashed, there may be network issues meaning it cant reconnect or it simply may not be configured to attempt recovery.
Re: HornetQ results
by
Andreas Mueller
Important is that send() blocks until disk write or ack.
It will not work if the producer does not retry sending. In that case you will have message lost. But this is always the case even with disk writes. Indoubt messages may occur and must be handled.
Re: HornetQ results
by
Andy Taylor
It will not work if the producer does not retry sending. In that case you will have message lost. But this is always the case even with disk writes. Indoubt messages may occur and must be handled.
You have hit the nail on the head, you are assuming that the producer will resend the message, if it doesn't the message is lost and breaks JMS guarantees. you say 'after a reconnect the very same message must be send again', I am saying, how is this possible if the producer has crashed or can't reconnect?
Re: HornetQ results
by
Tim Fox
I mentioned this before, but I'll say it again: Even if the producer does resend the message (and there is no guarantee that they do), then the message seen by the consumer will have a different JMS message ID so it's not the same message in the eyes of the JMS specification. (Message ID defines identify in JMS).
So, the producer resending is actually irrelevant. However you slice it, this "optimisation" breaks JMS semantics: Consumers can see JMS persistent messages, identified by message ID which spontaneous disappear after server restart, and haven't been acked.
Re: HornetQ results
by
Andreas Mueller
You have hit the nail on the head, you are assuming that the producer will resend the message, if it doesn't the message is lost and breaks JMS guarantees. you say 'after a reconnect the very same message must be send again', I am saying, how is this possible if the producer has crashed or can't reconnect?
If a producer sends a messages and is within the send call and the broker crashes here, from the producer's point of view it is as if the send did never happen. So because of missing recovery functionality in JMS the producer needs to resend it and may produce functional duplicate messages (different message ids, that is: 4.4.13). I think we can agree on that, right?
What could happen while producer was in send() until the broker crashes? The message could have been stored on disk or not, it could have been consumed by a consumer or not and even an ack could have been processed or not. This is all possible while the producer is blocked in send.
What is the outcome at the consumer? The message might have been received already and a duplicate may occur which is the JMS provider's responsibility to handle it (discard the duplicate). There is no description in the JMS spec how a duplicate must be detected (e.g. via a message id), even hornetq uses its own proprietary message property and allows to identify messages with different message ids as duplicates.
Do you agree with the above? This is *with* disk write *before* sending the message to the consumer. ;-)
And now please explain to me the difference in outcome if disk write and sending to the consumer is done in parallel with a chance that the ack wipes out the disk write before it happens.
(Btw, I was wrong in my statement about message lost in my previous post. No message is lost in any case; the send just never happen but duplicates may occur)
Re: HornetQ results
by
Andy Taylor
Re: HornetQ results
by
Andreas Mueller
And what is the difference in outcome? Are you not able to answer my question?
Re: HornetQ results
by
Andy Taylor
Re: HornetQ results
by
Samuel Tauil
4.7
A client uses a MessageProducer to send messages to a Destination. A MessageProducer is created by passing a Queue or Topic to a session’s createProducer method.
A client also has the option of creating a producer without supplying a destination. In this case, a destination must be input on every send operation. A typical use for this style of producer is to send replies to requests using the request’s JMSReplyTo destination.
A client can specify a default delivery mode, priority, and time-to-live for messages sent by a producer. It can also specify delivery mode, priority, and time-to-live per message.
Each time a client creates a MessageProducer, it defines a new sequence of messages that have no ordering relationship with the messages it has previously sent.
See Section 3.4.9 ”JMSExpiration,” for more information on time-to-live. See Section 3.4.10 ”JMSPriority,” for more information on priority.
Message Delivery Mode
JMS supports two modes of message delivery.
• The NON_PERSISTENT mode is the lowest-overhead delivery mode because it does not require that the message be logged to stable storage. A JMS provider failure can cause a NON_PERSISTENT message to be lost.
• The PERSISTENT mode instructs the JMS provider to take extra care to insure the message is not lost in transit due to a JMS provider failure.
A JMS provider must deliver a NON_PERSISTENT message at-most-once. This means that it may lose the message, but it must not deliver it twice.
A JMS provider must deliver a PERSISTENT message once-and-only-once. This means a JMS provider failure must not cause it to be lost, and it must not deliver it twice.
and if the message is not acknowledged the client just can see the message...
Re: HornetQ results
by
Andreas Mueller
Anyway, to get an advantage out of this the acks need to be very fast which may be the case on a loopback interface. Also keep in mind that auto-acks are being sent after onMessage has been called so processing time must be added too. Pre-acks are only suitable for non-durable subscribers for which persistent messages make no sense.
cheers...
Re: HornetQ results
by
Tim Fox
Here are the facts again:
JMS spec 4.7:
"A JMS provider must deliver a PERSISTENT message once-and-only-once. This
means a JMS provider failure must not cause it to be lost..."
We have all now agreed that it is possible for consumers to see persistent messages which then disappear after a server crash. Those messages will *never* be seen again, even if they are resent from the producer since they will then get a different message ID. JMS message identity is defined by message id. That means that message, as uniquely identified by its id is lost.
QED
Re: HornetQ results
by
Andreas Mueller
Message ids can be disabled, even hornetq uses its own property. Persistent messages are sent once-and-only-once! These are the facts.
Educational Content
Intro to CLP with core.logic
Ryan Senior Jun 18, 2013
Spock: A Highly Logical Way To Test
Howard Lewis Ship Jun 18, 2013
Java Garbage Collection Distilled
Martin Thompson Jun 17, 2013
C++11 The Future is Here
Bjarne Stroustrup Jun 16, 2013
The Big Data Revolution
Claudia Perlich Jun 16, 2013
Hello stranger!
You need to Register an InfoQ account or Login to post comments. But there's so much more behind being registered.Get the most out of the InfoQ experience.
Tell us what you think