Message Queuing Options for .NET
When building larger scale applications, message queues are often very helpful for both distributing and aggregating workloads. Distributed workloads are a natural fit for message queues, simply having multiple readers attached to the same queue is often enough. Aggregation, usually implemented with one reader and multiple writers, is used to bundle lots of small updates into a large block. This facilitates the use of advanced database techniques such as the use bulk inserts instead of individual insert/update statements.
In the .NET ecosystem there are several options available for message queuing. This article highlights some of the more popular and unique offerings as well as the basic terminology needed to evaluate them.
Queues: Queues are quite easy to understand. One or more writers add messages to the queue and one or more readers remove them in the same order. Queue are designed so that each message is delivered to one and only one reader, though the strength of this guarantee varies by product and the settings used.
Topics: The concept of topics may be new to .NET developers who have only used MSMQ. Unlike a queue, this uses a publish/subscribe mechanism where in each message is available to be read by every subscriber to the topic. The use of topics can be is very convenient for sending control messages, but care must be used to prevent a slow reader from fouling the topic for all other readers. Again, the exact effect of a slow reader varies by product and the settings used.
Transactions: Just like databases, message queues often support transactions. The usual semantic is messages are hidden, but not removed, between the time the message is read and the final commit is received by the server.
Durable Messages: A durable message is one that can survive a system reboot. For some types of messages this may be absolutely essential, while for others the performance cost for durable messages may exceed the benefits.
Service Bus: The term service bus is rather nebulous and it is often used to cover a wide variety of products and features. In the context of message queuing we are referring to products that extend the capabilities of a message queuing library.
Microsoft Message Queuing or MSMQ is a venerable product dating back to the Windows NT 4 era. It offers a wide variety of bindings including C++, COM, straight .NET, and WCF. MSMQ doesn’t support topics, which severely limits the design patterns one can use without a lot of development effort. For this reason developers often consider layering a service bus such as MassTransit or NServiceBus on top of MSMQ. For those who need the upmost in reliability, this product has the ability to support Windows-style distributed transactions.
ActiveMQ from Apache is a popular choice for those looking cross platform support. The primary means of communicating with ActiveMQ is Apache’s OpenWire protocol. This is advertised as their high-performance client with bindings for Java, C, C++, and .NET. Another product called Stomp is recommend for accessing ActiveMQ from Ruby, Perl, Python, PHP, ActionScript/Flash, and Smalltalk. ActiveMQ is fully compliant with JMS 1.1 and J2EE 1.4 and offers both queues and topics. This product supports JMS and XA style transactions.
ZeroMQ is a very lightweight message queuing product. Unlike the other products it doesn’t have a stand-alone server; messages are sent directly from application to application. It is designed to offer very high performance, Mike Hadlow was able to show ZeroMQ runs at least 10 times faster than MSMQ or ActiveMQ using 1K messages. It does this by sacrificing the reliability features built into the other products including durable messages and distributed transactions. Like ActiveMQ this is available for most operating systems and languages.
WebSphere MQ: This is actually a family of messaging products from IBM that include support for “HTTP, REST, File Transfers and JMS”. Formally known as MQSeries, it is the only product on our list that requires a separate commercial license. (MSMQ is part of Windows, the rest are open source.) Dating back to 1992, this product is as old as MSMQ but has been seeing far more active development. It can be found on virtually every server-class operating system still in use and has bindings for everything from COBOL through .NET WCF.
To me, this shows the sad state of affairs in the .NET world when it comes to Message Queuing.
Re: Seems to be missing a few
As for venerable, I was just trying to be polite. What would call a product that is over a decade old and yet has nearly all of the same limitations as its original version?
Apache Qpid and WSO2 MB
Paul Fremantle, CTO, WSO2 and Apache Qpid contributor
Microsoft MQ? Really?
Re: Microsoft MQ? Really?
What would call a product that is over a decade old and yet has nearly all of the same limitations as its original version?
Um, I would call it Visual SourceSafe ;)
Ok, good point, and thanks for updating the article. Part of the fault lies with IBM marketing. They decided to latch on to the "websphere" name, and renamed all their MQ products to use the word "WebSphere". But it's an excellent product.
.NET applications should be using the NMS API for messaging.
It's maintained by the Apache Software Foundation so it's open source and actively maintained. If your favorite messaging solution does not have a NMS implementation, and you want to use it from .NET, push that vendor to implement a NMS API binding! Avoid him locking you in. :)
Microsoft BizTalk Server
If you mentioned IBM WebSphere, the BizTalk must be in this list.
Then maybe it worth to tell about Queues in AppFabric (Windows and Azure).
And the verdict is...
ZeroMQ is lightweight because it is low-level
I write here because you said "It [is fast] by sacrificing the reliability features built into the other products including durable messages and distributed transactions." I believe this unfairly compares ZeroMQ's intentional sparseness with heavier protocols. It is quite simple to build distributed transactions in a lightweight manner on top of ZeroMQ, and it does have durable messages.
Thank you for the article, it opened a door for me to understand how to integrate true messaging into my C# and PHP skillset.
Re: ZeroMQ is lightweight because it is low-level
Chris Mattmann Apr 15, 2014