Message Queuing Options for .NET

Posted by Jonathan Allen on Apr 22, 2011

Sections

Enterprise Architecture,

Architecture & Design,

Development

Topics

WebSphere MQ ,

Websphere ,

Messaging ,

IBM ,

Application Servers ,

Web Services ,

Companies ,

Java ,

.NET ,

SOA ,

Enterprise Architecture ,

Languages ,

MSMQ ,

Programming ,

ZeroMQ ,

Architecture ,

ActiveMQ

When building larger scale applications, message queues are often very helpful for both distributing and aggregating workloads. Distributed workloads are a natural fit for message queues, simply having multiple readers attached to the same queue is often enough. Aggregation, usually implemented with one reader and multiple writers, is used to bundle lots of small updates into a large block. This facilitates the use of advanced database techniques such as the use bulk inserts instead of individual insert/update statements.

In the .NET ecosystem there are several options available for message queuing. This article highlights some of the more popular and unique offerings as well as the basic terminology needed to evaluate them.

Terminology

Queues: Queues are quite easy to understand. One or more writers add messages to the queue and one or more readers remove them in the same order. Queue are designed so that each message is delivered to one and only one reader, though the strength of this guarantee varies by product and the settings used.

Topics: The concept of topics may be new to .NET developers who have only used MSMQ. Unlike a queue, this uses a publish/subscribe mechanism where in each message is available to be read by every subscriber to the topic. The use of topics can be is very convenient for sending control messages, but care must be used to prevent a slow reader from fouling the topic for all other readers. Again, the exact effect of a slow reader varies by product and the settings used.

Transactions: Just like databases, message queues often support transactions. The usual semantic is messages are hidden, but not removed, between the time the message is read and the final commit is received by the server.

Durable Messages: A durable message is one that can survive a system reboot. For some types of messages this may be absolutely essential, while for others the performance cost for durable messages may exceed the benefits.

Service Bus: The term service bus is rather nebulous and it is often used to cover a wide variety of products and features. In the context of message queuing we are referring to products that extend the capabilities of a message queuing library.

Products

Microsoft Message Queuing or MSMQ is a venerable product dating back to the Windows NT 4 era. It offers a wide variety of bindings including C++, COM, straight .NET, and WCF. MSMQ doesn’t support topics, which severely limits the design patterns one can use without a lot of development effort. For this reason developers often consider layering a service bus such as MassTransit or NServiceBus on top of MSMQ. For those who need the upmost in reliability, this product has the ability to support Windows-style distributed transactions.

ActiveMQ from Apache is a popular choice for those looking cross platform support. The primary means of communicating with ActiveMQ is Apache’s OpenWire protocol. This is advertised as their high-performance client with bindings for Java, C, C++, and .NET. Another product called Stomp is recommend for accessing ActiveMQ from Ruby, Perl, Python, PHP, ActionScript/Flash, and Smalltalk. ActiveMQ is fully compliant with JMS 1.1 and J2EE 1.4 and offers both queues and topics. This product supports JMS and XA style transactions.

ZeroMQ is a very lightweight message queuing product. Unlike the other products it doesn’t have a stand-alone server; messages are sent directly from application to application. It is designed to offer very high performance, Mike Hadlow was able to show ZeroMQ runs at least 10 times faster than MSMQ or ActiveMQ using 1K messages. It does this by sacrificing the reliability features built into the other products including durable messages and distributed transactions. Like ActiveMQ this is available for most operating systems and languages.

WebSphere MQ: This is actually a family of messaging products from IBM that include support for “HTTP, REST, File Transfers and JMS”. Formally known as MQSeries, it is the only product on our list that requires a separate commercial license. (MSMQ is part of Windows, the rest are open source.) Dating back to 1992, this product is as old as MSMQ but has been seeing far more active development. It can be found on virtually every server-class operating system still in use and has bindings for everything from COBOL through .NET WCF.

1. Back to top

   RabbitMQ

   by John Downey

   I was surprised to see that RabbitMQ was left out of the products section as they have a pretty solid .NET client including some WCF integration. ZeroMQ on the other hand is really nothing more then a bunch of useful abstractions around raw sockets.

   Reply

2. Back to top

   Re: RabbitMQ

   by Robert Sullivan

   Yes, it wasn't a very comprehensive list. MSMQ "venerable"? Not so much. That title belongs to MQSeries.

   Reply

3. Back to top

   Really?

   by Mark N

   As others said, no mention of RabbitMQ nor MQSeries? And ZeroMQ does not really seem to be any more of message queuing than WCF or Web Services. It is just messaging.
   
   To me, this shows the sad state of affairs in the .NET world when it comes to Message Queuing.

   Reply

4. Back to top

   Seems to be missing a few

   by Ken Wiebke

   TIBCO EMS, IBM MQ, Oracle Adv Queues, Weblogic, Redis, ...

   Reply

5. Back to top

   Re: RabbitMQ

   by Richard Clayton

   Agreed. RabbitMQ is a pretty big player in this space.

   Reply

6. Back to top

   Re: Seems to be missing a few

   by Richard Clayton

   The author was talking about MQ's targeting .NET. I wouldn't call Redis a Message Queue, though it could be used to "back" a Message Queue implementation (RestMQ [www.restmq.com/] being one such implementation). Does Weblogic and IBM MQ support a .NET client?

   Reply

7. Back to top

   Re: Seems to be missing a few

   by Bryan Ross

   Both Weblogic and WebSphere MQ support .NET clients.

   Reply

8. Back to top

   Re: RabbitMQ

   by Jonathan Allen

   Thank you for that link. I was going to include MQSeries but I couldn't find any information on it. I swear it is like IBM doesn't want us to learn about their products.
   
   As for venerable, I was just trying to be polite. What would call a product that is over a decade old and yet has nearly all of the same limitations as its original version?

   Reply

9. Back to top

   Re: RabbitMQ

   by Jonathan Allen

   If you are interested in writing a case study on how you use RabbitMQ (or another message queuing product) in your professional work then please contact me at jonathan@infoq.com . We are always looking for good pieces on how people actually use this stuff.

   Reply

10. Back to top

    Apache Qpid and WSO2 MB

    by Paul Fremantle

    Both Apache Qpid and the (currently beta) WSO2 Message Broker support native .NET clients.
    
    Paul Fremantle, CTO, WSO2 and Apache Qpid contributor

    Reply

11. Back to top

    Microsoft MQ? Really?

    by Harry Lord

    I think the information in this post is terribly out of date. MSMQ was not included in SQL Server 2008 and was only added back when a small loud group complained. Reporting Services and StreamInsight have replaced MQ. Please be sure to have updated information before suggesting a product.

    Reply

12. Back to top

    Re: Microsoft MQ? Really?

    by Robert Sullivan

    This article is about Message Queues, not about Reporting Services or event managers, especially as they are tied to SQL Server. The products in this article are standalone queue engines. You're comparing add-on services with standalone products. And note the author doesn't suggest MSMQ, noting that it is a bit dated and doesn't support modern features such as topics.

    Reply

13. Back to top

    Re: RabbitMQ

    by Robert Sullivan

    What would call a product that is over a decade old and yet has nearly all of the same limitations as its original version?

    
    
    Um, I would call it Visual SourceSafe ;)
    
    Ok, good point, and thanks for updating the article. Part of the fault lies with IBM marketing. They decided to latch on to the "websphere" name, and renamed all their MQ products to use the word "WebSphere". But it's an excellent product.

    Reply

14. Back to top

    .NET applications should be using the NMS API for messaging.

    by Hiram Chirino

    If your an application developer that needs to use a messaging API, why lock yourself into a single vendor's API? Instead, use the NMS API. It's an abstract messaging API which has several vendor implementations so that you can easily switch your application between vendors as needed. It currently support ActiveMQ, STOMP, MSMQ, Tibco EMS, and even WCF messaging servers.
    
    It's maintained by the Apache Software Foundation so it's open source and actively maintained. If your favorite messaging solution does not have a NMS implementation, and you want to use it from .NET, push that vendor to implement a NMS API binding! Avoid him locking you in. :)

    Reply

15. Back to top

    Microsoft BizTalk Server

    by Leo Gan

    Microsoft BizTalk Server is a great implementation of the message server.
    If you mentioned IBM WebSphere, the BizTalk must be in this list.
    
    Then maybe it worth to tell about Queues in AppFabric (Windows and Azure).

    Reply

16. Back to top

    And the verdict is...

    by Morgoth Melkor

    Unless you spend your life making something like RabbitMQ work with .Net, there is nothing really useful. MSMQ.. You must be joking. Why not include Biztalk then?

    Reply

17. Back to top

    ZeroMQ is lightweight because it is low-level

    by Jared Smith

    Your mention of ZeroMQ caught my attention. I investigated that site and was delighted to find something for which I have been searching a long time. I devoured The Guide and will be using ZeroMQ going forward in all of my experimental programming, until I know it well enough to introduce it to my team.
    
    I write here because you said "It [is fast] by sacrificing the reliability features built into the other products including durable messages and distributed transactions." I believe this unfairly compares ZeroMQ's intentional sparseness with heavier protocols. It is quite simple to build distributed transactions in a lightweight manner on top of ZeroMQ, and it does have durable messages.
    
    Thank you for the article, it opened a door for me to understand how to integrate true messaging into my C# and PHP skillset.
    
    -Jared

    Reply

18. Back to top

    Re: ZeroMQ is lightweight because it is low-level

    by rajendar reddy

    www.4microsoftsolutions.com/post/Message-Queue-...

    Reply