BT

Producer/Consumer Processing with TPL Dataflow

by Jonathan Allen on Jan 27, 2014 |

In the simplest case, producer-consumer scenarios are quite easy to deal with. The producer pushes messages into a thread-safe queue and the consumer pulls them back out. A thread can be dedicated to each side of the equation and we’re done. At least until the complications arise.

These complications are many fold. Rather than one consumer you may have multiples that need to distribute or broadcast the message to. For performance reasons you may need to batch up requests before performing some operations. You may have multiple steps where the overhead of having a dedicated thread for each step is onerous. Or perhaps the context switching as the message jumps from thread to thread is impacting latency.

These are the kinds of issues that TPL Dataflow was designed to address. Rather than using threads, the lightweight blocks that make up a dataflow use and share thread pool threads as needed. But like any new framework, there is a lot to learn and many of the more advanced options can obscure the basics.

To make it easier to get started, Dave Marini of Taskmatics has written an article titled Going with the Flow: Simplifying Producer/Consumer Processing with TPL Dataflow Structures. Using an order shipping workflow (shown above), Dave introduces the basic blocks that you’ll need for most scenarios.

  • BufferBlock
  • BroadcastBlock
  • BatchBlock
  • TransformManyBlock
  • ActionBlock

The article also covers some of the more easily overlooked pitfalls such as flushing batches and completing workflows.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT