BT

Producer/Consumer Processing with TPL Dataflow

| by Jonathan Allen Follow 576 Followers on Jan 27, 2014. Estimated reading time: 1 minute |

In the simplest case, producer-consumer scenarios are quite easy to deal with. The producer pushes messages into a thread-safe queue and the consumer pulls them back out. A thread can be dedicated to each side of the equation and we’re done. At least until the complications arise.

These complications are many fold. Rather than one consumer you may have multiples that need to distribute or broadcast the message to. For performance reasons you may need to batch up requests before performing some operations. You may have multiple steps where the overhead of having a dedicated thread for each step is onerous. Or perhaps the context switching as the message jumps from thread to thread is impacting latency.

These are the kinds of issues that TPL Dataflow was designed to address. Rather than using threads, the lightweight blocks that make up a dataflow use and share thread pool threads as needed. But like any new framework, there is a lot to learn and many of the more advanced options can obscure the basics.

To make it easier to get started, Dave Marini of Taskmatics has written an article titled Going with the Flow: Simplifying Producer/Consumer Processing with TPL Dataflow Structures. Using an order shipping workflow (shown above), Dave introduces the basic blocks that you’ll need for most scenarios.

  • BufferBlock
  • BroadcastBlock
  • BatchBlock
  • TransformManyBlock
  • ActionBlock

The article also covers some of the more easily overlooked pitfalls such as flushing batches and completing workflows.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT