BT

InfoQ Homepage News Producer/Consumer Processing with TPL Dataflow

Producer/Consumer Processing with TPL Dataflow

Bookmarks

In the simplest case, producer-consumer scenarios are quite easy to deal with. The producer pushes messages into a thread-safe queue and the consumer pulls them back out. A thread can be dedicated to each side of the equation and we’re done. At least until the complications arise.

These complications are many fold. Rather than one consumer you may have multiples that need to distribute or broadcast the message to. For performance reasons you may need to batch up requests before performing some operations. You may have multiple steps where the overhead of having a dedicated thread for each step is onerous. Or perhaps the context switching as the message jumps from thread to thread is impacting latency.

These are the kinds of issues that TPL Dataflow was designed to address. Rather than using threads, the lightweight blocks that make up a dataflow use and share thread pool threads as needed. But like any new framework, there is a lot to learn and many of the more advanced options can obscure the basics.

To make it easier to get started, Dave Marini of Taskmatics has written an article titled Going with the Flow: Simplifying Producer/Consumer Processing with TPL Dataflow Structures. Using an order shipping workflow (shown above), Dave introduces the basic blocks that you’ll need for most scenarios.

  • BufferBlock
  • BroadcastBlock
  • BatchBlock
  • TransformManyBlock
  • ActionBlock

The article also covers some of the more easily overlooked pitfalls such as flushing batches and completing workflows.

Rate this Article

Adoption
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

BT

Is your profile up-to-date? Please take a moment to review and update.

Note: If updating/changing your email, a validation request will be sent

Company name:
Company role:
Company size:
Country/Zone:
State/Province/Region:
You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.