News

More on Parallel LINQ

Posted by Jonathan Allen on Sep 20, 2007 06:31 AM

Community: .NET
Topics: Performance & Scalability
Tags: PLINQ ,; LINQ

LINQ, or Language Integrated Query, is the poster child of Visual Studio 2008. It allows for a SQL-like syntax to be used for querying any kind of data. Out of the box it supports SQL Server, XML, and in-memory objects and the extensible framework allows developer to support other providers including MySQL, Amazon, and Google Desktop.

Queries are special in that they are inherently parallelizable. While queries can have side effects that prevent parallelization, they are rare. This makes them an ideal candidate for parallel processing libraries like PLINQ.

PLINQ, formally known as Parallel LINQ, operates on XML and in-memory objects. Queries that are executed on a remote server such as LINQ to SQL are obviously not candidates.

To turn a LINQ query into a PLINQ query is surprisingly easy. Just add the call ".AsParallel()" to the end of the data source specified in the From clause. The Where, OrderBy, and Select clauses are automatically changed to call the parallel version, as are any joins.

According to MSDN Magazine, PLINQ can execute queries in one of three modes. In pipeline processing, one thread is used to read the data source and the other threads are used to process the query. The results can be used while this is going on, though the single consumer thread may have difficulty keeping up with the multiple producers. But when the workload is well balanced, this can have a significant reduction in memory consumption.

The second mode is called "stop and go". This is used when the entire result set is needed at one time, such as when calling ToList, ToArray, or when sorting the output. This completes all the processing at once and then turns over all the resources to the consumer thread. This can have better performance than the first method because the need for synchronization between the query threads is reduced.

The final method is called "inverted enumeration". Instead of collecting all the output and processing it on a single thread, the final function called is passed to each thread via the ForAll extension method. This is by far the fastest method, but it requires that the function passed to ForAll be thread safe and preferably free of locks and side effects.

An exception in any PLINQ thread causes all the other threads to terminate. If multiple exceptions occur, they are all bundled into a single MultipleFailuresException with their original stack traces preserved.