The code below continues to create threads, even when the queue is empty..until eventually an OutOfMemory exception occurs. If i replace the Parallel.ForEach with a regular foreach, this does not happen. anyone know of reasons why this may happen?

Have you tried running the foreach with a low parallelism degree (like 4, 8, etc.) and seeing if the problem persists?
–
TudorJun 22 '12 at 0:21

That does limit the number of threads created (but i'm not able to keep up w/ the producers). What does this mean about the original code then? Should the TPL pool manage this properly?
–
mike01010Jun 22 '12 at 0:33

How do you instantiate the consumer? Are you doing this in a loop perhaps?
–
Panagiotis KanavosJun 22 '12 at 11:01

3 Answers
3

I think Parallel.ForEach() was made primarily for processing bounded collections. And it doesn't expect collections like the one returned by GetConsumingPartitioner(), where MoveNext() blocks for a long time.

The problem is that Parallel.ForEach() tries to find the best degree of parallelism, so it starts as many Tasks as the TaskScheduler lets it run. But the TaskScheduler sees there are many Tasks that take a very long time to finish, and that they're not doing anything (they block) so it keeps on starting new ones.

I think the best solution is to set the MaxDegreeOfParallelism.

As an alternative, if you are willing to use pre-release code, you could use TPL Dataflow's ActionBlock. The main difference in this case is that ActionBlock doesn't block any threads when there are no items to process, so the number of threads wouldn't get anywhere near the limit.

Internally in the Task Parallel Library, the Parallel.For and Parallel.Foreach follow a hill-climbing algorithm to determine how much parallelism should be utilized for the operation.

More or less, they start with running the body on one task, move to two, and so on, until a break-point is reached and they need to reduce the number of tasks.

This works quite well for method bodies that complete quickly, but if the body takes a long time to run, it may take a long time before the it realizes it needs to decrease the amount of parallelism. Until that point, it continues adding tasks, and possibly crashes the computer.

I learned the above during a lecture given by one of the developers of the Task Parallel Library.

Specifying the MaxDegreeOfParallelism is probably the easiest way to go.

ThreadPool uses hill-climbing, not Parallel.ForEach. That starts as many Tasks as the TaskScheduler will let it run. And since here it seems the Tasks can block for a (relatively) long time, that will be a high number. But yeah, MDOP is probably the best thing to do here.
–
svickJun 23 '12 at 0:06

The Producer/Consumer pattern is mainly used when there is just one Producer and one Consumer.

However, what you are trying to achieve (multiple consumers) more neatly fits in the Worklist pattern. The following code was taken from a slide for unit2 slide "2c - Shared Memory Patterns" from a parallel programming class taught at the University of Utah, which is available in the download at http://ppcp.codeplex.com/