How to parallelize a sequential algorithm?

Question

What thought process should one follow to convert a sequential algorithm to a parallel one ? Are there any specific code pattern that can be parallelized. Some patterns that I normally use are:

• Looking for loops that can parallelized.
• Try to break algo in some form of Map-reduce format.
• Looking for producer-consumer pattern.

• David Thornley: This is a more general question, and better suited for programmers.se.
• Andrew Rasmussen: Look for divide-and-conquer, or something like quick-sort where it breaks the problem in half recursively. In that case, every time it splits it in two, and each half isn't affected by the other half, you could parallalize there.
• Jim Clay: Look for dependencies. Dependencies imply serialization. Non-dependencies imply parallelization.

Answer 1

Ian Foster describes four basic steps in his book "Designing and Building Parallel Programs":

1. Partitioning - find possible ways to split the data among the workers as fine-grain as possible.
2. Communication - identify the communication patterns.
3. Agglomeration - reduce the initial partitions to coarse-grain tasks according to the resources available.
4. Mapping - map the tasks onto the processing units.

The book is free online, you can find it here: http://www.mcs.anl.gov/~itf/dbpp/text/book.html

Answer 2

To somewhat formalize what you have already said about looking for loops that can be parallelized, the key concept being exploited is flat data parallelism. A majority of the parallel libraries (that I know of) seem to exhibit this (e.g. Microsoft Task Parallel library for .NET, Microsoft Parallel Patterns Library for C++)

Flat data parallelism happens when you have a chunk of (flat) data (e.g. an array) that you want to perform work on, you then divide up the data into chunks of how many processors you have, does independent work in parallel for each of those chunk, and finally potentially aggregate/combine the result of each chunk to get your final result.

A sample article on this: http://goose.ycp.edu/~dhovemey/spring2011/cs365/lecture/lecture18.html