When I should I use message passing over shared memory?

Question

I know a little about how to use message passing and shared memory, but I still don't know when to choose one or another.

Can you tell me about their traits (not the implementation)?

• Oded: Anything specific? This is not a small subject.

Answer 1

The quick summary:

Shared memory parallelism (e.g., OpenMP) is strictly more powerful than message passing (e.g., MPI) as you can simulate message passing with a shared memory system but not vice versa. However, it's also much more difficult to scale up the shared memory model (it needs very elaborate and expensive hardware once you go to more than one system board) and it is also much more difficult to analyze due to the number of possible interactions. The message passing model is considerably simpler to understand — you just have a stream of messages coming in that you want to process — and it maps much more nicely to what is actually possible with networks so it obviously can scale up massively.

Answer 2

From the point of view of how you actually design systems with them, they are in some sense mathematical duals, as this adage by Joe Armstrong, one of the original designers of Erlang, nicely demonstrates:

With shared memory, you communicate by sharing data, with messaging, you share data by communicating.