What can distributed computing do besides parallel computing?

Question

I have some experiences of working on clusters and grids for parallel computing. But I wonder what distributed computing can do besides parallel computing? If I understand correctly, parallel computing is to solve a common task by multiple processors (computers).

Are the Internet and a intranet considered distributed systems? They don't necessarily involve parallel computing, or solving a common task by multiple computers.

Answer 1

Distributed computing can be seen as a special case of parallel computing with some additional features and restrictions, some of which I'll outline here.

• Multiply IO throughput

  In some settings, e.g. databases, IO is the main limiting factor of performance, e.g. hard drive accesses. By serving requests on many machines, you can parallelise hard drive accesses (not possible in typical many-core machines).

• Avoid slow IO

  Real machines have technical limits regarding quick memory. These limits change constantly, but at any given point in time every machine can only hold so much data in RAM. A distributed system can be scaled so that all data resides in RAM, thus eliminating the need to resort to slower mediums.

• Focus on throughput over response time

  When faced with many small requests already dominated by, say, IO, you can not speed up individual requests anymore. Having multiple nodes serve requests can increase throughput.

• Redundancy

  Many nodes should be able to serve any given request so you can perform load balancing.

• Consistency

  When a requests not only retrieves but also changes data, many nodes may need to be informed of these changes. This is probably the factor with the most impact on performance.

• Failure tolerance

  The system as a whole should be robust against failure of individual nodes, i.e. continue to work (if with reduced throughput). This is not trivial since you can typically not hold all data on all nodes.

• Communication cost

  In a distributed system, nodes typically communicate over some kind of network. Delays in the order of milliseconds (typical response times in Ethernet) are prohibitive for systems/algorithms that require frequent communication. In specialised network implementations, wire length actually becomes a significant factor in delay times even inside a single server farm.

• Security

  If you don't control all aspects of the network connecting your nodes, the inner workings of your system are open to outside attack.

Note that all of these are interconnected.

• Raphael: let us continue this discussion in chat