Which programming style is easier --- shared memory with semaphores, etc. or distributed memory with message passing --- is often a matter of personal preference. The message passing style fits very well with the object oriented programming methodology, and if a program is already organized in terms of objects it may be quite easy to adapt it for a distributed memory system. When faced with a decision of whether to implement a program in shared memory or distributed memory the outcome is usually based on the amount of information that must be shared by parallel tasks. Whatever information is shared among tasks must be copied from one node to another via messages in a distributed memory system, and this overhead may reduce efficiency to the point where a shared memory system is preferred.
Figure 9: Distributed Memory Parallel Processor.
A PMS diagram of a simple distributed memory parallel processor is shown in Figure 9. On the left is the diagram of a single node, often called a processing element, or PE. The organization of a PE explains how messages are passed from one PE to another. As far as any one processor is concerned, the other processors are simply I/O devices. To send a message to another PE, a processor copies information into a data block in its local memory and then tells its local controller to transfer the information to an external device, much the same way a disk controller in a microcomputer would write a block on a disk drive. In this case, however, the block of data is transferred over the interconnection network to an I/O controller in the receiving node. That controller finds room for the incoming message in its local memory and then notifies the processor that a message has arrived.