When designing a machine the architect must make a decision to use a relatively small number of powerful processors or a large number of simple processors to achieve the desired performance. The latter approach is often termed massively parallel. At one extreme are the systems built by Cray Research Inc. that consist of two to sixteen very powerful vector processors; at the other extreme are arrays of tens of thousands of very simple processors, exemplified by the CM-1 from Thinking Machines Corporation, which has up to 65,536 single-bit processors. The motivation for a small number of powerful processors is that they are simpler to interconnect and they lend themselves to an implementation of memory organizations that make the systems relatively easy to program. On the other hand such processors are very expensive to build, power and cool. Some architects have used commodity microprocessors which offer great economies of scale at the expense of more complex interconnection strategies. With the rapid increase in the power of microprocessors, arrays of a few hundred processors have the same theoretical peak performance of the fastest machines offered by Cray Research Inc.
This section of the book explores the design space of high performance machines, including single processor vector machines, parallel systems with a few powerful processors, and massively parallel architectures. Section 3.1 introduces a popular taxonomy of parallel systems. The next three sections cover major concepts of parallel systems organization, including pipelining, schemes for interconnecting processors and memories, and scalability. Section 3.5 is a survey of the major types of parallel machines, with an emphasis on systems that have been used in computational science. Finally, Section 3.6 present models for analyzing the performance of parallel computer systems.