Table Of ContentParallel Computer Architecture
A Hardware / Software Approach
David Culler
University of California, Berkeley
Jaswinder Pal Singh
Princeton University
with Anoop Gupta
Stanford University
Morgan Kaufmann is pleased to present material from a preliminary draft of Parallel
Computer Architecture; the material is (c) Copyright 1997 Morgan Kaufmann Publishers. This
material may not be used or distributed for any commercial purpose without the express
written consent of Morgan Kaufmann Publishers. Please note that this material is a draft of
forthcoming publication, and as such neither Morgan Kaufmann nor the authors can be held
liable for changes or alterations in the final edition.
8/28/97 DRAFT: Parallel Computer Architecture 1
2 DRAFT: Parallel Computer Architecture 8/28/97
Preface Although parallel computing has a long and rich academic history, the close coupling with com-
modity technology has fundamentally changed the discipline. The emphasis on radical architec-
tures and exotic technology has given way to quantitative analysis and careful engineering trade-
offs. Our goal in writing this book is to equip designers of the emerging class of multiprocessor
systems, from modestly parallel personal computers to massively parallel supercomputers, with
an understanding of the fundamental architectural issues and the available techniques for
addressing design trade-offs. At the same time, we hope to provide designers of software systems
for these machines with an understanding of the likely directions of architectural evolution and
the forces that will determine the specific path that hardware designs will follow.
The most exciting recent development in parallel computer architecture is the convergence of tra-
ditionally disparate approaches, namely shared-memory, message-passing, SIMD, and dataflow,
on a common machine structure. This is driven partly by common technological and economic
forces, and partly by a better understanding of parallel software. This convergence allows us to
focus on the overriding architectural issues and to develop a common framework in which to
understand and evaluate architectural trade-offs. Moreover, parallel software has matured to the
point where the popular parallel programming models are available on a wide range of machines
and meaningful benchmarks exists. This maturing of the field makes it possible to undertake a
quantitative, as well as qualitative study of hardware/software interactions. In fact, it demands
such an approach. The book follows a set of issues that are critical to all parallel architectures –
communication latency, communication bandwidth, and coordination of cooperative work -
across the full range of modern designs. It describes the set of techniques available in hardware
and in software to address each issue and explores how the various techniques interact. Case
studies provide a concrete illustration of the general principles and demonstrate specific interac-
tions between mechanisms.
Our final motivation comes from the current lack of an adequate text book for our own courses at
Stanford, Berkeley, and Princeton. Many existing text books cover the material in a cursory fash-
Morgan Kaufmann is pleased to present material from a preliminary draft of Parallel Computer Architecture; the
ion, summarizing various architectures and research results, but not analyzing them in depth.
material is (c) Copyright 1997 Morgan Kaufmann Publishers. This material may not be used or distributed for any
Others focus on specific projects, but fail to recognize the principles that carry over to alternative
commercial purpose without the express written consent of Morgan Kaufmann Publishers. Please note that this
approaches. The research reports in the area provide sizable body of empirical data, but it has not
material is a draft of forthcoming publication, and as such neither Morgan Kaufmann nor the authors can be held
yet been distilled into a coherent picture. By focusing on the salient issues in the context of the
liable for changes or alterations in the final edition.
technological convergence, rather than the rich and varied history that brought us to this point,
we hope to provide a deeper and more coherent understanding of the field.
Motivation for the Book
Intended Audience
Parallel computing is a critical component of the computing technology of the 90s, and it is likely
to have as much impact over the next twenty years as microprocessors have had over the past
We believe the subject matter of this book is core material and should be relevant to graduate stu-
twenty. Indeed, the two technologies are closely linked, as the evolution of highly integrated
dents and practicing engineers in the fields of computer architecture, systems software, and appli-
microprocessors and memory chips is making multiprocessor systems increasingly attractive.
cations. The relevance for computer architects is obvious, given the growing importance of
Already multiprocessors represent the high performance end of almost every segment of the
multiprocessors. Chip designers must understand what constitutes a viable building block for
computing market, from the fastest supercomputers, to departmental compute servers, to the indi-
multiprocessor systems, while computer system designers must understand how best to utilize
vidual desktop. In the past, computer vendors employed a range of technologies to provide
modern microprocessor and memory technology in building multiprocessors.
increasing performance across their product line. Today, the same state-of-the-art microprocessor
is used throughout. To obtain a significant range of performance, the simplest approach is to
Systems software, including operating systems, compilers, programming languages, run-time
increase the number of processors, and the economies of scale makes this extremely attractive.
systems, performance debugging tools, will need to address new issues and will provide new
Very soon, several processors will fit on a single chip.
opportunities in parallel computers. Thus, an understanding of the evolution and the forces guid-
ing that evolution is critical. Researchers in compilers and programming languages have
8/29/97 DRAFT: Parallel Computer Architecture 3 4 DRAFT: Parallel Computer Architecture 8/29/97
addressed aspects of parallel computing for some time. However, the new convergence with com- and software are more significant and of a wider scope. In general, we are looking at performance
modity technology suggests that these aspects may need to be reexamined and perhaps addressed as the machine and the program scale. There is no way to scale one without the other example.
in very general terms. The traditional boundaries between hardware, operating system, and user Chapter 3 discusses how scaling interacts with various architectural parameters and presents a set
program are also shifting in the context of parallel computing, where communication, schedul- of benchmarks that are used throughout the later chapters.
ing, sharing, and resource management are intrinsic to the program.
Chapters 5 and 6 provide a complete understanding of the bus-based multiprocessors, SMPs, that
Applications areas, such as computer graphics and multimedia, scientific computing, computer form the bread-and-butter of modern commercial machines beyond the desktop, and even to
aided design, decision support and transaction processing, are all likely to see a tremendous some extent on the desktop. Chapter 5 presents the logical design of “snooping” bus protocols
transformation as a result of the vast computing power available at low cost through parallel com- which ensure that automatically replicated data is conherent across multiple caches. This chapter
puting. However, developing parallel applications that are robust and provide good speed-up provides an important discussion of memory consistency models, which allows us to come to
across current and future multiprocessors is a challenging task, and requires a deep understand- terms with what shared memory really means to algorithm designers. It discusses the spectrum of
ing of forces driving parallel computers. The book seeks to provide this understanding, but also design options and how machines are optimized against typical reference patterns occuring in
to stimulate the exchange between the applications fields and computer architecture, so that bet- user programs and in the operating system. Given this conceptual understanding of SMPs, it
ter architectures can be designed --- those that make the programming task easier and perfor- reflects on implications for parallel programming.
mance more robust.
Chapter 6 examines the physical design of bus-based multiprocessors. Itdigs down into the engi-
neering issues that arise in supporting modern microprocessors with multilevel caches on modern
Organization of the Book busses, which are highly pipelined. Although some of this material is contained in more casual
treatments of multiprocessor architecture, the presentation here provides a very complete under-
The book is organized into twelve chapters. Chapter 1 begins with the motivation why parallel standing of the design issues in this regime. It is especially important because these small-scale
architectures are inevitable based on technology, architecture, and applications trends. It then designs form a building block for large-scale designs and because many of the concepts will
briefly introduces the diverse multiprocessor architectures we find today (shared-memory, mes- reappear later in the book on a larger scale with a broader set of concerns.
sage-passing, data parallel, dataflow, and systolic), and it shows how the technology and architec-
tural trends tell a strong story of convergence in the field. The convergence does not mean the end Chapter 7 presents the hardware organization and architecture of a range of machines that are
to innovation, but on the contrary, it implies that we will now see a time of rapid progress in the scalable to large or very large configurations. The key organizational concept is that of a network
field, as designers start talking to each other rather than past each other. Given this convergence, transaction, analogous to the bus transaction, which is the fundamental primitive for the designs
the last portion of the chapter introduces the fundamaental design issues for multiprocessors: in Chapters 5 and 6. However, in large scale machines the global information and global arbitra-
naming, synchronization, latency, and bandwidth. These four issues form an underlying theme tion of small-scale designs is lost. Also, a large number of transactions can be outstanding. We
throughout the rest of this book. The chapter ends with a historical perspective, poviding a show how conventional programming models are realized in terms of network transactions and
glimpse into the diverse and rich history of the field. then study a spectrum of important design points, organized according to the level of direct hard-
ware interpretation of the network transaction, including detailed case studies of important com-
Chapter 2 provides a brief introduction to the process of parallel programming and what are the mercial machines.
basic components of popular programming models. It is intended to ensure that the reader has a
clear understanding of hardware/software trade-offs, as well as what aspects of performance can Chapter 8 puts the results of the previous chapters together to demonstrate how to realize a global
be addressed through architectural means and what aspects much be addressed either by the com- shared physical address space with automatic replication on a large scale. It provides a complete
piler or the programmer in providing to the hardware a well designed parallel program. The anal- treatment of directory based cache coherence protocols and hardware design alternatives.
ogy in sequential computing is that architecture cannot transform an O(n2) algorithm into an
O(nlogn)algorithm, but it can improve the average access time for common memory reference Chapter 9 examines a spectrum of alternatives that push the boundaries of possible hardware/
patterns. The brief discussion of programming issues is not likely to turn you into an expert par- software trade-offs to obtain higher performance, to reduce hardware complexity, or both. It
allel programmer, if you are not already, but it will familiarize you with the issues and what pro- looks at relaxed memory consistency models, cache-only memory architectures, and software
grams look like in various programming models. Chapter 3 outlines the basic techniques used in based cache coherency. This material is currently in the transitional phase from academic
programming for performance and presents a collection of application case studies, that serve as research to commercial product at the time of writing.
a basis for quantitative evaluation of design trade-offs throughout the book.
Chapter 10 addresses the design of scable high-performance communication networks, which
Chapter 4 takes up the challenging task of performing a solid empirical evaluation of design underlies all the large scale machines discussed in previous chapters, but was deferred in order to
trade-offs. Architectural evaluation is difficult even for modern uni-processors where we typi- complete our understanding of the processor, memory system, and network interface design
cally look at variations in pipeline design or memory system design against a fixed set of pro- which drive these networks. The chapter builds a general framework for understanding where
grams. In parallel architecture we have many more degrees of freedom to explore, the hardware costs, transfer delays, and bandwidth restrictions arise in networks. We then look at a
interactions between aspects of the design are more profound, the interactions between hardware variety of trade-offs in routing techniques, switch design, and interconnection topology with
8/29/97 DRAFT: Parallel Computer Architecture 5 6 DRAFT: Parallel Computer Architecture 8/29/97
respect to these cost-performance metrics. These trade-offs are made concrete through case stud-
ies of recent designs.
Given the foundation established by the first ten chapters, Chapter 11 examines a set of cross-cut-
ting issues involved in tolerating significant communication delays without impeding perfor-
mance. The techniques essentially exploit two basic capabilities: overlapping communication
with useful computation and pipelinng the communication of a volume of data. The simplest of
these are essentially bulk transfers, which pipeline the movement of a large regular sequence of
data items and often can be off-loaded from the processor. The other techniques attempt to hide
the latency incured in collections of individual loads and stores. Write latencies are hidden by
exploiting weak consistency models, which recognize that ordering is convey by only a small set
of the accesses to shared memory in a program. Read latencies are hidden by implicit or explicit
prefetching of data, or by look-ahead techniques in modern dynamically scheduled processors.
The chapter provides a thorough examination of these alternatives, the impact on compilation
techniques, and quantitative evaluation of the effectiveness. Finally, Chapter 12 examines the
trends in technology, architecture, software systems and applications that are likely to shape evo-
lution of the field.
We believe parallel computer architecture is an exciting core field whose importance is growing;
it has reached a point of maturity that a textbook makes sense. From a rich diversity of ideas and
approaches, there is now a dramatic convergence happening in the field. It is time to go beyond
surveying the machine landscape, to an understanding of fundamental design principles. We have
intimately participated in the convergence of the field; this text arises from this experience of ours
and we hope it conveys some of the excitement that we feel for this dynamic and growing area.
8/29/97 DRAFT: Parallel Computer Architecture 7 8 DRAFT: Parallel Computer Architecture 8/29/97
CHAPTER 1 Introduction ..................................................................................19
1.1 Introduction ..............................................................................................................................19
1.2 Why Parallel Architecture ........................................................................................................22
1.2.1 Application Trends...........................................................................................................23
1.2.2 Technology Trends...........................................................................................................30
1.2.3 Architectural Trends.........................................................................................................32
1.2.4 Supercomputers................................................................................................................36
1.2.5 Summary..........................................................................................................................38
1.3 Convergence of Parallel Architectures .....................................................................................40
1.3.1 Communication Architecture...........................................................................................40
1.3.2 Shared Memory................................................................................................................44
1.3.3 Message-Passing..............................................................................................................50
1.3.4 Convergence.....................................................................................................................53
1.3.5 Data Parallel Processing...................................................................................................56
1.3.6 Other Parallel Architectures.............................................................................................59
1.3.7 A Generic Parallel Architecture.......................................................................................62
1.4 Fundamental Design Issues ......................................................................................................63
1.4.1 Communication Abstraction.............................................................................................64
1.4.2 Programming Model Requirements.................................................................................64
1.4.3 Naming.............................................................................................................................66
1.4.4 Ordering...........................................................................................................................67
1.4.5 Communication and Replication......................................................................................68
1.4.6 Performance.....................................................................................................................69
1.5 Concluding Remarks ................................................................................................................73
1.6 References ................................................................................................................................76
1.7 Exercises ...................................................................................................................................85
CHAPTER 2 Parallel Programs .........................................................................89
2.1 Introduction ..............................................................................................................................89
2.2 Parallel Application Case Studies .............................................................................................90
2.2.1 Simulating Ocean Currents..............................................................................................91
2.2.2 Simulating the Evolution of Galaxies..............................................................................93
2.2.3 Visualizing Complex Scenes using Ray Tracing..............................................................94
2.2.4 Mining Data for Associations...........................................................................................94
2.3 The Parallelization Process .......................................................................................................95
2.3.1 Steps in the Process..........................................................................................................96
2.3.2 Parallelizing Computation versus Data..........................................................................102
2.3.3 Goals of the Parallelization Process...............................................................................103
2.4 Parallelization of an Example Program ..................................................................................104
2.4.1 A Simple Example: The Equation Solver Kernel...........................................................104
2.4.2 Decomposition...............................................................................................................105
2.4.3 Assignment.....................................................................................................................110
2.4.4 Orchestration under the Data Parallel Model.................................................................111
2.4.5 Orchestration under the Shared Address Space Model..................................................111
9/4/97 DRAFT: Parallel Computer Architecture 9
2.4.6 Orchestration under the Message Passing Model...........................................................119
2.5 Concluding Remarks ..............................................................................................................125
2.6 References ...............................................................................................................................126
2.7 Exercises .................................................................................................................................127
CHAPTER 3 Programming for Performance ...................................................131
3.1 Introduction .............................................................................................................................131
3.2 Partitioning for Performance ..................................................................................................133
3.2.1 Load Balance and Synchronization Wait Time..............................................................134
3.2.2 Reducing Inherent Communication................................................................................140
3.2.3 Reducing the Extra Work................................................................................................144
3.2.4 Summary.........................................................................................................................144
3.3 Data Access and Communication in a Multi-Memory System ..............................................145
3.3.1 A Multiprocessor as an Extended Memory Hierarchy...................................................145
3.3.2 Artifactual Communication in the Extended Memory Hierarchy..................................147
3.4 Orchestration for Performance ...............................................................................................149
3.4.1 Reducing Artifactual Communication............................................................................149
3.4.2 Structuring Communication to Reduce Cost..................................................................156
3.5 Performance Factors from the Processors’ Perspective ..........................................................161
3.6 The Parallel Application Case Studies: An In-Depth Look ....................................................164
3.6.1 Ocean..............................................................................................................................165
3.6.2 Barnes-Hut......................................................................................................................170
3.6.3 Raytrace..........................................................................................................................175
3.6.4 Data Mining....................................................................................................................179
3.7 Implications for Programming Models ...................................................................................182
3.8 Concluding Remarks ..............................................................................................................188
3.9 References ...............................................................................................................................189
3.10 Exercises .................................................................................................................................191
CHAPTER 4 Workload-Driven Evaluation ......................................................195
4.1 Introduction .............................................................................................................................195
4.2 Scaling Workloads and Machines ...........................................................................................198
4.2.1 Why Worry about Scaling?.............................................................................................199
4.2.2 Key Issues in Scaling......................................................................................................202
4.2.3 Scaling Models...............................................................................................................202
4.2.4 Scaling Workload Parameters.........................................................................................208
4.3 Evaluating a Real Machine .....................................................................................................209
4.3.1 Performance Isolation using Microbenchmarks.............................................................209
4.3.2 Choosing Workloads.......................................................................................................211
4.3.3 Evaluating a Fixed-Size Machine...................................................................................214
4.3.4 Varying Machine Size.....................................................................................................220
4.3.5 Choosing Performance Metrics......................................................................................220
10 DRAFT: Parallel Computer Architecture 9/4/97
4.3.6 Presenting Results..........................................................................................................224
4.4 Evaluating an Architectural Idea or Tradeoff .........................................................................225
4.4.1 Multiprocessor Simulation.............................................................................................226
4.4.2 Scaling Down Problem and Machine Parameters for Simulation..................................227
4.4.3 Dealing with the Parameter Space: An Example Evaluation.........................................230
4.4.4 Summary........................................................................................................................235
4.5 Illustrating Workload Characterization ...................................................................................235
4.5.1 Workload Case Studies...................................................................................................236
4.5.2 Workload Characteristics...............................................................................................243
4.6 Concluding Remarks ..............................................................................................................250
4.7 References ..............................................................................................................................251
4.8 Exercises .................................................................................................................................253
CHAPTER 5 Shared Memory Multiprocessors ...............................................259
5.1 Introduction ............................................................................................................................259
5.2 Cache Coherence ....................................................................................................................263
5.2.1 The Cache Coherence Problem......................................................................................263
5.2.2 Cache Coherence Through Bus Snooping......................................................................266
5.3 Memory Consistency ..............................................................................................................272
5.3.1 Sequential Consistency..................................................................................................274
5.3.2 Sufficient Conditions for Preserving Sequential Consistency........................................276
5.4 Design Space for Snooping Protocols ....................................................................................279
5.4.1 A 3-state (MSI) Write-back Invalidation Protocol.........................................................280
5.4.2 A 4-state (MESI) Write-Back Invalidation Protocol......................................................285
5.4.3 A 4-state (Dragon) Write-back Update Protocol............................................................287
5.5 Assessing Protocol Design Tradeoffs .....................................................................................290
5.5.1 Workloads.......................................................................................................................292
5.5.2 Impact of Protocol Optimizations..................................................................................296
5.5.3 Tradeoffs in Cache Block Size.......................................................................................298
5.5.4 Update-based vs. Invalidation-based Protocols..............................................................310
5.6 Synchronization ......................................................................................................................314
5.6.1 Components of a Synchronization Event.......................................................................315
5.6.2 Role of User, System Software and Hardware...............................................................316
5.6.3 Mutual Exclusion...........................................................................................................317
5.6.4 Point-to-point Event Synchronization............................................................................328
5.6.5 Global (Barrier) Event Synchronization.........................................................................330
5.6.6 Synchronization Summary.............................................................................................334
5.7 Implications for Software .......................................................................................................335
5.8 Concluding Remarks ..............................................................................................................341
5.9 References ..............................................................................................................................342
5.10 Exercises .................................................................................................................................346
9/4/97 DRAFT: Parallel Computer Architecture 11
CHAPTER 6 Snoop-based Multiprocessor Design ..........................................355
6.1 Introduction .............................................................................................................................355
6.2 Correctness Requirements ......................................................................................................356
6.3 Base Design: Single-level Caches with an Atomic Bus .........................................................359
6.3.1 Cache controller and tags...............................................................................................359
6.3.2 Reporting snoop results..................................................................................................360
6.3.3 Dealing with Writebacks................................................................................................362
6.3.4 Base Organization..........................................................................................................362
6.3.5 Non-atomic State Transitions.........................................................................................363
6.3.6 Serialization....................................................................................................................365
6.3.7 Deadlock.........................................................................................................................366
6.3.8 Livelock and Starvation..................................................................................................367
6.3.9 Implementing Atomic Primitives....................................................................................367
6.4 Multi-level Cache Hierarchies ................................................................................................369
6.4.1 Maintaining inclusion.....................................................................................................370
6.4.2 Propagating transactions for coherence in the hierarchy................................................372
6.5 Split-transaction Bus ...............................................................................................................374
6.5.1 An Example Split-transaction Design............................................................................375
6.5.2 Bus Design and Request-response Matching.................................................................375
6.5.3 Snoop Results and Conflicting Requests........................................................................377
6.5.4 Flow Control...................................................................................................................377
6.5.5 Path of a Cache Miss......................................................................................................379
6.5.6 Serialization and Sequential Consistency.......................................................................380
6.5.7 Alternative design choices..............................................................................................382
6.5.8 Putting it together with multi-level caches.....................................................................384
6.5.9 Supporting Multiple Outstanding Misses from a Processor...........................................386
6.6 Case Studies: SGI Challenge and Sun Enterprise SMPs ........................................................387
6.6.1 SGI Powerpath-2 System Bus........................................................................................389
6.6.2 SGI Processor and Memory Subsystems........................................................................392
6.6.3 SGI I/O Subsystem.........................................................................................................393
6.6.4 SGI Challenge Memory System Performance...............................................................395
6.6.5 Sun Gigaplane System Bus............................................................................................396
6.6.6 Sun Processor and Memory Subsystem.........................................................................397
6.6.7 Sun I/O Subsystem.........................................................................................................398
6.6.8 Sun Enterprise Memory System Performance...............................................................399
6.6.9 Application Performance................................................................................................399
6.7 Extending Cache Coherence ...................................................................................................401
6.7.1 Shared-Cache Designs....................................................................................................401
6.7.2 Coherence for Virtually Indexed Caches........................................................................405
6.7.3 Translation Lookaside Buffer Coherence.......................................................................407
6.7.4 Cache coherence on Rings..............................................................................................409
6.7.5 Scaling Data Bandwidth and Snoop Bandwidth............................................................412
6.8 Concluding Remarks ..............................................................................................................413
6.9 References ...............................................................................................................................413
6.10 Exercises .................................................................................................................................417
12 DRAFT: Parallel Computer Architecture 9/4/97
CHAPTER 7 Scalable Multiprocessors ...........................................................423
7.1 Introduction ............................................................................................................................423
7.2 Scalability ...............................................................................................................................426
7.2.1 Bandwidth Scaling.........................................................................................................426
7.2.2 Latency Scaling..............................................................................................................429
7.2.3 Cost Scaling....................................................................................................................431
7.2.4 Physical scaling..............................................................................................................432
7.2.5 Scaling in a Generic Parallel Architecture.....................................................................436
7.3 Realizing Programming Models .............................................................................................437
7.3.1 Primitive Network Transactions.....................................................................................438
7.3.2 Shared Address Space....................................................................................................442
7.3.3 Message Passing.............................................................................................................444
7.3.4 Common challenges.......................................................................................................449
7.3.5 Communication architecture design space.....................................................................451
7.4 Physical DMA ........................................................................................................................452
7.4.1 A Case Study: nCUBE/2................................................................................................454
7.5 User-level Access ....................................................................................................................456
7.5.1 Case Study: Thinking Machines CM-5..........................................................................458
7.5.2 User Level Handlers.......................................................................................................459
7.6 Dedicated Message Processing ...............................................................................................461
7.6.1 Case Study: Intel Paragon..............................................................................................464
7.6.2 Case Study: Meiko CS-2................................................................................................467
7.7 Shared Physical Address Space ..............................................................................................470
7.7.1 Case study: Cray T3D....................................................................................................472
7.7.2 Cray T3E........................................................................................................................474
7.7.3 Summary........................................................................................................................475
7.8 Clusters and Networks of Workstations .................................................................................476
7.8.1 Case Study: Myrinet SBus Lanai...................................................................................478
7.8.2 Case Study: PCI Memory Channel................................................................................480
7.9 Comparison of Communication Performance ........................................................................483
7.9.1 Network Transaction Performance.................................................................................483
7.9.2 Shared Address Space Operations..................................................................................487
7.9.3 Message Passing Operations..........................................................................................489
7.9.4 Application Level Performance......................................................................................490
7.10 Synchronization ......................................................................................................................495
7.10.1 Algorithms for Locks.....................................................................................................497
7.10.2 Algorithms for Barriers..................................................................................................499
7.11 Concluding Remarks ..............................................................................................................505
7.12 References ..............................................................................................................................506
7.13 Exercices .................................................................................................................................510
CHAPTER 8 Directory-based Cache Coherence .............................................513
8.1 Introduction ............................................................................................................................513
9/4/97 DRAFT: Parallel Computer Architecture 13
