Using OpenMP: Portable Shared Memory Parallel Programming (Scientific Computation and Engineering)

377 Pages·2007·16.38 MB·english

by  Barbara Chapman

COMPUTER SCIENCE SCIENTIFIC
Using OpenMP
AND
PORTABLE SHARED MEMORY PARALLEL PROGRAMMING
ENGINEERING
Barbara Chapman, Gabriele Jost, and Ruud van der Pas
foreword by David J. Kuck
COMPUTATION
OpenMP, a portable programming interface for shared memory parallel com- “I hope that readers will learn to use the full expressibility and power of
SERIES
puters, was adopted as an informal standard in 1997 by computer scientists OpenMP. This book should provide an excellent introduction to beginners,
who wanted a unified model on which to base programs for shared memory and the performance section should help those with some experience who
systems. OpenMP is now used by many software developers; it offers signif- want to push OpenMP to its limits.” AC
NH
icant advantages over both hand-threading and MPI. Using OpenMPoffers from the foreword by David J. Kuck, Intel Fellow, Software and Solutions Group, D AP
VM
a comprehensive introduction to parallel programming concepts and a and Director, Parallel and Distributed Solutions, Intel Corporation A
NA
detailed overview of OpenMP. N
D,
“The advent of readily available, inexpensive multicore processors has E J
RO
Using OpenMPdiscusses hardware developments, describes where OpenMP made parallel programming more important and more accessible than ever PST
is applicable, and compares OpenMP to other programming interfaces for before. OpenMP is a popular way to write parallel programs, and this book AS,
shared and distributed memory parallel architectures. It introduces the makes OpenMP knowledge available to the average programmer in an
individual features of OpenMP, provides many source code examples that understandable, easy-to-apply fashion, while still providing information
demonstrate the use and functionality of the language constructs, and offers for those who wish to dive more deeply into the subject.”
tips on writing an efficient OpenMP program. It describes how to use Larry Meadows, CEO, the OpenMP ARB Using OpenMP
U
OpenMP in full-scale applications to achieve high performance on large-scale
s
architectures, discussing several case studies in detail, and offers in-depth SCIENTIFIC AND ENGINEERING COMPUTATION SERIES i
n
troubleshooting advice. It explains how OpenMP is translated into explicitly
g
multithreaded code, providing a valuable behind-the-scenes account of
O
OpenMP program performance. Finally, UsingOpenMPconsiders trends likely
p
to influence OpenMP development, offering a glimpse of the possibilities of
e
a future OpenMP 3.0 from the vantage point of the current OpenMP 2.5. n
M
With multicore computer use increasing, the need for a comprehensive P
introduction and overview of the standard interface is clear. Using OpenMP PORTABLE SHARED MEMORY PARALLEL PROGRAMMING
provides an essential reference not only for students at both undergraduate
and graduate levels but also for professionals who intend to parallelize
existing codes or develop new parallel programs for shared memory com-
puter architectures.
Barbara Chapman is Professor of Computer Science at the University of
Houston. Gabriele Jost is Principal Member of Technical Staff, Application
978-0-262-53302-7
Server Performance Engineering, at Oracle, Inc. Ruud van der Pas is Senior
BARBARA CHAPMAN, foreword by
Staff Engineer at Sun Microsystems, Menlo Park.
GABRIELE JOST, DAVID J. KUCK
AND RUUD VAN DER PAS
THE MIT PRESS MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASSACHUSETTS 02142 HTTP://MITPRESS.MIT.EDU
Using OpenMP
Scientific and Engineering Computation
William Gropp and Ewing Lusk, editors; Janusz Kowalik, founding editor
Data-ParallelProgrammingonMIMDComputers,PhilipJ.HatcherandMichaelJ.Quinn,1991
Unstructured Scientific Computation on Scalable Multiprocessors,editedbyPiyushMehrotra,
JoelSaltz,andRobertVoigt,1992
Parallel Computational Fluid Dynamics: Implementation and Results,editedbyHorst
D.Simon,1992
Enterprise Integration Modeling: Proceedings of the First International Conference,editedby
CharlesJ.Petrie,Jr.,1992
The High Performance Fortran Handbook,CharlesH.Koelbel,DavidB.Loveman,Robert
S.Schreiber,GuyL.SteeleJr.,andMaryE.Zosel,1994
PVM: Parallel Virtual Machine—A Users’ Guide and Tutorial for Network Parallel
Computing,AlGeist,AdamBeguelin,JackDongarra,WeichengJiang,BobManchek,and
VaidySunderam,1994
Practical Parallel Programming,GregoryV.Wilson,1995
Enabling Technologies for Petaflops Computing,ThomasSterling,PaulMessina,andPaul
H.Smith,1995
An Introduction to High-Performance Scientific Computing,LloydD.Fosdick,Elizabeth
R.Jessup,CarolynJ.C.Schauble,andGittaDomik,1995
Parallel Programming Using C++,editedbyGregoryV.WilsonandPaulLu,1996
Using PLAPACK: Parallel Linear Algebra Package,RobertA.vandeGeijn,1997
Fortran 95 Handbook,JeanneC.Adams,WalterS.Brainerd,JeanneT.Martin,BrianT.Smith,
andJerroldL.Wagener,1997
MPI—The Complete Reference: Volume 1, The MPI Core,MarcSnir,SteveOtto,Steven
Huss-Lederman,DavidWalker,andJackDongarra,1998
MPI—The Complete Reference: Volume 2, The MPI-2 Extensions,WilliamGropp,Steven
Huss-Lederman,AndrewLumsdaine,EwingLusk,BillNitzberg,WilliamSaphir,andMarcSnir,
1998
A Programmer’s Guide to ZPL,LawrenceSnyder,1999
How to Build a Beowulf,ThomasL.Sterling,JohnSalmon,DonaldJ.Becker,andDaniel
F.Savarese,1999
Using MPI: Portable Parallel Programming with the Message-Passing Interface, second
edition,WilliamGropp,EwingLusk,andAnthonySkjellum,1999
Using MPI-2: Advanced Features of the Message-Passing Interface,WilliamGropp,Ewing
Lusk,andRajeevThakur,1999
Beowulf Cluster Computing with Linux,ThomasSterling,2001
Beowulf Cluster Computing with Windows,ThomasSterling,2001
Scalable Input/Output: Achieving System Balance,DanielA.Reed,2003
Using OpenMP
Portable Shared Memory Parallel Programming
Barbara Chapman, Gabriele Jost, Ruud van der Pas
The MIT Press
Cambridge, Massachusetts
London, England
(cid:2)c 2008MassachusettsInstituteofTechnology
Allrightsreserved. Nopartofthisbookmaybereproducedinanyformbyanyelectronicor
mechanicalmeans(includingphotocopying,recording,orinformationstorageandretrieval)
withoutpermissioninwritingfromthepublisher.
ThisbookwassetinLATEXbytheauthorsandwasprintedandboundintheUnitedStatesof
America.
LibraryofCongressCataloging-in-PublicationData
Chapman,Barbara,1954-
UsingOpenMP:portablesharedmemoryparallelprogramming/BarbaraChapman,Gabriele
Jost,RuudvanderPas.
p. cm. –(Scientificandengineeringcomputation)
Includesbibliographicalreferencesandindex.
ISBN-13: 978-0-262-53302-7(paperback: alk. paper)
1. Parallelprogramming(Computerscience)2. Applicationprograminterfaces
(Computersoftware)I.Jost,Gabriele. II.Pas,Ruudvander. III.Title.
QA76.642.C492007
005.2’75–dc22
2007026656
Dedicated to the memory of Ken Kennedy, who inspired in so many of us a
passion for High Performance Computing

Contents
Series Foreword xiii
Foreword xv
Preface xix
1 Introduction 1
1.1 Why Parallel Computers Are Here to Stay 1
1.2 Shared-Memory Parallel Computers 3
1.2.1 Cache Memory Is Not Shared 4
1.2.2 Implications of Private Cache Memory 6
1.3 Programming SMPs and the Origin of OpenMP 6
1.3.1 What Are the Needs? 7
1.3.2 A Brief History of Saving Time 7
1.4 What Is OpenMP? 8
1.5 Creating an OpenMP Program 9
1.6 The Bigger Picture 11
1.7 Parallel Programming Models 13
1.7.1 Realization of Shared- and Distributed-Memory 14
Models
1.8 Ways to Create Parallel Programs 15
1.8.1 A Simple Comparison 16
1.9 A Final Word 21
2 Overview of OpenMP 23
2.1 Introduction 23
2.2 The Idea of OpenMP 23
2.3 The Feature Set 25
2.3.1 Creating Teams of Threads 25
2.3.2 Sharing Work among Threads 26
2.3.3 The OpenMP Memory Model 28
2.3.4 Thread Synchronization 29
2.3.5 Other Features to Note 30
2.4 OpenMP Programming Styles 31
viii Contents
2.5 Correctness Considerations 32
2.6 Performance Considerations 33
2.7 Wrap-Up 34
3 Writing a First OpenMP Program 35
3.1 Introduction 35
3.2 Matrix Times Vector Operation 37
3.2.1 C and Fortran Implementations of the Problem 38
3.2.2 A Sequential Implementation of the Matrix Times 38
Vector Operation
3.3 Using OpenMP to Parallelize the Matrix Times Vector 41
Product
3.4 Keeping Sequential and Parallel Programs as a Single 47
Source Code
3.5 Wrap-Up 50
4 OpenMP Language Features 51
4.1 Introduction 51
4.2 Terminology 52
4.3 Parallel Construct 53
4.4 Sharing the Work among Threads in an OpenMP Program 57
4.4.1 Loop Construct 58
4.4.2 The Sections Construct 60
4.4.3 The Single Construct 64
4.4.4 Workshare Construct 66
4.4.5 Combined Parallel Work-Sharing Constructs 68
4.5 Clauses to Control Parallel and Work-Sharing Constructs 70
4.5.1 Shared Clause 71
4.5.2 Private Clause 72
4.5.3 Lastprivate Clause 73
4.5.4 Firstprivate Clause 75
4.5.5 Default Clause 77
4.5.6 Nowait Clause 78
Contents ix
4.5.7 Schedule Clause 79
4.6 OpenMP Synchronization Constructs 83
4.6.1 Barrier Construct 84
4.6.2 Ordered Construct 86
4.6.3 Critical Construct 87
4.6.4 Atomic Construct 90
4.6.5 Locks 93
4.6.6 Master Construct 94
4.7 Interaction with the Execution Environment 95
4.8 More OpenMP Clauses 100
4.8.1 If Clause 100
4.8.2 Num threads Clause 102
4.8.3 Ordered Clause 102
4.8.4 Reduction Clause 105
4.8.5 Copyin Clause 110
4.8.6 Copyprivate Clause 110
4.9 Advanced OpenMP Constructs 111
4.9.1 Nested Parallelism 111
4.9.2 Flush Directive 114
4.9.3 Threadprivate Directive 118
4.10 Wrap-Up 123
5 How to Get Good Performance by Using 125
OpenMP
5.1 Introduction 125
5.2 Performance Considerations for Sequential Programs 125
5.2.1 Memory Access Patterns and Performance 126
5.2.2 Translation-Lookaside Buffer 128
5.2.3 Loop Optimizations 129
5.2.4 Use of Pointers and Contiguous Memory in C 136
5.2.5 Using Compilers 137
5.3 Measuring OpenMP Performance 138
5.3.1 Understanding the Performance of an OpenMP 140
Program