Table Of ContentSOURCEBOOK
OF PARALLEL
COMPUTING
JACK DONGARRA
UniversityofTennessee
IAN FOSTER
ArgonneNationalLaboratory
GEOFFREY FOX
IndianaUniversity
WILLIAM GROPP
ArgonneNationalLaboratory
KEN KENNEDY
RiceUniversity
LINDA TORCZON
RiceUniversity
ANDY WHITE
LosAlamosNationalLaboratory
SeniorEditor DenisePenrose
PublishingServicesManager EdwardWade
ProductionEditor HowardSeverson
EditorialCoordinator EmiliaThiuri
CoverDesign FrancesBaca
TextDesign DettaPenna
Illustration DartmouthPublishing,Inc.
Composition WindfallSoftware,usingZzTEX
Copyeditor BarbaraKohl
Proofreader CarolLeyba
Indexer SteveRath
Printer TheMaple-VailBookManufacturingGroup
Covercredit:PaulKlee,Greenchurchsteepleatcenter,1917.©Nimatallah/ArtResource,NY.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks
orregisteredtrademarks.InallinstancesinwhichMorganKaufmannPublishersisawareofa
claim,theproductnamesappearininitialcapitalorallcapitalletters.Readers,however,should
contacttheappropriatecompaniesformorecompleteinformationregardingtrademarksand
registration.
MorganKaufmannPublishers
AnimprintofElsevierScience
340PineStreet,SixthFloor
SanFrancisco,CA94104-3205
www.mkp.com
©2003bybyElsevierScience(USA)
Allrightsreserved.
PrintedintheUnitedStatesofAmerica
07 06 05 04 03 5 4 3 2 1
Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin
anyformorbyanymeans—electronic,mechanical,photocopying,orotherwise—withoutthe
priorwrittenpermissionofthepublisher.
LibraryofCongressControlNumber:2002107244
ISBN:1-55860-871-0
Thisbookisprintedonacid-freepaper.
PREFACE
Duringits11-yearlifetime,theCenterforResearchonParallelComputation(CRPC),
aNationalScienceFoundation(NSF)ScienceandTechnologyCenter,wasfocusedon
researchandtechnologydevelopmentthatwouldbeneededtomakeparallelcom-
puting“trulyusable.”Overthatperiod,aremarkablenumberofapplicationswere
convertedtoworkonscalableparallelcomputers,andanequallyremarkablenum-
ber of software tools were developed to assist in that process. These developments
emerged both from within CRPC and from the community at large. Although the
toolsandstrategiesdevelopedandrefinedduringthisperiodarewidelyknowninthe
communityofprofessionalparallelapplicationdevelopers,toourknowledgethere
existsnocompendiumofmaterialdescribingtheseadvancesinaformsuitablefor
use by a newcomer to the field. This volume attempts to fill that gap by bringing
togetheracollectionofworkscoveringbothapplicationsandthetechnologiesused
toconvertthemtorunonscalableparallelsystems.
The principal goal of this book is to make it easy for newcomers to the field
of parallel computing to understand the technologies available and how to apply
them.Thebookisintendedforstudentsandpractitionersoftechnicalcomputing
whoneedtounderstandboththepromiseandpracticeofhigh-performanceparallel
computation.Itassumesthatthereaderhasagoodbackgroundineitherapplications
orcomputationalscience,butnotinparallelcomputation.Thus,thebookincludes
asignificantamountoftutorialmaterial.
Wealsointendthatthebookserveasausefulreferenceforthepracticingparallel
applicationdeveloper.Thus,itcontainsin-depthtreatmentsofspecifictechnologies
andmethodsthatareusefulinparallelcomputing.Thesemethodsareeasilyaccessi-
blebystartingfromtheapplicationoverviewsectionsorbyreadingthetechnology
overviewchaptersprovidedatthebeginningofeachmajorpart.
xiii
xiv Preface
Wehopethatyoufindthisbookusefulandthatithelpsyouexploittheknowledge
gained over the past fifteen years, while avoiding some of the pitfalls that we
ourselvesencounteredingainingthatknowledge.
Overview of Content
Thebookisorganizedintofivemajorparts.PartI,entitledParallelism,isatutorial
introductiontothefieldofparallelcomputersandcomputing,withseparatechapters
providingabroadoverviewofthefield,anin-depthtreatmentofthearchitectureof
modernparallelcomputersystems,andasurveyoftheissuesthatshouldbetaken
intoconsiderationwhenprogrammingthem.
Part II, Applications, is designed to help new developers understand how high-
performancecomputationcanbeappliedinavarietyofspecificapplicationareas.It
consistsofanoverviewoftheprocessbywhichoneidentifiesappropriatesoftware
andalgorithmsandtheissuesinvolvedinimplementation.Thistreatmentiscom-
plemented by chapters containing in-depth studies in the areas of computational
fluiddynamics,environmentalengineeringandenergy,andcomputationalchem-
istry,andbyaseparatechapterwith11vignettesthatbrieflydescribesuccessfuluses
ofparallelsystemsinotherareas.Thesehavebeenchosentocoverabroadrangeof
bothscientificareasandnumericalapproaches.Theapplicationsarecross-referenced
to the material in later chapters that cover the needed software technologies and
algorithmsindepth.
PartIII,SoftwareTechnologies,discussestheprogressmadeonavarietyoftechno-
logicalstrategiesforapplicationdevelopment, includingmessage-passinglibraries;
run-timelibrariesforparallelcomputing,suchasclasslibrariesforHPC++,languages
likeHPF,Co-ArrayFortran,andHPC++;performanceanalysisandtuningtoolssuch
asPablo;andhigh-levelprogrammingsystems.Thegoalofthispartistoprovidea
survey of progress with hints to the user that will help in selecting the right tech-
nologyforuseinagivenapplication.
PartIV,EnablingTechnologiesandAlgorithms,treatsnumericalalgorithmsandcov-
ersparallelnumericalalgorithmsforavarietyofproblemsinscienceandengineer-
ing,includinglinearalgebra,continuousanddiscreteoptimization,andsimulation.
Eachchaptercoversadifferentalgorithmicarea.Thegoalhereistoserveasaresource
fortheapplicationdeveloperseekinggoodalgorithmsfordifficultproblems.
Thefinalpartofthebookisdevotedtoasummaryoftheissuesandadiscussion
of important future problems for the high-performance science and engineering
community,includingdistributedcomputinginagridenvironment.
Using This Book
Thisbookcanbeusedinseveraldifferentways.Thenewcomertoparallelcompu-
tationseekingatutorialintroductionshouldreadallofPartI,alongwithChapters
4,9,16,17,and25.Chapters4,9,and17provideoverviewsofPartsII,III,andIV,
Preface xv
respectively,sothesemayprovidehintsonotherchaptersthatmaybeofinterestto
thereader.
Ontheotherhand, thedeveloperwishingtounderstandtechnologiesthatcan
helpwithaspecificapplicationareashouldstartwithPartII,whichcoversanumber
ofapplicationsalongwiththestrategiesusedtoparallelizethem.Oncethedeveloper
hasidentifiedasimilarapplication,heorshecanfollowthecross-referencestofind
in-depthtreatmentsoftheusefultechnologiesinPartsIIIandIV.
Finally,theparallelcomputingprofessionalcanusethebookasareference.The
introductorychaptersatthebeginningofeachmajorpartprovideexcellentguides
tothematerialintherestofthepart.
Thebookshouldserveasaresourceforusersofsystemsthatspantherangefrom
smalldesktopSMPsandPCclusterstohigh-endsupercomputerscosting$100mil-
lionormore.Itfocusesonsoftwaretechnologies, alongwiththelarge-scaleappli-
cationsenabledbythem.Ineacharea,thetextcontainsageneraldiscussionofthe
stateofthefieldfollowedbydetaileddescriptionsofkeytechnologiesormethods.
Insomecases,suchasMPIformessagepassing,thisisthedominantapproach.In
others,suchasthediscussionofproblem-solvingenvironments,theauthorschoose
systemsrepresentingkeyconceptsinanemergingarea.
Supplementary Online Material
Look for links to the Sourcebook of Parallel Computing website at www.mkp.com,
whereyoucanfindnumeroussupplementarymaterials.Inadditiontoupdatesand
correctionsdiscoveredafterpublication,thereaderwillfindlinkstoservers(suchas
Netlib)fromwhichsoftwaretoolsandlibrariescanbedownloaded.
Acknowledgments
This book was inspired by the research carried out in the high-performance paral-
lel computing commmunity during the lifetime of the CRPC, an NSF Science and
TechnologyCenterthatincludedRiceUniversity,CaliforniaInstituteofTechnology,
ArgonneNationalLaboratory,LosAlamosNationalLaboratory,SyracuseUniversity,
the University of Tennessee–Knoxville, and the University of Texas–Austin. With-
out the original grant from the NSF that established CRPC, the book would not
have been possible. In addition, the support for research provided through other
grants and contracts from NSF, the Department of Energy, the Department of De-
fense(especiallytheDefenseAdvancedResearchProjectsAgency),andtheNational
Aeronautics and Space Administration has been critical to the advances described
in this volume. Substantive additional support for research and technology devel-
opment was provided by the CRPC home institutions, and the home institutions
ofmanyofthechapterauthors.Inparticular, matchingfundsforCRPCfromRice
Universitywereusedtoprovidetechnicalsupportforthecompletionofthisvolume
aftertheNSFfundingforCRPCended.
xvi Preface
Theeditorswouldalsoliketothankthepeoplewhohelpedusfinishthisvolume.
TeresaParksimprovedtheentiremanuscriptwithhereditorialandtechnicalwriting
skills.SarahGonzalesprovidedtechnicalsupportandpreparedthegraphicsforpub-
lication.GailPiepercarefullyandknowledgeablyeditedsomeofthechapters.Keith
CooperconsultedwithusonthedeepermysteriesofLaTEX.EllenButlerandShiliang
Chang collected, collated, merged, and corrected the bibliographic entries for this
volume.PennyAndersonmanagedanumberofproductiontasksontheRiceside,
includingassemblingauthorbiographiesandcollectingcopyrighttransferformsand
permissionsforuseofcopyrightedmaterials.EllenButler,KathrynO’Brien,Penny
Anderson, andTheresaChatmanarrangedthemeetingsandteleconferences, han-
dledthecorrespondence,managedthemanycontactlists,andlenthandswherever
they were needed. All these people deserve our thanks; without their efforts, this
volumewouldnothavebeenfinished.
Denise Penrose, our editor at Morgan Kaufmann, provided valuable advice on
the preparation of the manuscript and arranged for independent reviews of the
manuscript that were very helpful. Emilia Thiuri provided editorial support and
extensive guidance as the final manuscript neared completion. Howard Severson,
the production editor, did a superb job of bringing the book to completion on a
tight schedule and managed to keep his sense of humor throughout the process.
Theproductionstaff,includingthecopyeditorBarbaraKohl,theproofreaderCarol
Leyba,andtheindexerSteveRath,weretirelessintheireffortstobringoutahigh-
qualitybookontime.Wearealsogratefultothosewhocontributedtotheoverall
appearanceofthebook:DettaPennafortextdesign,FrancesBacaforcoverdesign,
DartmouthPublishingforillustration,andWindfallSoftwareforcomposition.We
especiallyappreciatethecompositor’seffortstoincorporatethesubstantivenumber
of changes we made late in the production cycle. Finally, the reviewers provided
many intelligent and helpful comments on the book that significantly affected its
structure,content,andquality.
Toallofthesepeople,andtotheauthorsofindividualchaptersinthistext,we
extendourheartfeltthanks.
Part I Parallelism1..................................................
Introduction ........................................................ 3
Parallel Computing Hardware .................................. 4
What Have We Learned from Applications? ............ 8
Software and Algorithms.......................................... 11
Toward a Science of Parallel Computation.............. 13
Parallel Computer Architectures ........................ 15
Uniprocessor Architecture........................................ 16
The CPU.............................................................. 17
Memory................................................................ 21
I/O and Networking.............................................. 25
Design Tradeoffs ................................................. 26
Parallel Architectures ............................................... 26
Memory Parallelism............................................. 26
Interconnects....................................................... 33
CPU Parallelism .................................................. 35
I/O and Networking for Parallel Processors......... 38
Support for Programming Models........................ 39
Parallel Architecture Design Tradeoffs................ 39
Future Directions for Parallel Architectures.............. 40
Conclusion ............................................................... 41
Parallel Programming Considerations ............... 43
Architectural Considerations.................................... 45
Shared Memory................................................... 45
Distributed Memory ............................................. 46
Hybrid Systems ................................................... 47
Memory Hierarchy ............................................... 47
Decomposing Programs for Parallelism................... 49
Identi.cation of Parallelism................................... 49
Decomposition Strategy ...................................... 50
Programming Models .......................................... 51
Implementation Styles ......................................... 51
A Simple Example ............................................... 54
Enhancing Parallel Performance.............................. 56
Scalability and Load Balance .............................. 57
Pipeline Parallelism............................................. 59
Regular versus Irregular Problems...................... 60
Memory-Hierarchy Management.............................. 63
Uniprocessor Memory-Hierarchy Management... 63
Multiprocessor Memory Hierarchies.................... 65
Parallel Debugging................................................... 66
Performance Analysis and Tuning........................... 67
Parallel Input/Output ................................................ 69
Conclusion ............................................................... 70
Further Reading....................................................... 70
Part II Applications7..3.............................................
General Application Issues ................................ 75
Application Characteristics in a Simple Example..... 75
Communication Structure in Jacobi(cid:144)s Method for
Poisson(cid:144)s Equation .................................................. 79
Communication Overhead for More General
Update Stencils........................................................ 82
Applications as Basic Complex Systems ................. 84
Time-Stepped and Event-Driven Simulations .......... 87
Temporal Structure of Applications.......................... 88
Summary of Parallelization of Basic Complex
Systems ................................................................... 89
Meta-Problems......................................................... 90
Conclusion ............................................................... 91
Parallel Computing in Computational Fluid
Dynamics ........................................................... 93
Introduction to Computational Fluid Dynamics......... 94
Basic Equations of Fluid Dynamics ..................... 94
Physical Regimes and Dimensionless
Variables.............................................................. 95
The Role of High-Performance Computing ......... 97
Incompressible Flows............................................... 98
Semi-discrete Formulation................................... 99
Spectral Element Methods .................................. 100
Basic Operations ................................................. 108
Global Matrix Operations..................................... 111
Data Structures.................................................... 113
Solution Techniques............................................ 116
Adaptive Mesh Re.nement .................................. 121
Implementation for Parallel Architectures............ 126
An Example(cid:132)The Cylinder Wake .......................... 129
Compressible Flows................................................. 132
Governing Equations of Motion........................... 133
Numerical Methods for Hyperbolic
Conservation Laws.............................................. 134
An Application: The Richtmyer (cid:133)Meshkov
Instability.............................................................. 137
Adaptive Mesh Re.nement .................................. 138
Conclusion ............................................................... 144
Parallel Computing in Environment and
Energy................................................................ 145
Subsurface-Flow Modeling....................................... 146
IPARS Motivation ................................................ 147
IPARS Description............................................... 148
IPARS and Grid Computing by NetSolve................. 152
Integrating IPARS into NetSolve ......................... 153
Client-Side Web-Browser Interface ..................... 154
Tracking and Interactive Simulation in IPARS ......... 155
An Interactive Computational Collaboration:
DISCOVER.......................................................... 157
Integrating IPARS with DISCOVER..................... 158
Surface-Water Simulation ........................................ 159
A Water Quality Model: CE-QUAL-ICM............... 159
A Parallel Water-Quality Model:
PCE-QUAL-ICM .................................................. 160
Parallel Algorithm ................................................ 161
A Coupled Simulation of Flow and Transport with
ADR.......................................................................... 162
The Active Data Repository................................. 163
Implementation.................................................... 164
Conclusion ............................................................... 165
Parallel Computational Chemistry: An
Overview of NWChem ....................................... 167
Molecular Quantum Chemistry................................. 168
