Table Of ContentIPPRRAACCTlIlClCAALl
IPPAARRAALLllEELl
CCOOMMIPPUUTlllIMNGC
HH.. SStteepphheenn MMoorrssee
^~ AAPP PPRROOFFEESSSSIIOONNAALL
'-J
1Φ>^0^^^^^~ ~~ BBoossttoonn SSaann DDiieeggoo NNeeww YYoorrkk
'f:c£SSSSv\O^ LLoonnddoonn SSyyddnneeyy TTookkyyoo TToorroonnttoo
®
TThhiiss bbooookk iiss pprriinntteedd oonn aacciidd--ffrreeee ppaappeerr., (oo
CCooppyyrriigghhtt ©© 11999944 bbyy AAccaaddeemmiicc PPrreessss,, IInncc..
AAllll rriigghhttss rreesseerrvveedd..
NNoo ppaarrtt ooff tthhiiss ppuubblliiccaattiioonn mmaayy bbee rreepprroodduucceedd oorr
ttrraannssmmiitttteedd iinn aannyy ffoorrmm oorr bbyy aannyy mmeeaannss,, eelleeccttrroonniicc
oorr mmeecchhaanniiccaall,, iinncclluuddiinngg pphhoottooccooppyy,, rreeccoorrddiinngg,, oorr
aannyy iinnffoorrmmaattiioonn ssttoorraaggee aanndd rreettrriieevvaall ssyysstteemm,, wwiitthhoouutt
ppeerrmmiissssiioonn iinn wwrriittiinngg ffrroomm tthhee ppuubblliisshheerr..
AAllll bbrraanndd nnaammeess aanndd pprroodduucctt nnaammeess mmeennttiioonneedd iinn tthhiiss bbooookk aarree
ttrraaddeemmaarrkkss oorr rreeggiisstteerreedd ttrraaddeemmaarrkkss ooff tthheeiirr rreessppeeccttiivvee ccoommppaanniieess..
AAPP PPRROOFFEESSSSIIOONNAALL
995555 MMaassssaacchhuusseettttss AAvveennuuee,, CCaammbbrriiddggee,, MMAA 0022113399
AAnn IImmpprriinntt ooff AACCAADDEEMMIICC PPRREESSSS,, IINNCC..
AA DDiivviissiioonn ooff HHAARRCCOOUURRTT BBRRAACCEE && CCOOMMPPAANNYY
UUnniitteedd KKiinnggddoommE Edditiitoionnp puubblilsihshededb byy
AACCAADDEEMMIICC PPRREESSSS LLIIMMIITTEEDD
2244--2288 OOvvaall RRooaadd,, LLoonnddoonn NNWW1I 77DDXX
LLiibbrraarryy ooff CCoonnggrreessss CCaattaallooggiinngg--iinn--PPuubblliiccaattiioonn DDaattaa
MMoorrssee,, HH.. SStteepphheenn..
PPrraaccttiiccaall ppaarraalllleell ccoommppuuttiinngg // HH.. SStteepphheenn MMoorrssee..
pp.. ecmm..
IInncclluuddeess iinnddeexx..
IISSBBNN 00--1122--550088116600--XX
11.. PPaarraalllleell pprroocceessssiinngg ((EElleeccttrroonniicc ccoommppuutteerrss)) II.. TTiittllee..
QQAA7766..5588..MM6677 11999944
000044''..3355-~-ddcc2200 9944--3322005533
CCIIPP
PPrriinntteedd iinn tthhee UUnniitteedd SSttaatteess ooff AAmmeerriiccaa
9944 9955 9966 9977 9988 IIPP 99 88 77 66 55 44 33 22 1I
ddeeddiiccaatteedd totom myyc chhilidldrerne,n,
KKaattee aanndd PPaauull,,
wwhhoosseeo onn-b-booaardrdp apraarlalellleplr porcoecsseosrssors
aarree aa ccoonnssttaanntt ssoouurrcceeo offw woonndderer
PPrreeffaaccee
TThhiiss iiss tthhee tthhee bbooookk II wwiisshh II hhaadd hhaadd nniinnee yyeeaarsr saaggoo wwhhene nI Ififrisrts tvvenentuturerde dinintot o
tthhee rreeaallmm ooff hhiigghh--ppeerrffoorrmmaannccee ppaarraalllleell pprroocceessssiinngg.. TThhee ccoonntrtraacct ttoto wwhhicihch I I
wwaass tthheenn aassssiiggnneedd iinnvvoollvveedd sseelleeccttiioonn ooff aa ppaarraalllleel lccoommppuutteer rwwitihth nnoommiinnaal l
ppeerrffoorrmmaannccee ggooaallss tthhaatt wweerree,, aatt tthhee ttiimmee, ,trtruulyly sstataggggeerriningg. .TThhisi sseselelectcitoino npproro-
cceessss,, iinn ttuurrnn,, rreeqquuiirreedd aann uunnddeerrssttaannddiinngg oof ftthhee ffuulll lrraannggee oof fisisssuuees sfafaccining gaa
ppaarraalllleell iimmpplleemmeennttaattiioonn..
'W\Thhaatt aarree tthhee hhaarrddwwaarree aarrcchhiitteeccttuurree ooppttiioonnss?? HHooww wweelll lddoo ththeye ymmaatcthc hththe e
ccuussttoommeerr''ss aallggoorriitthhmmiicc aanndd ppeerrffoorrmmaannccee ggooaallss?? DDooeess tthhee pprroobblelemm lleenndd
iittsseellff ttoo ppaarraalllleelliissmm;; iiss tthheerree aannyytthhiinngg aabboouut tiit ttthhaat tmmigighht thhininddere raa ppaararalllelle l
aapppprrooaacchh?? WWhhaatt aabboouutt iinntteeggrraattiinngg ssuucchh aa mmaacchhiinnee iinntoto ththee eexxisitsitning gccoomm-
ppuuttiinngg eennvviirroonnmmeenntt?? WWhhaatt aabboouut tssoofftwtwaarer—e-aallggoorriitthhmmss, ,ooppeerraattiinngg ssyysstteemm, ,
ssooffttwwaarree ddeevveellooppmmeenntt ssuuppppoorrt t ttoooollss aanndd eennvviriroonnmmeennt,t , tetesstitningg, ,
bbeenncchhmmaarrkkiinngg,, eettcc..?? WWhhaatt aabboouutt rriisskk,, eessppeecciiaallllyy bbuussiinneessss rriisskk ffoorr ssmmaalll lhhiigghh-
tteecchh ssttaarrtt--uuppss?? HHooww ccaann ssuuiittaabbllyy bbaallaanncceedd ppaarraalllleel lII//OO bbee aaccccoommpplilsishheedd??
TThhee lliisstt ggooeess oonn aanndd,, wwhheenn tthhee sseelelecctitoionn pprroocceesss sbbeeggaann, ,ththe eininffoormrmaatitoion n
rreeqquuiirreedd ttoo aannsswweerr tthheessee qquueessttiioonnss wwaass ssccaatttteerreedd aanndd cclolouuddeedd bbyy mmiissleleaaddiningg
((oorr,, aatt lleeaasstt,, ffllaawweedd)) aasssseerrttiioonnss ffrroomm bbootthh ssiiddeess ooff tthhee bboorrddeer rsseeppaararatitningg sese-
qquueennttiiaall ffrroomm ppaarraalllleell vveennddoorrss..
TThhee rreemmeeddyy ffoorr tthhiiss,, aanndd tthhee kkeeyy toto mmaakkiningg ththisi sbbooook khhavave evvalaulue,e ,hhasa sbbeeene n
tthhee eexxppeerriieennccee ooff wwoorrkkiinngg aanndd ccoonnssuullttiinngg, ,oovveer rtthhee ininteterIvVeneniningg yyeeaarsrs, ,oon nththe e
ddeevveellooppmmeenntt ooff ffuullll--uupp,, eenndd ttoo eenndd ppaarraalllleel laapppplliiccaatitoionns soonn aa vvaarireietyty ooff eeqquuiipp-
mmeenntt.. TThhee tteerrmm ""pprraaccttiiccaalF'i' nint htheet ittiltelew wasasc chhoosseennt otoc caappttuurree tthhiiss aassppeecctt ooff tthhee
bbooookk''ss iinntteenntt.. WWhhaatt iitt iiss tthhaatt tthhee vveennddoorrss ddoonn't' ttetelll lyyoouu, ,bbuut tththata t(i(ti ttuturnrsn soouut)t )
iiss ccrriittiiccaall,, iinn aa pprraaccttiiccaall sseennssee,, ttoo tthhee ssuucccceessss ooff tthhee aapppplliiccaattiioonn. .AA pprrooppeer raapp-
pprreecciiaattiioonn ffoorr tthheessee ""ggoottcchhaa''ss"" ooff ppaarraalllleell ccoommppuuttiinngg iiss,, II tthhiinnkk,, ooff ggrreeaatt vvaalluuee
iinn mmoovviinngg ppaarraalllleell ccoommppuuttiinngg oouutt ooff ggoovveerrnnmmeenntt--ssppoonnssoorreedd llaabboorraatotorrieies saanndd
iinnttoo aa ccoommmmeerrcciiaall,, ppaayy--yyoouurr--wwaayy sseettttiinngg..
xxiiiiii
xxiivv PPrreeffaaccee
TThhee iinntteennddeedd aauuddiieennccee,, tthheenn, ,aarree tthhoossee wwhhoo, ,lilkikee mmyyseslefl fnninine eyyeaerasr saaggoo, ,
ffiinndd tthheemmsseellvveess ccoonnffrroonnttiinngg tthhee pprraaccttiiccaall rreeaalliittiieess ooff ppaarraalllleel lccoommppuuttiinngg ffoorr
tthhee ffiirrsstt ttiimmee.. AAnn eexxaammppllee ooff ssuucchh aa ppeerrssoonn ((oonnee II hhaadd inin mmyy mminindd's' seeyye eaas s
tthhee wwrriittiinngg pprrooggrreesssseedd)) iiss aa mmiidd--lleevveell tteecchhnniiccaall mmaannaaggeerr ffaacceedd wwiitthh aa pprroocceessss-
iinngg llooaadd tthhaatt iiss iinnccrreeaassiinngg rraappiiddllyy aanndd wwiitthh wwhhiicchh tthhee trtraadditiitoionnaal lvveennddoor rccaann
nnoo lloonnggeerr ccoosstt eeffffeeccttiivveelyly kkeeeepp ppaaccee.. TThhee ppoossssibibiliiltiyty oof fsshhifitfitning gssoomme eoof fththee
ccoommppuuttaattiioonnaallllyy ddeemmaannddiinngg aapppplliiccaattiioonnss ooffff tthhee mmaaiinn ffrraammee aanndd oonntoto aann
iinneexxppeennssiivvee bbuutt ddaazzzzlliinnggllyy ffaasstt ppaarraalllleell pprroocceessssoorr iiss aa ddrreeaamm tthhaatt, ,II ffeeeel lssuurree, ,
hhaass ccrroosssseedd tthhee mmiinnddss ooff mmaannyy aann MMIISS/A/ADDPP pprrooffeessssioionnaal.l .AAt tththaat tppooinint,t ,hhooww-
eevveerr,, tthhee mmaannaaggeerr wwiillll bbee iinn tthhee ssaammee ssiittuuaattioionn II wwaass inin nniinnee yyeeaarsr saaggoo—--tthhaat t
iiss,, ffaacceedd wwiitthh aann aarrrraayy ooff pprraaccttiiccaall qquueessttiioonnss aabboouut ttthhee tteecchhnnoolologgyy ththaat tmmuusts t
bbee aannsswweerreedd bbeeffoorree ssoolliidd cchhooiicceess aanndd ppllaannss ccaann bbee mmaadde eaannd djujustsitfiifeide dtot ouup-p
ppeerr mmaannaaggeemmeenntt..
TThhiiss bbooookk iiss aaddddrreesssseedd ttoo aa ppeerrssoonn iinn jjuusst ttthhaat tssiittuuaattiioonn. .
TThhee ttoopp--lleevveell oorrggaanniizzaattiioonn ooff tthhee bbooookk sseeppaarraatteess tthhee mmaateteriraial linintoto ththrreee e
ppaarrttss::
PPaarrtt 11:: PPrreelliimmiinnaarriieess ((CChhaapptteerrss 11--33))
PPaarrtt 22:: SSooffttwwaarree IIssssuueess ((CChhaapptteerrss 44--99))
PPaarrtt 33:: MMaannaaggeemmeenntt IIssssuueess ((CChhaapptteerrss 1100--1122))
CChhaapptteerr 1122 iiss tthhee ccuullmmiinnaattiioonn ooff tthhee bbooookk.. IIt tpprroovviiddeess aa ssttrruucctuturreedd, ,fflleexx-
iibbllee mmeetthhooddoollooggyy ffoorr sseelleeccttiinngg aa ppaarraalllleell mmaacchhiinnee aanndd ffoor riinntteeggrraatitningg iti tiinntoto
eexxiissttiinngg ooppeerraattiioonnss.. TThhee mmaatteerriiaall aanndd lleevveel loof fddeettaaili linin ooththeer rcchhaapptetersr swwasa s
cchhoosseenn,, iinn llaarrggee ppaarrtt,, ttoo pprroovviiddee tthhee nneecceessssaaryry bbaacckkggrroouunndd ffoor rththe eddisisccuussssioionn
iinn CChhaapptteerr 1122.. FFoorr eexxaammppllee,, CChhaapptteerrss 44 tthhrroouugghh 77 pprroovviiddee wwoorrkkeedd-o-ouut teexx-
aammpplleess,, bbaasseedd oonn aa ttooyy pprroobblleemm,, ooff pprrooggrraammss uussiinngg sseevveerraal lppaarraalllelel lllaanngguuaaggeess. .
SSiinnccee aapppplliiccaattiioonn ssooffttwwaarree iiss,, ppeerrhhaappss,, tthhee kkeeyy mmiissssiinngg iinnggrreeddiieennt tiinn ppaarraalllleel l
ccoommppuuttaattiioonn,, iitt sseeeemmeedd tthhaatt aann eexxtteennddeedd ddiissccuussssioionn wwaass ccaalllleedd ffoorr.: AAss aannootthheerr
eexxaammppllee,, CChhaapptteerr 1100 ddeeaallss wwiitthh bbeenncchhmmaarrkkiinngg aanndd ppeerrffoorrmmaanncce eeessttiimmaattiioonn
oonn ppaarraalllleell mmaacchhiinneess.. SSuucchh aa ddiissccuussssiioonn iiss aapppprroopprriiaattee bbeeccaauussee,, oofftteenn, ,tthhee hhooppee
ooff aacchhiieevviinngg ggrreeaattllyy iinnccrreeaasseedd ppeerrffoorrmmaannccee iiss tthhee ddrriivviinngg mmootitvivaatitoionn ffoor r
mmoovviinngg ttoo ppaarraalllleell eeqquuiippmmeenntt.. IInn CChhaapptteer r33, ,tthhee rreeaaddeer rwwilill lffinindd aa tututotorriaial l
iinnttrroodduuccttiioonn ttoo ppaarraalllleell hhaarrddwwaarree aarrcchhiitteeccttuurreess.. WWhhiillee wwee mmiigghht tbbee aabblele toto
aavvooiidd ssuucchh nniittttyy--ggrriittttyy mmaatttteerrss wwhheenn ddiissccuussssiinngg sseeqquueennttiiaal lmmaacchhiinneess, ,tthhee ccuurr-
rreenntt ssttaattee ooff tthhee aarrtt iinn ppaarraalllleel lpprroocceessssiinngg ddooeess nnoot taallloloww hhaardrdwwaarer eisissusuese stoto
bbee ffiinneesssseedd..
MMyy iinntteenntt hhaass bbeeeenn ttoo pprroovviiddee aa bbaallaanncceedd aasssseessssmmeenntt ooff tthhee tteecchhnonloolgoy~. TThhiiss
iiss nnoott aa ssaalleess bbrroocchhuurree ffoorr ppaarraalllleell pprroocceessssiinngg.. IInnddeeeedd, ,tthhee vvieieww pprreesseenntteedd
Preface XV
comes perilously close to: If you aren't really sure you need massively parallel
computing, you had probably better think twice before getting involved with
it. On the other hand, there has also been an attempt to debunk some of the
more egregious and technically unsupportable charges sometimes made
against parallel machines (see, for example, the discussion of Amdahl's Law
in Chapter 2). The book is liberally sprinkled with examples and case stud-
ies. The tone is, at times, a trifle irreverent—the result of dealing in a field
that is filled with the most extravagant claims and counterclaims, charges and
countercharges. Still, despite these disclaimers, I hope the reader will also
sense my genuine admiration for the technical pioneers who have made pap
allel computing a reality and who continue to work at the difficult remaining
tasks of making it more broadly available and usable.
HSM
May 1994
Ac knowledgments
The person who got this project going in the fall of 1992 was Tom Wheeler,
at that time a senior technical manager with American Express. MRJ, Inc. had
done some work for Tom on a consulting basis, and he approached the com-
pany with the idea of writing a volume on parallel computing as one in a se-
ries on open systems for which he was the editor. Tom was tragically afflicted
with a brain tumor shortly after work began and died in the spring of 1993.
Without Tom's initial impetus and vision, the book would never have been
written.
Another shock came in the spring of 1993 when the intended publisher
underwent a major reorganization. At that point, Alan Rose of Intertext took
the book under his care and eventually found it a home at AP PROFES-
SIONAL. Being novices in the publishing world, MRJ is very grateful for Alan's
help and patience. Without his initiative, the project might have foundered
for lack of publishing interest.
As author, I am indebted to the management at MRJ for their support dur-
ing the writing. Particular thanks go to my Division Manager, Scott Miller, who
gave me flexibility in my schedule to free time for writing; and to Dr. Ed
McMahon, President of MRJ, who made funding available to support the
project. Ed also read through the entire draft manuscript and made a num-
ber of perceptive and valuable suggestions.
The example programs shown in Chapters 4 through 7 are intended to be
compilable and runnable on the target machines. In many cases, the codes
were actually prepared by technical staff from the vendors involved, and in
all cases the sample code, comments, and annotations were reviewed by the
vendors for accuracy. The companies involved were:
Silicon Graphics, Inc. (Power C on the SGI Challenge)
nCube Corporation (nCube 2)
ParaSoft, Inc. (Express)
Intel Corporation (Paragon)
xvii
xviii Acknowledgments
MasPar, Inc. (MPL on the MP-1)
Thinking Machines Corporation (C* on the CM2 and CM5)
Scientific Computing Associates (Linda)
Many thanks to these companies and to their technical staff for helping to
create and review these programming examples.
In many ways, the real heroes behind this book are the members of the
parallel computing technical staff at MRJ. We are small, but we are very good;
the material in this book owes its immediacy and relevance to lessons learned
from real application development on real parallel equipment. Nothing focuses
the attention like having to meet a deliverable—on time and in budget! Spe-
cial thanks go to Steve Geyer, Alan Broder, and George Wilson who read por-
tions of the text and tempered my more extravagant claims (e.g., on debug-
ging) with a healthy dose of common sense and experience.
Finally, thanks to my wife, Mary, and my family for patiently enduring the
emotional vicissitudes of authorship over the past 18 months.
P AR T I
Preliminaries
The three chapters of Part 1 constitute an introduction to the subject of pap
allel processing. The current status is examined in Chapter 1, including dis-
cussions of the performance potential of these machines and the reasons why
they have, thus far, been so unsuccessful in the marketplace. Chapter 2 con-
siders the merits of a commonly cited charge against parallel processors—that
there are theoretical bounds on how many processors can be used efficiently
on a single problem. The analysis concludes that the size of the problem be-
ing solved must enter into such a calculation in a critical way often overlooked
by advocates of sequential and vector machines. Finally, Chapter 3 provides a
tutorial introduction to the three major classes of parallel architecture and
points out strengths and weaknesses in each case.
1
The Status and Future
of Massively
Parallel Processing
Asked to complete the sentence, "Parallel processing is ...," an honest tech-
nical assessment necessarily includes the characterizations "... painful" and
"... inevitable."
If we view the vendors of high-performance computing as residing in two
camps—traditional vector and sequential processing faced off against upstart
parallel processing—both camps have strong vested interests in exaggerating
or minimizing estimates of the kinds and levels of pain. The strong depend-
ence of this pain both on the application and on the operational environment
will defeat any "one size fits all" resolution of these opposing claims: There is
justice on both sides. However, on the issue of inevitability there is broad agree-
ment. Major vendors of traditional architectures have active R&D programs
focused on the development of hardware and software for massively parallel
processors (MPP). This amounts to a tacit admission, marketing hype aside,
that they, too, have seen the technical handwriting on the wall.
For the most part, we will concentrate on the pain of parallel processing.
What are the sources (expected and unexpected)? How can they be avoided
or at least minimized? What characteristics of an application or an operational
environment might facilitate a transition to parallel processing? At the out-
set, however, it is appropriate to consider carefully the inevitability of this tech-
nology. If MPP is inevitably in an organization's future, an understanding of
3
