Executing Multithreaded Programs Efficiently Robert D. Blumofe

138 Pages·2010·0.97 MB·English

Executing Multithreaded Programs Efficiently
by
Robert D. Blumofe
Sc.B., BrownUniversity(1988)
S.M., Massachusetts InstituteofTechnology(1992)
Submittedto theDepartmentofElectricalEngineeringandComputerScience
inpartialfulfillmentoftherequirementsforthedegree of
Doctor of Philosophy
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
September 1995
c RobertD.Blumofe,MCMXCV.Allrightsreserved.
(cid:13)
The authorherebygrantstoMITpermissiontoreproduceanddistributepubliclypaper
andelectroniccopies ofthisthesis documentinwholeorinpart,andtograntothersthe
righttodo so.
Author
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
DepartmentofElectricalEngineeringandComputerScience
August 18,1995
Certified by
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Charles E.Leiserson
Professor
Thesis Supervisor
Accepted by
::::::::::::::::::::::::::::::::::::::::::::::::::::::::
FredericR.Morgenthaler
Chairman, DepartmentalCommitteeonGraduateStudents

Executing Multithreaded Programs Efficiently
by
Robert D. Blumofe
Submittedto theDepartmentofElectricalEngineeringandComputerScience
onAugust18,1995,inpartialfulfillmentofthe
requirementsforthedegreeof
DoctorofPhilosophy
Abstract
Thisthesispresentsthetheory,design,andimplementationofCilk(pronounced“silk”)and
Cilk-NOW. Cilk is a C-based language and portableruntimesystem forprogrammingand
executing multithreaded parallel programs. Cilk-NOW is an implementation of the Cilk
runtimesystemthattransparentlymanagesresourcesforparallelprogramsrunningonanet-
workofworkstations.
Cilk isbuiltarounda provablyefficientalgorithmforschedulingthe executionoffully
strict multithreadedcomputations. Based onthetechniqueofwork stealing,thisalgorithm
achievestime,space,andcommunicationboundsthatareallwithinasmallconstantfactor
ofoptimal. Usingtheseperformanceguarantees,Cilkprovidesasimpleabstractionofper-
formance that allows programmers to predict accurately how programchanges will affect
executiontime.
Cilk-NOWprovideshigh-levelresourcemanagementservicessothatCilkprogramscan
run efficiently on a network of workstations. These services include adaptive parallelism
andfaulttolerance. Withadaptiveparallelism,programsexecuteonapoolofworkstations
that dynamicallygrowsand shrinks in response toboth the availabilityof“idle” machines
andtheavailabilityofparallelismwithintheprogram. Cilk-NOWalsoprovidestransparent
faulttolerancewithafullydistributedcheckpointingmechanism.
parallelcomputing,distributedcomputing,dataflow,multithreadedprograms,
Keywords:
parallel languages, parallel runtime systems, Cilk, strict execution, stack memory, thread
scheduling,schedulingalgorithms,randomizedalgorithms,delaysequence, workstealing,
computerchess, massively parallelsupercomputer,network ofworkstations,adaptive par-
allelism,faulttolerance.
CharlesE. Leiserson
Thesis Supervisor:
Professor
Title:

Acknowledgments
IrememberthefirsttimeImetCharlesLeiserson anddiscussed possibleresearchprojects.
He spoke of some grand ideas, and he waved his arms a lot. Something about trees. We
were going to solve some importantproblems together. Only one difficulty: I had no idea
what he was talking about. For better or worse, I now seem to understand him, and our
collaborationdidproducesomeresults thatIamveryproudof. Charles hasaterrificsense
of research direction,and ratherthan push me in any direction,he pulled. Rather than tell
me where to go, he showed the way. All the while, Charles left me room to be creative
and pursue paths that I found myself. He supported me in all my research pursuits with
hisownenthusiasmandwithagenerousallocationofresources,bothhumanandmachine.
Charles also has a great sense of humor, and we had fun, even giggling like children one
afternoonas we imaginedhowmuch morefunwe wouldhave playingthe recyclinggame
(see Section 4.2) instead of hockey. Charles tells me that now I speak in grand ideas and
wave myarmsa lot. Couldbe worse.
Besides serving as my advisor, collaborator, and friend, Charles also showed me how
toclarifymywritingwithfocused workand patience. Itdidnotcomeeasy. Afterone par-
ticularlylongandgruelingsession ofwriting(Charleshasinfinitepatience),IleftCharles’
office and the first person I stumbled upon was Tom Cormen. With a sigh, I asked him,
“have you ever written a paper with Charles?” He just looked at me like that is the stupi-
dest questionI’veeverasked him(see[33]). Itisn’t.
Much of the work in this thesis could not have been done without the collaboration
of the entire Cilk team. Led by Charles, this crackerjack team includes or has included
Shail Aditya now ofHewlett Packard, Matteo Frigo, Michael Halbherrnow of the Boston
Consulting Group, Chris Joerg, Bradley Kuszmaul now of Yale University, Phil Lisiecki,
The research reportedin thisthesiswas supportedinpart by theAdvanced Research Projects Agency
(ARPA)underGrants N00014-94-1-0985and N00014-92-J-1310. Inaddition,Iwas generouslysupported
throughoutmygraduatecareerbyaNationalScienceFoundation(NSF)GraduateStudentFellowshipandan
ARPAHigh-PerformanceComputingGraduateFellowshipadministeredthroughtheUniversityofMaryland
InstituteforAdvancedComputerStudy.
iii
iv
Howard Lu, Rob Miller now of Carnegie Mellon University, David Park, Keith Randall,
LarryRudolph,SivanToledonowofIBM,andYuliZhounowofAT&TBellLaboratories.
ManypeopleatMIThavemadesignificantcontributionstotheworkinmythesis. Much
ofthecombinatorialtechniqueandthedelay-sequenceargumentusedinChapter4,Ilearned
from Tom Leighton. Bonnie Berger helped with some of my early probabilistic analysis.
EstherJesurumandEthanWolfhelpedwiththelowerbound. EricBrewernowoftheUni-
versityof CaliforniaatBerkeley was my collaboratorin some ofmy earlyforaysinto sys-
tembuilding. OtherswhohavelentinspirationandideasincludeArvind,TomCormennow
ofDartmouthCollege,Frans Kaashoek,ArthurLent,GregPapadopoulosnowof Sun,and
BillWeihl. ScottBlomquistkepttheCM5andournetworkofworkstationsupandrunning.
DavidJonesprovidedextensivehelpwithsomeofthesubtletiesofLATEX. Otherswhohave
helpedmakemylifeeasy(andfun)includeBeHubbard,IrenaKveraga,andCherylPatton.
SeveralpeoplefromoutsideMITalsomadesignificantcontributionstotheworkinthis
thesis. BruceMaggsofCarnegieMellonUniversityoutlinedthedelaysequenceargument,
andGregPlaxtonoftheUniversityofTexasatAustingavetechnicalcommentsontherecy-
clinggameanalysis. Otherswhocontributedideas includeGuyBlellochofCarnegieMel-
lon University,WarrenBurton ofSimonFraser University,and Yanjun Zhang ofSouthern
MethodistUniversity.
I would not have made it to MIT from Brown (where I was an undergraduate) had it
not been for some of the inspiring professors in Brown’s math and computer science de-
partments. TomBanchofffirstturnedmeontocomputergraphics;JeffVitter,nowofDuke
University, first turned me on to theory and algorithms; John Savage first turned me on to
parallelcomputation;andAndyvanDamfirstturnedmeontomyfutureincomputation. I
tookmyfirstcomputerscienceclassfromAndy,andItookmyfirstadventureintoresearch
inAndy’sgraphicsgroup. Fromthisbeginning,Andyinfectedmewithhisexuberantsense
of excitement and wonder over a future in which anything can happen when computation
meetsimagination. IwasalessthanmodelstudentinmyearlyyearsatBrown,andIknow
Ilethimdownonmorethanoneoccasion,butAndymusthaveseensomethingthatnobody
elsecould,becausehecontinuedtotakeriskswithme,presentingmewithoneopportunity
afteranother. WhenItookayearofffromBrown,Andygotmeagreatjobatasmallstartup
companycalledCadreTechnologies,andlateritwasAndywhofirstsuggested thatIapply
to graduate school and then backed up the suggestion with a strong letterof recommenda-
v
tion. Onthoselasttwoopportunities,IthinkIfinallycamethrough. Idon’tknowwhyAndy
took such a personal interest in me or why he was so unwaivering in his confidence, but I
thinkabouthimoftenandwithgratitudeandaffection.
Myyearstodatehavebeenfilledwithhumor,adventure,satisfaction, andfun,because
I have had the good fortune of good friends, some of whom I shall now mention ordered
fromoldesttomostrecent. DannyKlein: Dannywasmysummerbrotherandbestfriendfor
many years. The gang outat HortonLakes, including theHortonfamily—Jack, Gretchen,
andJohn—GregCooper,andmanyotherswhohavecome andgone: Ithasbeenmyprivi-
legeandjoytowaterskiwiththesefolksattheirworld-classsite. MaxKennedy: I’venever
metanyonewithmoreenergy,lessfear,orabiggerheart,andheyman,MaxandIhadsome
seriousfun. JeffCutler: JeffandIdrovecrosscountryfourtimes,discoveredtheprecipice,
watched for suicide deer, went to Pete Freehafer’s wedding as the Goldberg brothers, and
had countless other silly adventures. I can’t even write this without laughing. Liz Cohen:
LizandIseemtohaveanunbreakablebondforgedthroughsomestrangedaysandstranger
nights. AnnaHedly: EveryonewhoknowsmewellknowshowimportantAnnawasandis
tome. Werarelytalkorsee eachother,butIstillthinkaboutheroften. Also,Annagotme
started as a pilot. Janet Barber: Janet is probablymybest friendandthe person I mosten-
joytalkingtoandhangingaroundwith. GoingtoDeadshowsandlivinglargewithJanetin
Vegas was surelythe best possible way to turnthirty. The gangfromMotion Dynamics—
Slim, Dave, Mike, and Leo: These guys fixed my little race car whenever I wadded it up
in a ball (often),and they were a hell ofa lot offun to hang with. Formerand present-day
Cadroids—Lou Bershad, Reed Flemming, Alan Hecht, Lou Mazzuchelli, Lou Reynolds,
andothers: Theywereandstillareatalentedandfungroupofpeopletoworkandplaywith.
TheentireBar-Yamfamily—Miriam,Zvi, Yaneer,Aureet,andSageet: The Bar-Yamsfirst
were my landlords in Cambridge but foremost are my cherished friends. Esther Jesurum:
Esther got me started playing hockey and was a fun waterskiing partner. Helen Greiner:
Helenwas always upforsome fun. Shewoulddoanythingexcept wearasilly hat.
Myfamilyissomewhatdistantandratherpeculiarbutalsoquitewonderful,andIthank
themfortheirsupportandtolerance. MyolderbrotherMichaelwasmyrolemodelgrowing
up, and he inspired my interest in math and science. Michael is now an emergency room
doctor and saves lives every day. (On a good day, I can improve an algorithm.) My older
sisterMariafirstshowedmetheadventurous,andoccasionally dangerous,sideoflife. My
vi
youngersister Joanna is myclosest relativespatially (shelives nearbyin theBoston area),
temporally(weareoneyearandsix daysapartinage),andemotionally. Joanna ismarried
to Brad Meiseles and has two sons Benjamin and Brandon. I have really enjoyed having
family, especially Benjamin and Brandon, nearby. My mother gave me tremendous con-
fidence and high expectations, and these qualities have broughtme throughsome difficult
times. IthinkIgettherationalsideofmycharacterfromher. Myfathercametoparenthood
ratherlateinlife,butIdon’tknowthatanyoneeverworkedatitharderorenjoyeditmore.
IthinkIgettheemotionalside ofmycharacterfromhim. IhopeIcanbeas welllikedand
wellrespected inmyprofessionas heseems tohavebeen inhis.
ROBERT D. BLUMOFE
Cambridge, Massachusetts
August 18,1995

Contents
1 Executingmultithreadedprograms efficiently 1
1.1 Schedulingmultithreadedprograms . . . . . . . . . . . . . . . . . . . . . 1
1.2 Previousresults andrelatedwork . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Contributionsofthisthesis . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Amodel ofmultithreadedcomputation 11
2.1 Multithreadedcomputation . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Execution time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Execution space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 Strictmultithreadedcomputations 25
3.1 A lowerboundforthegeneralcase . . . . . . . . . . . . . . . . . . . . . . 25
3.2 The strictness condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 The busy-leaves property . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4 Workstealing 37
4.1 A randomizedwork-stealingalgorithm . . . . . . . . . . . . . . . . . . . . 37
4.2 Atomicmessages andtherecyclinggame . . . . . . . . . . . . . . . . . . 41
4.3 Analysis ofthework-stealingalgorithm . . . . . . . . . . . . . . . . . . . 45
5 Parallelprogramming inCilk 53
5.1 The Cilklanguage andruntimesystem . . . . . . . . . . . . . . . . . . . . 53
5.1.1 Spawnsandsyncs . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.1.2 Continuation-passingthreads . . . . . . . . . . . . . . . . . . . . . 56
5.1.3 The Cilkwork-stealingscheduler . . . . . . . . . . . . . . . . . . 62
5.2 PerformanceofCilkapplications . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.1 Cilkapplications . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.2 Applicationperformance . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.3 Performancemodeling . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3 A theoreticalanalysis ofthe Cilkscheduler . . . . . . . . . . . . . . . . . 75
6 Cilkonanetworkofworkstations 89
6.1 Systemarchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2 Adaptiveparallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.3 Faulttolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7 Conclusions 105