High Performance Datacenter Networks. Architectures,Algorithms,and Opportunities

116 Pages·2011·3.31 MB·English

by  Dennis Abts, John Kim

SSSeeerrireiieess sI IISSSSSSNNN: ::1 11999333555---333222333555 AAA
BBB MMM
TTT
SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN S • KS • KS • K &&&CCC MMMooorrrgggaaannn& & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss
III
CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE MMM
SSSeeerrriiieeesss E EEdddiiitttooorrr::: M MMaaarrrkkk D DD. .. H HHiiillllll, ,, UUUnnniiivvveeerrrsssiiitttyyy o oofff W WWiiissscccooonnnsssiiinnn
HHHiiiggghhh PPPeeerrrfffooorrrmmmaaannnccceee
HHHiiiggghhh PPPeeerrrfffooorrrmmmaaannnccceee DDDaaatttaaaccceeennnttteeerrr NNNeeetttwwwooorrrkkksss
AAArrrccchhhiiittteeeccctttuuurrreeesss,,, AAAlllgggooorrriiittthhhmmmsss,,, aaannnddd OOOppppppooorrrtttuuunnniiitttyyy HHH
III DDDaaatttaaaccceeennnttteeerrr NNNeeetttwwwooorrrkkksss
GGG
HHH
DDDeeennnnnniiisss A AAbbbtttsss,, ,G GGoooooogggllelee I IInnnccc.. . a aannnddd J JJooohhhnnn K KKiiimmm,, ,K KKooorrreeeaaa A AAdddvvvaaannnccceeeddd I IInnnssstttiiitttuuuttteee o oofff S SSccceeeiiinnnccceee a aannnddd T TTeeeccchhhnnnooolloloogggyyy

PPP
EEE
RRR
DDDaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss p pprrrooovvviididdeee t tthhheee c ccooommmmmmuuunnniiciccaaatttiioioonnn s ssuuubbbssstttrrraaattteee f ffooorrr l lalaarrrgggeee p ppaaarrraaallllleleell l c ccooommmpppuuuttteeerrr s ssyyysssttteeemmmsss t tthhhaaattt FFF AAArrrccchhhiiittteeeccctttuuurrreeesss,,, AAAlllgggooorrriiittthhhmmmsss,,, aaannnddd OOOppppppooorrrtttuuunnniiitttyyy
OOO
fffooorrrmmm t tthhheee e eecccooosssyyysssttteeemmm f ffooorrr h hhiigigghhh p ppeeerrrfffooorrrmmmaaannnccceee c ccooommmpppuuutttiininnggg ( ((HHHPPPCCC))) s ssyyysssttteeemmmsss a aannnddd m mmooodddeeerrrnnn I IInnnttteeerrrnnneeettt a aappppppllili-i-- RRR
MMM
cccaaatttiioioonnnsss. .. T TThhheee d ddeeesssiigiggnnn o oofff n nneeewww d ddaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss i isiss m mmoootttiivivvaaattteeeddd b bbyyy a aannn a aarrrrrraaayyy o oofff a aapppppplliliciccaaatttiioioonnnsss r rraaannngggiininnggg
AAA
fffrrrooommm c ccooommmmmmuuunnniiciccaaatttiioioonnn i ininnttteeennnsssiivivveee c ccllilimimmaaatttooolloloogggyyy, ,, c ccooommmppplleleexxx m mmaaattteeerrriiaiaall l s ssiimimmuuullalaatttiioioonnnsss a aannnddd m mmooolleleecccuuullalaarrr d ddyyynnnaaammmiiciccsss NNN
CCC
tttooo s ssuuuccchhh I IInnnttteeerrrnnneeettt a aapppppplliliciccaaatttiioioonnnsss a aasss W WWeeebbb s sseeeaaarrrccchhh, ,, l lalaannnggguuuaaagggeee t ttrrraaannnsssllalaatttiioioonnn, ,, c ccooolllllalaabbbooorrraaatttiivivveee I IInnnttteeerrrnnneeettt a aapppppplliliciccaaatttiioioonnnsss,,, EEE

ssstttrrreeeaaammmiininnggg v vviididdeeeooo a aannnddd v vvoooiicicceee---ooovvveeerrr---IIIPPP. .. F FFooorrr b bbooottthhh S SSuuupppeeerrrcccooommmpppuuutttiininnggg a aannnddd C CClloloouuuddd C CCooommmpppuuutttiininnggg t tthhheee n nneeetttwwwooorrrkkk DDD
AAA
eeennnaaabbblleleesss d ddiisisstttrrriibibbuuuttteeeddd a aapppppplliliciccaaatttiioioonnnsss t ttooo c ccooommmmmmuuunnniiciccaaattteee a aannnddd i ininnttteeerrrooopppeeerrraaattteee i ininn a aannn o oorrrccchhheeessstttrrraaattteeeddd a aannnddd e eeffffffiicicciieieennnttt TTT
AAA
wwwaaayyy... CCC
EEE DDDeeennnnnniiisss AAAbbbtttsss
TTThhhiisiss b bbooooookkk d ddeeessscccrrriibibbeeesss t tthhheee d ddeeesssiigiggnnn a aannnddd e eennngggiininneeeeeerrriininnggg t ttrrraaadddeeeoooffffffsss o oofff d ddaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss. .. I IIttt d ddeeessscccrrriibibbeeesss NNN
TTT
iininnttteeerrrcccooonnnnnneeeccctttiioioonnn n nneeetttwwwooorrrkkksss f ffrrrooommm t ttooopppooolloloogggyyy a aannnddd n nneeetttwwwooorrrkkk a aarrrccchhhiititteeeccctttuuurrreee t ttooo r rrooouuutttiininnggg a aallglggooorrriititthhhmmmsss, ,, a aannnddd p pprrreeessseeennntttsss EEE JJJooohhhnnn KKKiiimmm
RRR
oooppppppooorrrtttuuunnniitittiieieesss f ffooorrr t ttaaakkkiininnggg a aadddvvvaaannntttaaagggeee o oofff t tthhheee e eemmmeeerrrgggiininnggg t tteeeccchhhnnnooolloloogggyyy t ttrrreeennndddsss t tthhhaaattt a aarrreee i ininnffflluluueeennnccciininnggg r rrooouuuttteeerrr
NNN
mmmiiciccrrroooaaarrrccchhhiititteeeccctttuuurrreee. .. W WWiititthhh t tthhheee e eemmmeeerrrgggeeennnccceee o oofff “ ““mmmaaannnyyy---cccooorrreee”” ” p pprrroooccceeessssssooorrr c cchhhiipippsss, ,, i ititt i isiss e eevvviididdeeennnttt t tthhhaaattt w wweee w wwiilillll l a aallslssooo EEE
TTT
nnneeeeeeddd “ ““mmmaaannnyyy---pppooorrrttt”” ” rrrooouuutttiininnggg ccchhhiipippsss tttooo ppprrrooovvviididdeee aaa bbbaaannndddwwwiididdttthhh---rrriicicchhh nnneeetttwwwooorrrkkk tttooo aaavvvoooiididd ttthhheee pppeeerrrfffooorrrmmmaaannnccceee WWW
OOO
llilimimmiitittiininnggg eeeffffffeeeccctttsss ooofff AAAmmmdddaaahhhll’ls’’ss LLLaaawww. .. W WWeee ppprrrooovvviididdeee aaannn ooovvveeerrrvvviieieewww ooofff cccooonnnvvveeennntttiioioonnnaaall l tttooopppooolloloogggiieieesss aaannnddd ttthhheeeiirirr
RRR
rrrooouuutttiininnggg a aallglggooorrriititthhhmmmsss a aannnddd s sshhhooowww h hhooowww t tteeeccchhhnnnooolloloogggyyy, ,, s ssiigiggnnnaaallilininnggg r rraaattteeesss a aannnddd c ccooosssttt---eeeffffffeeeccctttiivivveee o ooppptttiiciccsss a aarrreee m mmoootttiivivvaaatttiininnggg KKK
SSS
nnneeewww nnneeetttwwwooorrrkkk tttooopppooolloloogggiieieesss ttthhhaaattt ssscccaaallelee uuuppp tttooo mmmiilillllilioioonnnsss ooofff hhhooossstttsss. .. T TThhheee bbbooooookkk aaallslssooo ppprrrooovvviididdeeesss dddeeetttaaaiilileleeddd cccaaassseee
ssstttuuudddiieieesss o oofff t ttwwwooo h hhiigigghhh p ppeeerrrfffooorrrmmmaaannnccceee p ppaaarrraaallllleleell l c ccooommmpppuuuttteeerrr s ssyyysssttteeemmmsss a aannnddd t tthhheeeiirirr n nneeetttwwwooorrrkkksss...
AAAbbbooouuuttt S SSYYYNNNTTTHHHEEESSSIIIsss
MMM
TTThhhiisiss v vvooolluluummmeee i isiss a aa p pprrriininntteteeddd v vveeerrrsssiioioonnn o ooff f a aa w wwooorrrkkk t ththhaaatt t a aappppppeeeaaarrrsss i ininn t ththheee S SSyyynnntththheeesssiisiss
OOO
DDDiigiggiititataall lL LLiibibbrrraaarrryyy o ooff fE EEnnngggiininneeeeeerrriininnggg a aannnddd C CCooommmpppuuutteteerrr S SSccciieieennnccceee. ..S SSyyynnntththheeesssiisiss L LLeeecccttutuurrreeesss RRR
ppprroroovvvidiiddeee c ccooonnncccisiisseee, ,,o oorrirgiigginiinnaaal llp pprrereessseeennnttataattitoiioonnnsss o ooff fi miimmpppooorrrttataannntt tr rereessseeeaaarrcrcchhh a aannnddd d ddeeevvveeelolloopppmmmeeennnttt GGG
AAA
ttotoopppiiciccsss, ,,p ppuuubbblillisisshhheeeddd q qquuuiicicckkkllylyy, ,,i ininn d ddiigiggiititataal lla aannnddd p pprrriininntt tf fofoorrrmmmaaattstss. ..F FFooorrr m mmooorreree i ininnffofoorrrmmmaaattitioioonnn NNN
vvviisissiitit t w wwwwwwww..m.mmooorrrgggaaannncccllalaayyypppooooooll.l.c.ccooommm &&&
CCC SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN
IIISSSBBBNNN::: 999777888---111---666000888444555---444000222---000 LLL
MMMooorrrgggaaannn & & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss 999000000000000 AAA
YYY CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE
PPP
wwwwwwwww...mmmooorrrgggaaannnccclllaaayyypppoooooolll...cccooommm OOO
999777888111666000888444555444000222000 OOO
MMMaaarrrkkk DDD.. . HHHiiillllll,, , SSSeeerrriiieeesss E EEdddiiitttooorrr
LLL
High Performance
Datacenter Networks
Architectures,Algorithms,andOpportunities
Synthesis Lectures on Computer
Architecture
Editor
MarkD.Hill,UniversityofWisconsin
SynthesisLecturesonComputerArchitecturepublishes50-to100-pagepublicationsontopics
pertainingtothescienceandartofdesigning,analyzing,selectingandinterconnectinghardware
componentstocreatecomputersthatmeetfunctional,performanceandcostgoals.Thescopewill
largelyfollowthepurviewofpremiercomputerarchitectureconferences,suchasISCA,HPCA,
MICRO,andASPLOS.
HighPerformanceDatacenterNetworks:Architectures,Algorithms,andOpportunities
DennisAbtsandJohnKim
2011
QuantumComputingforArchitects,SecondEdition
TzvetanMetodi,FredChong,andArvinFaruque
2011
ProcessorMicroarchitecture:AnImplementationPerspective
AntonioGonzález,FernandoLatorre,andGrigoriosMagklis
2010
TransactionalMemory,2ndedition
TimHarris,JamesLarus,andRaviRajwar
2010
ComputerArchitecturePerformanceEvaluationMethods
LievenEeckhout
2010
IntroductiontoReconfigurableSupercomputing
MarcoLanzagorta,StephenBique,andRobertRosenberg
2009
On-ChipNetworks
NatalieEnrightJergerandLi-ShiuanPeh
2009
iii
TheMemorySystem:YouCan’tAvoidIt,YouCan’tIgnoreIt,YouCan’tFakeIt
BruceJacob
2009
FaultTolerantComputerArchitecture
DanielJ.Sorin
2009
TheDatacenterasaComputer:AnIntroductiontotheDesignofWarehouse-ScaleMachines
freeaccess
LuizAndréBarrosoandUrsHölzle
2009
ComputerArchitectureTechniquesforPower-Efficiency
StefanosKaxirasandMargaretMartonosi
2008
ChipMultiprocessorArchitecture:TechniquestoImproveThroughputandLatency
KunleOlukotun,LanceHammond,andJamesLaudon
2007
TransactionalMemory
JamesR.LarusandRaviRajwar
2006
QuantumComputingforComputerArchitects
TzvetanS.MetodiandFredericT.Chong
2006
Copyright© 2011byMorgan&Claypool
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin
anyformorbyanymeans—electronic,mechanical,photocopy,recording,oranyotherexceptforbriefquotationsin
printedreviews,withoutthepriorpermissionofthepublisher.
HighPerformanceDatacenterNetworks:Architectures,Algorithms,andOpportunities
DennisAbtsandJohnKim
www.morganclaypool.com
ISBN:9781608454020 paperback
ISBN:9781608454037 ebook
DOI10.2200/S00341ED1V01Y201103CAC014
APublicationintheMorgan&ClaypoolPublishersseries
SYNTHESISLECTURESONCOMPUTERARCHITECTURE
Lecture#14
SeriesEditor:MarkD.Hill,UniversityofWisconsin
SeriesISSN
SynthesisLecturesonComputerArchitecture
Print1935-3235 Electronic1935-3243
High Performance
Datacenter Networks
Architectures,Algorithms,andOpportunities
Dennis Abts
GoogleInc.
John Kim
KoreaAdvancedInstituteofScienceandTechnology(KAIST)
SYNTHESISLECTURESONCOMPUTERARCHITECTURE#14
M
&C Morgan &cLaypool publishers
ABSTRACT
Datacenternetworksprovidethecommunicationsubstrateforlargeparallelcomputersystemsthat
form the ecosystem for high performance computing (HPC) systems and modern Internet appli-
cations.The design of new datacenter networks is motivated by an array of applications ranging
fromcommunicationintensiveclimatology,complexmaterialsimulationsandmoleculardynamics
tosuchInternetapplicationsasWebsearch,languagetranslation,collaborativeInternetapplications,
streamingvideoandvoice-over-IP.ForbothSupercomputingandCloudComputingthenetwork
enables distributed applications to communicate and interoperate in an orchestrated and efficient
way.
This book describes the design and engineering tradeoffs of datacenter networks. It de-
scribes interconnection networks from topology and network architecture to routing algorithms,
and presents opportunities for taking advantage of the emerging technology trends that are influ-
encing router microarchitecture.With the emergence of“many-core” processor chips,it is evident
thatwewillalsoneed“many-port”routingchipstoprovideabandwidth-richnetworktoavoidthe
performancelimitingeffectsofAmdahl’sLaw.Weprovideanoverviewofconventionaltopologies
andtheirroutingalgorithmsandshowhowtechnology,signalingratesandcost-effectiveopticsare
motivatingnewnetworktopologiesthatscaleuptomillionsofhosts.Thebookalsoprovidesdetailed
casestudiesoftwohighperformanceparallelcomputersystemsandtheirnetworks.
KEYWORDS
networkarchitectureanddesign,topology,interconnectionnetworks,fiberoptics,par-
allelcomputerarchitecture,systemdesign
vii
Contents
Preface .................................................................. xi
Acknowledgments.......................................................xiii
NotetotheReader........................................................xv
1 Introduction ..............................................................1
1.1 FromSupercomputingtoCloudComputing................................ 3
1.2 Beowulf:TheClusterisBorn............................................. 3
1.3 OverviewofParallelProgrammingModels................................. 4
1.4 Puttingitalltogether ................................................... 5
1.5 QualityofService(QoS)requirements..................................... 6
1.6 Flowcontrol ........................................................... 7
1.6.1 Lossyflowcontrol ................................................ 7
1.6.2 Losslessflowcontrol .............................................. 8
1.7 Theriseofethernet ..................................................... 9
1.8 Summary.............................................................. 9
2 Background ............................................................ 13
2.1 Interconnectionnetworks ............................................... 13
2.2 Technologytrends ..................................................... 13
2.3 Topology,RoutingandFlowControl ..................................... 16
2.4 CommunicationStack.................................................. 16
3 TopologyBasics ......................................................... 19
3.1 Introduction .......................................................... 19
3.2 TypesofNetworks ..................................................... 20
3.3 Mesh,Torus,andHypercubes ........................................... 20
3.3.1 Nodeidentifiers ................................................. 22
3.3.2 k-aryn-cubetradeoffs............................................ 22
viii
4 High-RadixTopologies .................................................. 25
4.1 TowardsHigh-radixTopologies.......................................... 25
4.2 TechnologyDrivers .................................................... 26
4.2.1 PinBandwidth.................................................. 26
4.2.2 EconomicalOpticalSignaling ..................................... 29
4.3 High-RadixTopology .................................................. 30
4.3.1 High-DimensionHypercube,Mesh,Torus .......................... 30
4.3.2 Butterfly ....................................................... 30
4.3.3 High-RadixFolded-Clos ......................................... 31
4.3.4 FlattenedButterfly .............................................. 34
4.3.5 Dragonfly ...................................................... 34
4.3.6 HyperX ........................................................ 37
5 Routing ................................................................ 39
5.1 RoutingBasics ........................................................ 39
5.1.1 ObjectivesofaRoutingAlgorithm................................. 40
5.2 MinimalRouting ...................................................... 40
5.2.1 DeterministicRouting ........................................... 40
5.2.2 ObliviousRouting ............................................... 41
5.3 Non-minimalRouting.................................................. 41
5.3.1 Valiant’salgorithm(VAL) ........................................ 42
5.3.2 UniversalGlobalAdaptiveLoad-Balancing(UGAL) ................ 42
5.3.3 ProgressiveAdaptiveRouting(PAR) ............................... 43
5.3.4 Dimensionally-Adaptive,Load-balanced(DAL)Routing ............. 43
5.4 IndirectAdaptiveRouting .............................................. 43
5.5 RoutingAlgorithmExamples ........................................... 44
5.5.1 Example1:Folded-Clos .......................................... 45
5.5.2 Example2:FlattenedButterfly .................................... 45
5.5.3 Example3:Dragonfly ............................................ 49
6 ScalableSwitchMicroarchitecture......................................... 51
6.1 RouterMicroarchitectureBasics ......................................... 51
6.2 Scalingbaselinemicroarchitecturetohighradix ............................ 52
6.3 FullyBufferedCrossbar ................................................ 54
6.4 HierarchicalCrossbarArchitecture....................................... 55
6.5 ExamplesofHigh-RadixRouters ........................................ 57
ix
6.5.1 CrayYARCRouter .............................................. 57
6.5.2 MellanoxInfiniScaleIV .......................................... 59
7 SystemPackaging ....................................................... 63
7.1 Packaginghierarchy.................................................... 63
7.2 Powerdeliveryandcooling .............................................. 63
7.3 TopologyandPackagingLocality ........................................ 68
8 CaseStudies ............................................................ 73
8.1 CrayBlackWidowMultiprocessor ....................................... 73
8.1.1 BlackWidowNodeOrganization .................................. 73
8.1.2 High-radixFolded-ClosNetwork.................................. 74
8.1.3 SystemPackaging ............................................... 75
8.1.4 High-radixFat-tree.............................................. 76
8.1.5 PacketFormat .................................................. 77
8.1.6 NetworkLayerFlowControl ..................................... 78
8.1.7 Data-linkLayerProtocol ......................................... 78
8.1.8 Serializer/Deserializer............................................ 80
8.2 CrayXTMultiprocessor................................................ 80
8.2.1 3-Dtorus ...................................................... 81
8.2.2 Routing ....................................................... 82
8.2.3 FlowControl ................................................... 84
8.2.4 SeaStarRouterMicroarchitecture ................................. 84
8.3 Summary............................................................. 88
9 ClosingRemarks ........................................................ 91
9.1 Programmingmodels .................................................. 91
9.2 Wireprotocols ........................................................ 91
9.3 Opportunities ......................................................... 92
Bibliography............................................................ 93
Authors’Biographies .................................................... 99