Resilient Overlay Networks David G. Andersen

86 Pages·2001·2.12 MB·English

Resilient Overlay Networks
by
David G. Andersen
B.S., ComputerScience, UniversityofUtah(1998)
B.S., Biology,UniversityofUtah(1998)
SubmittedtotheDepartmentofElectrical EngineeringandComputerScience
inpartial fulfillmentoftherequirementsforthedegreeof
MasterofScience inComputerScience andEngineering
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
May2001
c MassachusettsInstituteofTechnology2001. Allrightsreserved.
(cid:13)
Author...........................................................................
Department ofElectrical Engineeringand ComputerScience
May16, 2001
Certified by.......................................................................
Hari Balakrishnan
AssistantProfessor
ThesisSupervisor
Accepted by......................................................................
ArthurC. Smith
Chairman,DepartmentCommitteeon GraduateStudents
ResilientOverlayNetworks
by
DavidG. Andersen
SubmittedtotheDepartmentofElectrical Engineering andComputerScience
onMay16,2001,inpartialfulfillmentofthe
requirements forthedegreeof
MasterofScienceinComputerScienceandEngineering
Abstract
This thesis describes the design, implementation, and evaluation of a Resilient Overlay Network
(RON),anarchitecturethatallowsend-to-endcommunicationacrossthewide-areaInternettodetect
and recover from path outages and periods of degraded performance within several seconds. A
RON is an application-layer overlay on top of the existing Internet routing substrate. The overlay
nodes monitor the liveness and quality of the Internet paths among themselves, and they use this
information to decide whether to route packets directly over the Internet or by way of other RON
nodes, optimizingapplication-specific routingmetrics.
We demonstrate the potential benefits of RON by deploying and measuring a working RON
with nodes at thirteen sites scattered widely over the Internet. Over a 71-hour sampling period
in March 2001, there were 32 significant outages lasting over thirty minutes each, between the
156communicatingpairsofRONnodes. RON’sroutingmechanismwasabletodetectandrecover
aroundallofthem,showingthatthereis,infact,physicalpathredundancyintheunderlyingInternet
inmanycases. RONsarealsoabletoimprovethelossrate,latency,orthroughputperceivedbydata
transfers; for example, about 1% of the transfers doubled their TCP throughput and 5% of our
transfers saw their loss rate reduced by 5% inabsolute terms. These improvements, particularly in
theareaoffaultdetectionandrecovery,demonstratethebenefitsofmovingsomeofthecontrolover
routingintothehandsofend-systems.
ThesisSupervisor: HariBalakrishnan
Title: AssistantProfessor
2
Tomymother,whoseunparalleled chickenstockexpertise wasinvaluable.
3
Contents
1 Introduction 12
1.1 TheInternetandOverlayNetworks. . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 BackgroundandRelatedWork 17
2.1 Internet Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.1 TheOrganization oftheInternet . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.2 Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 OverlayNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.1 MBone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.2 6-bone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.3 TheX-Bone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.4 Yoid/Yallcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.5 EndSystemMulticast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.6 ALMI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.7 Overcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.8 ContentDeliveryNetworks . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.9 Peer-to-peer Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Performance MeasuringSystemsandTools . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 NIMI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 IDMaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.3 SPAND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.4 Detour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.5 CommercialMeasurementProjects . . . . . . . . . . . . . . . . . . . . . 26
4
2.3.6 LatencyandLossProbeTools . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.7 Available BandwidthProbeTools . . . . . . . . . . . . . . . . . . . . . . 27
2.3.8 LinkBandwidthProbeTools . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.9 Standards forTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 ProblemswithInternetReliability 28
3.1 Slowlinkfailurerecovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Inability todetectperformance failures . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3 Inability toeffectivelymulti-home . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4 Bluntpolicyexpression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.5 DefiningOutages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4 RONDesignandImplementation 34
4.1 DesignOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2 SystemModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 Bootstrap andMembershipManagement . . . . . . . . . . . . . . . . . . . . . . . 38
4.3.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.4 DataForwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.4.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.5 Monitoring VirtualLinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.6 Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.7 Link-StateDissemination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.8 PathEvaluation andSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.8.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.9 PolicyRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.10 Performance Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.10.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.11 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5
5 Evaluation 53
5.1 RONDeployment Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.1.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.1.2 OvercomingPathoutages . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.1.3 OutageDetectionTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.1.4 OvercomingPerformanceFailures . . . . . . . . . . . . . . . . . . . . . . 63
5.2 Handling PacketFloods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3 OverheadAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4 RON-induced FailureModes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.5 Simulations ofRONRoutingBehavior . . . . . . . . . . . . . . . . . . . . . . . . 77
5.5.1 Single-hop v. Multi-hopRONs . . . . . . . . . . . . . . . . . . . . . . . . 77
5.5.2 RONRouteInstability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6 ConclusionandFutureWork 79
6
List of Figures
1-1 TheRONoverlayapproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-1 AnexampleofInternet interconnections . . . . . . . . . . . . . . . . . . . . . . . 18
2-2 Tier1and2providers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-1 LimitationsofBGPpolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3-2 TCPthroughput vs. loss(Padhye’98) . . . . . . . . . . . . . . . . . . . . . . . . 32
4-1 TheRONoverlayapproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4-2 TheRONsystemarchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4-3 TheRONpacketheader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4-4 RONforwarding controlanddatapath . . . . . . . . . . . . . . . . . . . . . . . . 41
4-5 Implementation oftheIPencapsulating RONclient . . . . . . . . . . . . . . . . . 43
4-6 Theactiveprober . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4-7 TheRONperformance database. . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5-1 The13-nodeRONdeployment, geographically . . . . . . . . . . . . . . . . . . . 55
5-2 12RONnodes, bynetworklocation . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-3 Loss&RTTsamplingmethod . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5-4 Packetlossratescatterplot comparison . . . . . . . . . . . . . . . . . . . . . . . . 59
5-5 Packetlossratescatterplot comparison, dataset2 . . . . . . . . . . . . . . . . . . 60
5-6 Detailedexamination ofanoutage . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5-7 CDFof30-minute lossrateimprovementwithRON . . . . . . . . . . . . . . . . . 62
5-8 CDFof30-minute lossrateimprovementwithRON,dataset2 . . . . . . . . . . . 63
5-9 Thelimitationsofbidirectional lossprobes . . . . . . . . . . . . . . . . . . . . . . 65
5-10 5minuteaveragelatencyscatterplot, dataset1 . . . . . . . . . . . . . . . . . . . . 66
7
5-11 5minuteaveragelatencyscatterplot, dataset2 . . . . . . . . . . . . . . . . . . . . 67
5-12 Individual linklatencyCDFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5-13 Individual pathlatencies overtime . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5-14 Individual linklossratesovertime . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5-15 RONthroughput comparison, dataset1 . . . . . . . . . . . . . . . . . . . . . . . . 72
5-16 RONthroughput comparison, dataset2 . . . . . . . . . . . . . . . . . . . . . . . . 73
5-17 TCPtraceofafloodingattackmitigatedbyRON . . . . . . . . . . . . . . . . . . 74
5-18 Testbedconfiguration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5-19 RONencapsulation latencyoverhead . . . . . . . . . . . . . . . . . . . . . . . . . 75
8
List of Tables
3.1 Internet linkfailurerecoverytimes . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.1 Performance databasesummarization methods . . . . . . . . . . . . . . . . . . . . 52
5.1 Listofdeployed RONhosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.2 Ronoutageimprovements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.3 RONlossratechanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.4 RONlossratechangesindataset2 . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.5 Casesinwhichthelatencyandlossoptimizedpathsdiffer. . . . . . . . . . . . . . 71
5.6 TCPthroughput overheadwithRON . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.7 RONrouteduration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
9
Thesearejustafewofthetreasuresforthestockpot—thatmagicsourcefromwhichcomesthe
tellingcharacterofthecuisine.
-MarionRombauerBecker,TheJoyofCooking
ArtisI,scienceiswe.
-ClaudeBernard
Acknowledgments
Emerging frommyfirsttwoyearsatMITwithmysanity, andacompleted thesis, isnotsomething
Icould have completed alone. I am deeply indebted to many people for their support and patience
while I paced, grumbled, bounced, went climbing and running, and even occasionally got work
done. Thelistbelowdoesn’t standachanceofbeingcomprehensive, butI’llgiveitatry.
Imustfirstthankmyadvisor,HariBalakrishnan,forhisexpertguidanceandsupport. Hisvision,
excitement,andintelligence areaninspiration andanexampletowardwhichIstrive. Workingwith
Hari, and in his research group, has helped me grow academically in ways Ihadn’t even realized I
neededwhenIstarted—and asabonus,it’sbeenterriblyfun.
FransKaashoek andRobert Morrisgenerously letmedrawideas, insight, andexperience from
themandthePDOSgroup,andtheyprovidedconsiderable, appreciated mentorshipwhileIworked
on this research. I thank Hari, Frans, and Robert for building the bridges between their research
groupsthatallowedmetolearnfromtheexperienceofboth. DorothyCurtis,StephenGarland,and
JohnGuttagprovidedmuchwelcomefeedback andadvice,bothtechnical andotherwise.
My officemates, Alex Snoeren and Allen Miu, have revealed great depths of patience while I
empirically examined novel mechanisms for debugging and optimizing programs—pacing, bounc-
ing large yellow chair-balls, and having loud conversations with said programs. More seriously,
they’ve provided friendship, conversation, andfeedback about ahuge numberoftopics, alongwith
afunandstimulatingenvironmentinwhichtowork. I’mindebtedtoAlexforshowingmetheropes
atMITwhenIarrivedlookingconfused, andforworkingwithmeonseveralprojects.
I would like to thank the members of the Networks and Mobile Systems (NMS) and Parallel
and Distributed Operating Systems (PDOS)research groups. From architecture to art, and proxies
to philosophy, there was always someone with a clue to talk to, or something fun with which to
distract ourselves. Allen, Alex, Benjie, Bodhi, Bryan, Chuck, Dan, D.M., Deepak, Doug, Eddie,
Emma,Emmett,Frank,Godfrey,Greg,JJ,Jaeyeon, Jinyang, Kevin,Magda,Nick–thankyouall.
10