Decoupling Congestion Control and Bandwidth Allocation Policy With Application to High ...

129 Pages·2011·0.78 MB·English

Decoupling Congestion Control and Bandwidth Allocation Policy With
Application to High Bandwidth-Delay Product Networks
by
Dina Katabi
SubmittedtotheDepartmentofElectricalEngineeringandComputerScience
inpartialfulfillmentoftherequirementsforthedegreeof
DoctorofPhilosophyinComputerScience andEngineering
atthe
MASSACHUSETTSINSTITUTEOF TECHNOLOGY
March2003
c MassachusettsInstituteofTechnology2003. Allrightsreserved.
(cid:0)
Author...........................................................................
DepartmentofElectricalEngineeringandComputerScience
March7,2003
Certified by.......................................................................
DavidClark
SeniorResearch Scientist
ThesisSupervisor
Acceptedby......................................................................
ArthurC.Smith
Chairman,DepartmentCommitteeonGraduateStudents
DecouplingCongestionControl andBandwidth AllocationPolicyWithApplication
toHighBandwidth-Delay Product Networks
by
DinaKatabi
SubmittedtotheDepartmentofElectricalEngineeringandComputerScience
onMarch7,2003,inpartialfulfillmentofthe
requirementsforthedegreeof
DoctorofPhilosophyinComputerScienceandEngineering
Abstract
Inthisdissertation,weproposeanewarchitectureforInternetcongestioncontrolthatdecouplesthe
control of congestion from the bandwidth allocation policy. Weshow that the newprotocol, called
XCP,enables very large per-flow throughput(e.g., more than 1Gb/s), which is unachievable using
currentcongestion control. Additionally, weshowvia extensivesimulationsthat XCPsignificantly
improves the overall performance, reducing drop rate by threeorders of magnitude, increasing uti-
lization, decreasing queuing delay, and attaining fairness in a few RTTs. Using tools from control
theory, we model XCP and demonstrate that, in steady state, it is stable for any capacity, delay,
and number of sources. XCP does not maintain any per-flow state in routers and requires only a
fewCPUcycles perpacketmaking itimplementable inhigh-speedrouters. Itsflexible architecture
facilitatesthedesignandimplementationofqualityofservice,suchasguaranteedandproportional
bandwidthallocations. Finally,XCPisamenabletogradualdeployment.
ThesisSupervisor: David Clark
Title: SeniorResearchScientist
2
Acknowledgments
IhavebeenfortunatetohaveDaveClarkasmyadviser. Davesimultaneouslyprovidedthefreedom
to work on what I wanted and the guidance that enabled me to succeed in my work. My PhD
committee: Charles Rohrs, John Guttag, and Hari Balakrishnan provided valuable counsel on both
researchandwriting. MarkHandleycollaboratedonpartofthisworkandcontributedmuchinsight.
TimShepardandMangeshKasbekarreadthisdissertationandprovidedimportantcomments. John
Wroclawski, Karen Sollins, Barbara Liskov, Frans Kaashoek have supported me at various stages
of my graduate career. I am very grateful to all my friends on the 5th floor of LCS who made my
graduateyearsatMITexcitingandpleasurable.
3
Contents
1 Overview 13
1.1 DecouplingCongestionControlfromtheBandwidthAllocationPolicy . . . . . . . 14
1.2 CongestionControlinLargeBandwidth-Delay ProductNetworks . . . . . . . . . 15
1.3 Contributions&Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2 Background&RelatedWork 18
2.1 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 InternetCongestionControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.1 End-SystemCongestionControlProtocols . . . . . . . . . . . . . . . . . 20
2.2.2 RouterCongestionControllers . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Expressiveness&State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 ECN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 PacketPair&FairQueuing . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.3 CoreStatelessFairQueuing(CSFQ) . . . . . . . . . . . . . . . . . . . . . 26
2.3.4 ABRCongestionControl . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 TheChallengesforInternetCongestionControl . . . . . . . . . . . . . . . . . . . 28
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3 TheProblem 30
3.1 TCP’sPerformanceinHighBandwidth-DelayProductNetworks . . . . . . . . . . 30
3.2 SimulationResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4
4 XCP:eXplicitControlProtocol 37
4.1 DesignRationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 TheXCPProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.4 TheCongestionHeader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.5 TheXCPSender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.6 TheXCPReceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.7 TheXCPRouter: TheControlLaws . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.7.1 TheEfficiencyController(EC) . . . . . . . . . . . . . . . . . . . . . . . . 43
4.7.2 TheFairnessController(FC) . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.7.3 NotesontheEfficiencyandFairnessControllers . . . . . . . . . . . . . . 46
4.8 StabilityAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5 Performance 51
5.1 SimulationSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2 ComparisonwithTCPandAQMSchemes . . . . . . . . . . . . . . . . . . . . . . 53
5.2.1 ImpactofHighCapacity . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2.2 ImpactofFeedbackDelay . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.2.3 ImpactofNumberofFlows . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.2.4 ImpactofShortWeb-LikeTraffic . . . . . . . . . . . . . . . . . . . . . . 57
5.2.5 Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.2.6 AMoreComplexTopology . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.3 TheDynamicsofXCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.3.1 ConvergenceDynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.3.2 RobustnesstoSuddenIncreaseorDecreaseinTrafficDemands . . . . . . 62
5.4 Robustness toLargeRTTVariance . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.5 ImpactofError-BasedDrops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.6 Large ScaleSimulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6 QualityofService(QoS) 70
6.1 QoSModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5
6.2 ProvidingQoSWithintheXCPFramework . . . . . . . . . . . . . . . . . . . . . 72
6.3 RelativeBandwidthAllocationinXCP . . . . . . . . . . . . . . . . . . . . . . . . 73
6.3.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.4 ProvidingBandwidthGuaranteesinXCP . . . . . . . . . . . . . . . . . . . . . . 77
6.4.1 ServiceFramework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.4.2 ModifyingtheCongestionHeader . . . . . . . . . . . . . . . . . . . . . . 78
6.4.3 BehavioroftheSender . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.4.4 ControlPlane: DistributedAdmissionControlWithNoPer-FlowState . . 79
6.4.5 DataPlane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.4.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.5 ConcludingRemarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7 Non-CompliantFlows 90
7.1 FlowMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.2 NetworkswithNoFlowProtection . . . . . . . . . . . . . . . . . . . . . . . . . . 91
7.3 Senders’Misconduct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
7.3.1 TheT-liar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.3.2 TheD-liar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.3.3 ResponsiveAbusers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.3.4 UnresponsiveAbuser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.4 Receivers’Misconduct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.5 ConcludingRemarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
8 GradualDeployment 105
8.1 XCP-basedCoreStatelessFairQueuing . . . . . . . . . . . . . . . . . . . . . . . 105
8.2 ATCP-FriendlyXCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
8.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
9 Conclusion&FutureWork 110
9.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
9.1.1 GoodPerformanceinNetworkswithLargeBandwidth-Delay Product . . . 110
9.1.2 XCPOutperformsOtherProtocolsWithNoPer-FlowState . . . . . . . . . 111
9.1.3 Integrating ControlTheorywithInternetProtocols . . . . . . . . . . . . . 111
6
9.1.4 NewConceptsforInternetBandwidthManagement . . . . . . . . . . . . . 112
9.1.5 ASimpleApproachtoQualityofService . . . . . . . . . . . . . . . . . . 113
9.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
9.2.1 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
9.2.2 SecurityWithoutPer-FlowState . . . . . . . . . . . . . . . . . . . . . . . 114
9.3 Outstanding Problems&FutureWork . . . . . . . . . . . . . . . . . . . . . . . . 115
9.3.1 Multi-AccessLinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
9.3.2 IncreasingRobustnesstoRTTVariance . . . . . . . . . . . . . . . . . . . 116
9.3.3 SpecifyingtheBitFormatoftheCongestionHeader . . . . . . . . . . . . 116
9.4 FinalRemarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
A XCPImplementation 118
B AnalysisofXCPStability 121
B.1 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
B.2 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7
List of Figures
2-1 TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-1 Asinglebottlenecktopology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3-2 TCP’saverageutilizationdecreaseswithincreasedbottleneckbandwidth. . . . . . 34
3-3 TCP’saverageutilizationdecreaseswithincreaseddelay. . . . . . . . . . . . . . . 35
4-1 Congestionheader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-1 Asinglebottlenecktopology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5-2 Aparkinglottopology. Arrowsrepresenttrafficdirections. . . . . . . . . . . . . . 53
5-3 XCP significantly outperforms TCP in high bandwidth environments. The graphs
comparetheefficiencyofXCPwiththatofTCPoverRED,CSFQ,REM,andAVQ,
asafunctionofcapacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5-4 XCP significantly outperforms TCP in high delay environments. The graphs com-
parebottleneck utilization,averagequeue,andnumberofdropsasroundtripdelay
increases when flows are XCPs and when they are TCPs over RED, CSFQ, REM,
andAVQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-5 XCP is efficient with any number of flows. The graphs compare the efficiency of
XCPandTCPwithvariousqueuingschemesasafunctionofthenumberofflows. 56
5-6 XCP is robust and efficient in environments with arrivals and departures of short
web-like flows. The graphs compare the efficiency of XCP to that of TCP over
variousqueuingschemesasafunctionofthearrivalrateofweb-likeflows. . . . . 57
5-7 XCP is fair to both equal and different RTT flows. The graphs compare XCP’s
Fairness to that of TCP over RED, CSFQ, REM, and AVQ. Graph (b) also shows
XCP’srobustnesstoenvironmentswithdifferentRTTs. . . . . . . . . . . . . . . 59
8
5-8 Simulation with multiple congested queues. Utilization, average queue size, and
number of drops at nine consecutive links (topology in Figure 5-2). Link 5 has the
lowestcapacityalong thepath. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5-9 XCP’ssmoothconvergencetohighfairness,goodutilization,andsmallqueuesize.
FiveXCPflowsshare a45 Mb/sbottleneck. Theystarttheirtransfersattimes 0,2,
4,6,and8seconds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5-10 XCPismorerobustagainstsuddenincreaseordecreaseintrafficdemandsthanTCP.
Ten FTP flows share a bottleneck. At time seconds, we start 100 additional
(cid:0) (cid:0) (cid:2)
flows. At seconds, these 100 flows are suddenly stopped and the original 10
(cid:0) (cid:0) (cid:3)
flowsarelefttostabilizeagain. . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5-11 The RTT distribution at an OC3 access link at the Supercomputer center in San
Diego,(areproductionofFigure13-a-topin[42]). . . . . . . . . . . . . . . . . . 63
5-12 TheperformanceofXCPandTCP+RED+ECNwhentheRTTisdistributedaccord-
ingtoFigure5-11(valuesareloggedeverysecond). . . . . . . . . . . . . . . . . 64
5-13 The performance of XCP and TCP+RED when the RTT is uniformly distributed
over[5ms, ],where isthevalueonthex-axis. . . . . . . . . . . . . . . . . . . 65
(cid:2) (cid:2)
5-14 The throughput of a single XCP flow as a function of packet error rate. Capacity
is 4 Gb/s, RTT = 80 ms. For a reference, the figure also shows the steady state
throughputofaTCPflowasafunctionofthepacketerrorrate. . . . . . . . . . . 66
5-15 Alargesimulationwith50nodes,153links,and1000flows. . . . . . . . . . . . . 67
5-16 AcomparisonbetweentheefficiencyofXCPandTCPoverRED. . . . . . . . . . 67
6-1 ComparisonbetweenXCP’srelativebandwidthallocationandMulTCP[24],CSFQ
[75], and Weighted Fair Queuing (WFQ) [25]. Three flows each transferring a 10
MBfileoverashared10 Mb/sbottleneck. Flow1’s weight is5,Flow2’s weight is
10,andFlow3’sweightis15. Throughputisaveragedover200ms(5RTTs). . . . 75
6-2 The range of relative bandwidth allocation provided with XCP, MulTCP, CSFQ,
and WFQ. The figure shows the ratio of the throughputs of two flows sharing a
bottleneck as a function of the ratio of their weights. Each point is the average of
10runs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
9
6-3 R tracksthereservationmadebytheguaranteedbandwidthflows. Theguaranteed
(cid:0)
bandwidthflowsarriveseparatedby5sec. Eachflowrequests15%ofthebottleneck
bandwidth. Therestofthebandwidthisfilledupbybest-effortflows. SinceG=0.8
capacity, we can accommodate at most 5 guaranteed bandwidth flows. At ,
(cid:0) (cid:0) (cid:4)(cid:5)
active guaranteed flows are stopped, which frees up the reserved bandwidth. The
cyclerepeatswithnewarrivalsofguaranteedflows. . . . . . . . . . . . . . . . . 83
6-4 Admissioncontrolwithon-offflows. Theguaranteedserviceflowsallarrivearound
and stop at . The router accepts them as long as . The
(cid:0) (cid:0) (cid:6)(cid:5)(cid:3) (cid:0) (cid:0) (cid:7)(cid:5)(cid:3) (cid:4)(cid:0) (cid:5) (cid:6)
figureshowsthatalthoughtheon-offflowsdonotusealloftheirreservedbandwidth
( ),theroutermaintainsthereservations. . . . . . . . . . . . . . . . . . . 84
(cid:4)(cid:2) (cid:5)(cid:4)(cid:0)
6-5 The service perceived by the guaranteed bandwidth flows. 6-5-a shows that the
flows obtain their reserved throughput. 6-5-b shows that the delay perceived by
guaranteedserviceflowsisalmostalways theRTT. . . . . . . . . . . . . . . . . . 85
6-6 The service perceived by best-effort traffic. A 10 Mb/s bottleneck is shared by 10
best-effort flows and 5 guaranteed service flows. The guaranteed service flows are
consuming75%ofthebandwidth. Therestisusedbythebest-efforttraffic. . . . . 86
6-7 Alargesimulationtopologywith153linksand50nodes. . . . . . . . . . . . . . 87
6-8 Guaranteed bandwidth flowsexperience negligible worst case queuing delays. The
figure shows the maximum G-queue size at each of the links in Figure 6-7, which
is the worst case queuing delay experienced by guaranteed bandwidth flows at that
link. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7-1 The impact of T-lying on fairness. The throughput of a single T-liar that is sharing
abottleneckwith19compliantflows. . . . . . . . . . . . . . . . . . . . . . . . . 95
7-2 The impact of T-lying on efficiency. The utilization and queue size of a bottleneck
traversedby20T-liars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7-3 The impact of D-lying on fairness.The throughput of a single D-liar that is sharing
abottleneckwith19compliantflows. . . . . . . . . . . . . . . . . . . . . . . . . 97
7-4 The impact of D-lying on efficiency. The utilization and queuesize ofabottleneck
sharedby20flows. ThegraphsareforthecaseswhenallflowsareD-liars,75%of
theflowsareD-liars,and50%oftheflowsareD-liars. . . . . . . . . . . . . . . . 98
10