Understanding Distributed Systems: What every developer should know about large distributed applications

236 Pages·2021·10.301 MB·English

by  Roberto Vitillo

Roberto Vitillo
UNDERSTANDING
DISTRIBUTED
SYSTEMS

WHAT EVERY DEVELOPER SHOULD KNOW ABOUT
LARGE DISTRIBUTED APPLICATIONS
Understanding Distributed Systems
Version 1.0.4
Roberto Vitillo
February 2021
Contents
Copyright 6
Abouttheauthor 7
Acknowledgements 8
Preface 9
0.1 Whoshouldreadthisbook . . . . . . . . . . . . . . 10
1 Introduction 11
1.1 Communication . . . . . . . . . . . . . . . . . . . . 12
1.2 Coordination . . . . . . . . . . . . . . . . . . . . . 13
1.3 Scalability . . . . . . . . . . . . . . . . . . . . . . . 13
1.4 Resiliency . . . . . . . . . . . . . . . . . . . . . . . 15
1.5 Operations. . . . . . . . . . . . . . . . . . . . . . . 16
1.6 Anatomyofadistributedsystem . . . . . . . . . . 17
I Communication 20
2 Reliablelinks 23
2.1 Reliability . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Connectionlifecycle . . . . . . . . . . . . . . . . . 24
2.3 Flowcontrol . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Congestioncontrol . . . . . . . . . . . . . . . . . . 27
2.5 Customprotocols . . . . . . . . . . . . . . . . . . . 28
3 Securelinks 30
3.1 Encryption. . . . . . . . . . . . . . . . . . . . . . . 30
3.2 Authentication . . . . . . . . . . . . . . . . . . . . 31
3.3 Integrity . . . . . . . . . . . . . . . . . . . . . . . . 33
CONTENTS 2
3.4 Handshake . . . . . . . . . . . . . . . . . . . . . . 34
4 Discovery 35
5 APIs 39
5.1 HTTP . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Resources . . . . . . . . . . . . . . . . . . . . . . . 43
5.3 Requestmethods . . . . . . . . . . . . . . . . . . . 45
5.4 Responsestatuscodes . . . . . . . . . . . . . . . . 46
5.5 OpenAPI . . . . . . . . . . . . . . . . . . . . . . . 48
5.6 Evolution . . . . . . . . . . . . . . . . . . . . . . . 49
II Coordination 51
6 Systemmodels 54
7 Failuredetection 57
8 Time 59
8.1 Physicalclocks . . . . . . . . . . . . . . . . . . . . 60
8.2 Logicalclocks . . . . . . . . . . . . . . . . . . . . . 61
8.3 Vectorclocks . . . . . . . . . . . . . . . . . . . . . 63
9 Leaderelection 65
9.1 Raftleaderelection . . . . . . . . . . . . . . . . . . 65
9.2 Practicalconsiderations. . . . . . . . . . . . . . . . 67
10 Replication 71
10.1 Statemachinereplication . . . . . . . . . . . . . . . 72
10.2 Consensus . . . . . . . . . . . . . . . . . . . . . . . 75
10.3 Consistencymodels . . . . . . . . . . . . . . . . . . 75
10.3.1 Strongconsistency . . . . . . . . . . . . . . 77
10.3.2 Sequentialconsistency . . . . . . . . . . . . 78
10.3.3 Eventualconsistency . . . . . . . . . . . . . 80
10.3.4 CAPtheorem . . . . . . . . . . . . . . . . . 80
10.4 Practicalconsiderations. . . . . . . . . . . . . . . . 81
11 Transactions 83
11.1 ACID . . . . . . . . . . . . . . . . . . . . . . . . . 83
11.2 Isolation . . . . . . . . . . . . . . . . . . . . . . . . 84
11.2.1 Concurrencycontrol . . . . . . . . . . . . . 87
11.3 Atomicity . . . . . . . . . . . . . . . . . . . . . . . 88
11.3.1 Two-phasecommit . . . . . . . . . . . . . . 89
CONTENTS 3
11.4 Asynchronoustransactions . . . . . . . . . . . . . . 91
11.4.1 Log-basedtransactions . . . . . . . . . . . . 91
11.4.2 Sagas . . . . . . . . . . . . . . . . . . . . . . 94
11.4.3 Isolation . . . . . . . . . . . . . . . . . . . . 97
III Scalability 99
12 Functionaldecomposition 103
12.1 Microservices . . . . . . . . . . . . . . . . . . . . . 103
12.1.1 Benefits . . . . . . . . . . . . . . . . . . . . 105
12.1.2 Costs . . . . . . . . . . . . . . . . . . . . . . 106
12.1.3 Practicalconsiderations . . . . . . . . . . . 108
12.2 APIgateway. . . . . . . . . . . . . . . . . . . . . . 110
12.2.1 Routing . . . . . . . . . . . . . . . . . . . . 110
12.2.2 Composition . . . . . . . . . . . . . . . . . 111
12.2.3 Translation . . . . . . . . . . . . . . . . . . 112
12.2.4 Cross-cuttingconcerns . . . . . . . . . . . . 112
12.2.5 Caveats . . . . . . . . . . . . . . . . . . . . 115
12.3 CQRS . . . . . . . . . . . . . . . . . . . . . . . . . 117
12.4 Messaging . . . . . . . . . . . . . . . . . . . . . . . 118
12.4.1 Guarantees . . . . . . . . . . . . . . . . . . 121
12.4.2 Exactly-onceprocessing . . . . . . . . . . . 123
12.4.3 Failures . . . . . . . . . . . . . . . . . . . . 124
12.4.4 Backlogs . . . . . . . . . . . . . . . . . . . . 124
12.4.5 Faultisolation . . . . . . . . . . . . . . . . . 125
12.4.6 Referenceplusblob . . . . . . . . . . . . . . 126
13 Partitioning 128
13.1 Shardingstrategies . . . . . . . . . . . . . . . . . . 128
13.1.1 Rangepartitioning . . . . . . . . . . . . . . 129
13.1.2 Hashpartitioning . . . . . . . . . . . . . . . 129
13.2 Rebalancing . . . . . . . . . . . . . . . . . . . . . . 133
13.2.1 Staticpartitioning . . . . . . . . . . . . . . . 133
13.2.2 Dynamicpartitioning . . . . . . . . . . . . . 133
13.2.3 Practicalconsiderations . . . . . . . . . . . 134
14 Duplication 135
14.1 Networkloadbalancing . . . . . . . . . . . . . . . 135
14.1.1 DNSloadbalancing. . . . . . . . . . . . . . 138
14.1.2 Transportlayerloadbalancing . . . . . . . . 138
14.1.3 Applicationlayerloadbalancing . . . . . . 141
14.1.4 Geoloadbalancing . . . . . . . . . . . . . . 143
CONTENTS 4
14.2 Replication . . . . . . . . . . . . . . . . . . . . . . 145
14.2.1 Singleleaderreplication . . . . . . . . . . . 145
14.2.2 Multi-leaderreplication . . . . . . . . . . . 148
14.2.3 Leaderlessreplication. . . . . . . . . . . . . 150
14.3 Caching . . . . . . . . . . . . . . . . . . . . . . . . 152
14.3.1 Policies . . . . . . . . . . . . . . . . . . . . 152
14.3.2 In-processcache . . . . . . . . . . . . . . . . 153
14.3.3 Out-of-processcache . . . . . . . . . . . . . 154
IV Resiliency 157
15 Commonfailurecauses 160
15.1 Singlepointoffailure . . . . . . . . . . . . . . . . . 160
15.2 Unreliablenetwork . . . . . . . . . . . . . . . . . . 161
15.3 Slowprocesses . . . . . . . . . . . . . . . . . . . . 162
15.4 Unexpectedload . . . . . . . . . . . . . . . . . . . 163
15.5 Cascadingfailures . . . . . . . . . . . . . . . . . . 164
15.6 Riskmanagement . . . . . . . . . . . . . . . . . . . 165
16 Downstreamresiliency 167
16.1 Timeout . . . . . . . . . . . . . . . . . . . . . . . . 167
16.2 Retry . . . . . . . . . . . . . . . . . . . . . . . . . . 170
16.2.1 Exponentialbackoff . . . . . . . . . . . . . . 171
16.2.2 Retryamplification . . . . . . . . . . . . . . 172
16.3 Circuitbreaker . . . . . . . . . . . . . . . . . . . . 173
16.3.1 Statemachine . . . . . . . . . . . . . . . . . 174
17 Upstreamresiliency 176
17.1 Loadshedding . . . . . . . . . . . . . . . . . . . . 176
17.2 Loadleveling . . . . . . . . . . . . . . . . . . . . . 177
17.3 Rate-limiting . . . . . . . . . . . . . . . . . . . . . 178
17.3.1 Single-processimplementation . . . . . . . 180
17.3.2 Distributedimplementation . . . . . . . . . 183
17.4 Bulkhead . . . . . . . . . . . . . . . . . . . . . . . 184
17.5 Healthendpoint . . . . . . . . . . . . . . . . . . . . 187
17.5.1 Healthchecks . . . . . . . . . . . . . . . . . 187
17.6 Watchdog . . . . . . . . . . . . . . . . . . . . . . . 188
V Testing and operations 190
18 Testing 193
CONTENTS 5
18.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . 194
18.2 Size . . . . . . . . . . . . . . . . . . . . . . . . . . 196
18.3 Practicalconsiderations. . . . . . . . . . . . . . . . 198
19 Continuousdeliveryanddeployment 200
19.1 Reviewandbuild . . . . . . . . . . . . . . . . . . . 201
19.2 Pre-production . . . . . . . . . . . . . . . . . . . . 203
19.3 Production. . . . . . . . . . . . . . . . . . . . . . . 203
19.4 Rollbacks . . . . . . . . . . . . . . . . . . . . . . . 204
20 Monitoring 207
20.1 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 208
20.2 Service-levelindicators . . . . . . . . . . . . . . . . 211
20.3 Service-levelobjectives . . . . . . . . . . . . . . . . 214
20.4 Alerts . . . . . . . . . . . . . . . . . . . . . . . . . 217
20.5 Dashboards . . . . . . . . . . . . . . . . . . . . . . 219
20.5.1 Bestpractices . . . . . . . . . . . . . . . . . 221
20.6 On-call . . . . . . . . . . . . . . . . . . . . . . . . . 222
21 Observability 225
21.1 Logs . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21.2 Traces . . . . . . . . . . . . . . . . . . . . . . . . . 229
21.3 Puttingitalltogether . . . . . . . . . . . . . . . . . 231
22 Finalwords 233
Copyright
UnderstandingDistributedSystemsbyRobertoVitillo
Copyright©RobertoVitillo. Allrightsreserved.
Thebook’sdiagramshavebeencreatedwithExcalidraw.
Whiletheauthorhasusedgoodfaitheffortstoensurethatthein-
formation and instructions in this work are accurate, the author
disclaimsallresponsibilityforerrorsoromissions,includingwith-
outlimitationresponsibilityfordamagesresultingfromtheuseof
or reliance on this work. The use of the information and instruc-
tionscontainedinthisworkisatyourownrisk. Ifanycodesam-
plesorothertechnologythisworkcontainsordescribesissubject
toopensourcelicensesortheintellectualpropertyrightsofothers,
it is your responsibility to ensure that your use thereof complies
withsuchlicensesand/orrights.
About the author
Authorsgenerallywritethispageinthethirdpersonasifsomeone
elseiswritingaboutthem. Iliketodothingsalittlebitdifferently.
Ihaveover10yearsofexperienceinthetechindustryasasoftware
engineer,technicallead,andmanager.
In 2017, I joined Microsoft to work on an internal SaaS data plat-
form. Sincethen,IhavehelpedlaunchtwopublicSaaSproducts,
ProductInsightsandPlayfab. ThedatapipelineIamresponsible
forisoneofthelargestintheworld. Itprocessesmillionsofevents
persecondfrombillionsofdevicesworldwide.
Before that, I worked at Mozilla, where I set the direction of the
dataplatformfromitsveryearlydaysandbuiltalargepartofit,
includingtheteam.
After getting my master’s degree in computer science, I worked
onscientificcomputingapplicationsattheBerkeleyLab. Thesoft-
wareIcontributedisusedtothisdaybytheATLASexperimentat
theLargeHadronCollider.
Acknowledgements
Writing a book is an incredibly challenging but rewarding expe-
rience. I wanted to share what I have learned about distributed
systemsforaverylongtime.
I appreciate the colleagues who inspired and believed in me.
Thanks to Chiara Roda, Andrea Dotti, Paolo Calafiura, Vladan
Djeric,MarkReid,PawelChodarcewicz,andNunoCerqueira.
Doug Warren, Vamis Xhagjika, Gaurav Narula, Alessio Placitelli,
KofiSarfo,StefaniaVitilloandAlbertoSottilewereallkindenough
toprovideinvaluablefeedback. Withoutthem,thebookwouldn’t
bewhatitistoday.
Finally,andaboveall,thankstomyfamily: RachellandLeonardo.
Youalwaysbelievedinme. Thatmadeallthedifference.