Table Of ContentLARGE-SCALE CONVEX OPTIMIZATION
Starting from where a first course in convex optimization leaves off,
thistextpresentsaunifiedanalysisoffirst-orderoptimizationmethods–
includingparallel-distributedalgorithms–throughtheabstractionofmon-
otoneoperators.Withtheincreasedcomputationalpowerandavailability
ofbigdataoverthepastdecade,applieddisciplineshavedemandedthat
larger and larger optimization problems be solved. This text covers the
first-order convex optimization methods that are uniquely effective at
solving these large-scale optimization problems. Readers will have the
opportunitytoconstructandanalyzemanywell-knownclassicalandmod-
ern algorithms using monotone operators, and walk away with a solid
understandingofthediverseoptimizationalgorithms.Graduatestudents
andresearchersinmathematicaloptimization,operationsresearch,elec-
trical engineering, statistics, and computer science will appreciate this
conciseintroductiontothetheoryofconvexoptimizationalgorithms.
Ernest K. Ryu is Assistant Professor of Mathematical Sciences at Seoul
NationalUniversity.HepreviouslyservedasAssistantAdjunctProfessor
withtheDepartmentofMathematicsattheUniversityofCalifornia,Los
Angeles from 2016 to 2019, before joining Seoul National University in
2020.HereceivedaBSwithdistinctioninphysicsandelectricalengineer-
ingfromtheCaliforniaInstituteofTechnologyin2010;andthenanMSin
statisticsandaPhD–withtheGeneGolubBestThesisAward–incompu-
tationalmathematicsatStanfordUniversityin2016.Hiscurrentresearch
focusesonmathematicaloptimizationandmachinelearning.
Wotao Yin is Director of the Decision Intelligence Lab with Alibaba
Group(US),DAMOAcademy,andaformerProfessorofMathematicsat
theUniversityofCalifornia,LosAngeles.HereceivedhisPhDinopera-
tionsresearchfromColumbiaUniversityin2006.Hisnumerousaccolades
include an NSF CAREER Award in 2008, an Alfred P. Sloan Research
Fellowship in 2009, a Morningside Gold Medal in 2016, and a DAMO
AwardandanEgonBalasPrizein2021.Heinventedfastalgorithmsfor
sparse optimization, image processing, and large-scale distributed opti-
mization problems, and is among the top 1 percent of cited researchers
byClarivateAnalytics.Hisresearchinterestsincludecomputationalopti-
mizationanditsapplicationsinsignalprocessing,machinelearning,and
otherdatascienceproblems.
“Ryu and Yin’s Large-Scale Convex Optimization does a great job of covering a field
withalonghistoryandmuchcurrentinterest.Thebookdescribesdozensofalgorithms,
fromclassiconesdevelopedinthe1970stosomeveryrecentones,inunifiedandconsis-
tentnotation,allorganizedaroundthebasicconceptandunifyingthemeofamonotone
operator.Istronglyrecommendittoanymathematician,researcher,orengineerwho
uses,orhasaninterestin,convexoptimization.”
– StephenBoyd,StanfordUniversity
“Thisisanabsolutemust-readresearchmonographforsignalprocessing,communica-
tions,andnetworkingengineers,aswellasresearcherswhowishtochoose,design,and
analyze splitting-based convex optimization methods best suited for their perplexed
andchallengingengineeringtasks.”
– GeorgiosB.Giannakis,UniversityofMinnesota
“Thisisaverytimelybook.Monotoneoperatortheoryisfundamentaltothedevelop-
mentofmodernalgorithmsforlarge-scaleconvexoptimization.RyuandYinprovide
optimizationstudentsandresearcherswithaself-containedintroductiontotheelegant
mathematicaltheoryofmonotoneoperators,andtaketheirreadersonatourofcutting-
edgeapplications,demonstratingthepowerandrangeoftheseessentialtools.”
– LievenVandenberghe,UniversityofCalifornia,LosAngeles
“First-ordermethodsarethemainstreamoptimizationalgorithmsintheeraofbigdata.
Thismonographprovidesauniqueperspectiveonvariousfirst-orderconvexoptimiza-
tionalgorithmsviathemonotoneoperatortheory,withwhichtheseeminglydifferent
andunrelatedalgorithmsareactuallydeeplyconnected,andmanyproofscanbesignif-
icantlysimplified.Thebookisabeautifulexampleofthepowerofabstraction.Those
whoareinterestedinconvexoptimizationtheoryshouldnotmissthisbook.”
– ZhouchenLin,PekingUniversity
“Thebookcoverstopicsfromthebasicsofoptimizationtomoderntechniquessuchas
operator splitting, parallel and distributed optimization, and stochastic algorithms. It
isthenaturalnextstepafterBoydandVandenberghe’sConvexOptimizationforstu-
dentsstudyingoptimizationandmachinelearning.Theauthorsareexpertsinthiskind
ofoptimization.Someofmygraduatestudentstookthecoursebasedonthisbookwhen
WotaoYinwasatUCLA.Theylikedthecourseandfoundthematerialsveryusefulin
theirresearch.”
– StanleyOsher,UniversityofCalifornia,LosAngeles
LARGE-SCALE CONVEX
OPTIMIZATION
Algorithms & Analyses via Monotone
Operators
ERNEST K. RYU
SeoulNationalUniversity
WOTAO YIN
UniversityofCalifornia,LosAngeles
andDAMOAcademy,AlibabaGroup
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India
103PenangRoad,#05-06/07,VisioncrestCommercial,Singapore238467
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781009160858
DOI:10.1017/9781009160865
©ErnestK.RyuandWotaoYin2023
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2023
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
ISBN978-1-009-16085-8Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof
URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
Dedicatedtoourwives
BoraandRui
Contents
Preface xi
Acknowledgments xiv
1 IntroductionandPreliminaries 1
1.1 First-OrderMethodsintheModernEra 1
1.2 LimitationsofMonotoneOperatorTheory 2
1.3 Preliminaries 2
BibliographicalNotes 17
Exercises 18
PARTONE:MONOTONEOPERATORMETHODS
2 MonotoneOperatorsandBaseSplittingSchemes 23
2.1 Set-ValuedOperators 23
2.2 MonotoneOperators 25
2.3 NonexpansiveandAveragedOperators 30
2.4 Fixed-PointIteration 31
2.5 Resolvents 37
2.6 ProximalPointMethod 42
2.7 OperatorSplitting 43
2.8 VariableMetricMethods 50
2.9 CommonlyUsedFormulas 52
BibliographicalNotes 53
Exercises 56
3 Primal-DualSplittingMethods 66
3.1 InfimalPostcompositionTechnique 66
3.2 DualizationTechnique 69
3.3 VariableMetricTechnique 71
3.4 GaussianEliminationTechnique 74
3.5 LinearizationTechnique 78
3.6 Discussion 84
BibliographicalNotes 84
Exercises 86
vii
viii Contents
4 ParallelComputing 94
4.1 ComputationalComplexityviaFlopCount 94
4.2 ParallelComputing 96
BibliographicalNotes 103
Exercises 103
5 RandomizedCoordinateUpdateMethods 105
5.1 RandomizedCoordinateFixed-PointIteration 105
5.2 CoordinateandExtendedCoordinate-FriendlyOperators 109
5.3 Methods 112
5.4 Discussion 116
BibliographicalNotes 117
Exercises 118
6 AsynchronousCoordinateUpdateMethods 120
6.1 AsynchronousFixed-PointIteration 122
6.2 ExtendedCoordinate-FriendlyOperatorsandExclusiveMemoryAccess 130
6.3 Server-WorkerFramework 132
6.4 Methods 134
6.5 ExclusiveMemoryAccess 137
BibliographicalNotes 141
Exercises 143
PARTTWO:ADDITIONALTOPICS
7 StochasticOptimization 147
7.1 StochasticForward-BackwardMethod 148
7.2 Methods 155
BibliographicalNotes 156
Exercises 157
8 ADMM-TypeMethods 160
8.1 Function-LinearizedProximalADMM 160
8.2 DerivedADMM-TypeMethods 167
8.3 BregmanMethods 177
8.4 Conclusion 179
BibliographicalNotes 179
Exercises 184
9 DualityinSplittingMethods 190
9.1 FenchelDuality 190
9.2 Attouch–ThéraDuality 191
9.3 DualityinSplittingMethods 192
BibliographicalNotes 194
Exercises 195
10 MaximalityandMonotoneOperatorTheory 197
10.1 MaximalityofSubdifferential 197
10.2 FitzpatrickFunction 198
10.3 MaximalityandExtensionTheorems 201
BibliographicalNotes 203
Exercises 204
Contents ix
11 DistributedandDecentralizedOptimization 207
11.1 DistributedOptimizationwithCentralizedConsensus 207
11.2 DecentralizedOptimizationwithGraphConsensus 214
11.3 DecentralizedOptimizationwithMixingMatrices 217
BibliographicalNotes 223
Exercises 225
12 Acceleration 233
12.1 AcceleratedGradientMethod 233
12.2 AcceleratedProximalPointandOptimizedHalpernMethod 236
12.3 WhenDoesanAccelerationAccelerate? 237
BibliographicalNotes 238
Exercises 239
13 ScaledRelativeGraphs 242
13.1 BasicDefinitions 242
13.2 ScaledRelativeGraphs 245
13.3 OperatorandSRGTransformations 252
13.4 AveragednessCoefficients 264
BibliographicalNotes 267
Exercises 268
AppendixA Miscellaneous 271
References 273
Index 299
