Algorithms for Optimization

521 Pages·2019·12.64 MB·English

by  Mykel J. Kochenderfer

Algorithms for Optimization

Algorithms for Optimization
MykelJ.Kochenderfer
TimA.Wheeler
TheMITPress
Cambridge,Massachusetts
London,England
©2019MassachusettsInstituteofTechnology
Allrightsreserved.Permissionisgranted,freeofcharge,tousethecodesnippetsassociatedwiththisbook,
subjecttotheconditionthatthesourceofthecodeisacknowledged.Nootherpartsofthisbookmayberepro-
ducedinanyformbyanyelectronicormechanicalmeans(includingphotocopying,recording,orinformation
storageandretrieval)withoutpermissioninwritingfromthepublisher.
ThisbookwassetinTEXGyrePagellabytheauthorsinLATEX.
PrintedandboundintheUnitedStatesofAmerica.
LibraryofCongressCataloging-in-PublicationData
Names:Kochenderfer,MykelJ.,1980–author.|Wheeler,TimA.(TimAllan),author.
Title:Algorithmsforoptimization/MykelJ.KochenderferandTimA.Wheeler.
Description:Cambridge,MA:TheMITPress,[2019]|Includesbibliographicalreferencesandindex.
Identifiers:LCCN2018023091|ISBN9780262039420(hardcover:alk.paper)
Subjects:LCSH:Algorithms.|Algorithms—Problems,exercises,etc.
Classification:LCCQA9.58.K654252019|DDC518/.1—dc23
LCrecordavailableathttps://lccn.loc.gov/2018023091
10 9 8 7 6 5 4 3 2 1
Toourfamilies.

Contents
Preface xvii
Acknowledgments xix
1 Introduction 1
1.1 AHistory 2
1.2 OptimizationProcess 4
1.3 BasicOptimizationProblem 5
1.4 Constraints 6
1.5 CriticalPoints 7
1.6 ConditionsforLocalMinima 8
1.7 ContourPlots 11
1.8 Overview 11
1.9 Summary 17
1.10 Exercises 17
2 Derivatives and Gradients 19
2.1 Derivatives 19
viii contents
2.2 DerivativesinMultipleDimensions 21
2.3 NumericalDifferentiation 23
2.4 AutomaticDifferentiation 27
2.5 Summary 32
2.6 Exercises 33
3 Bracketing 35
3.1 Unimodality 35
3.2 FindinganInitialBracket 35
3.3 FibonacciSearch 37
3.4 GoldenSectionSearch 39
3.5 QuadraticFitSearch 43
3.6 Shubert-PiyavskiiMethod 45
3.7 BisectionMethod 49
3.8 Summary 51
3.9 Exercises 51
4 Local Descent 53
4.1 DescentDirectionIteration 53
4.2 LineSearch 54
4.3 ApproximateLineSearch 55
4.4 TrustRegionMethods 61
4.5 TerminationConditions 63
4.6 Summary 66
4.7 Exercises 66
contents ix
5 First-Order Methods 69
5.1 GradientDescent 69
5.2 ConjugateGradient 70
5.3 Momentum 75
5.4 NesterovMomentum 76
5.5 Adagrad 77
5.6 RMSProp 78
5.7 Adadelta 78
5.8 Adam 79
5.9 HypergradientDescent 80
5.10 Summary 84
5.11 Exercises 84
6 Second-Order Methods 87
6.1 Newton’sMethod 87
6.2 SecantMethod 91
6.3 Quasi-NewtonMethods 91
6.4 Summary 95
6.5 Exercises 95
7 Direct Methods 99
7.1 CyclicCoordinateSearch 99
7.2 Powell’sMethod 100
7.3 Hooke-Jeeves 102
7.4 GeneralizedPatternSearch 103