Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions

1137 Pages·2022·31.296 MB·English

by  Warren B. Powell

ReinforcementLearningandStochasticOptimization:
AUnifiedFrameworkforSequentialDecisions
Reinforcement Learning and Stochastic
Optimization
A Unified Framework for Sequential Decisions
Warren B. Powell
PrincetonUniversity
Princeton,NJ
Thiseditionfirstpublished2022
©2022JohnWiley&Sons,Inc.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or
transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingor
otherwise,exceptaspermittedbylaw.Adviceonhowtoobtainpermissiontoreusematerialfrom
thistitleisavailableathttp://www.wiley.com/go/permissions.
TherightofWarrenB.Powelltobeidentifiedastheauthorofthisworkhasbeenassertedin
accordancewithlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice
111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWiley
productsvisitusatwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Some
contentthatappearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Thecontentsofthisworkareintendedtofurthergeneralscientificresearch,understanding,and
discussiononlyandarenotintendedandshouldnotberelieduponasrecommendingor
promotingscientificmethod,diagnosis,ortreatmentbyphysiciansforanyparticularpatient.In
viewofongoingresearch,equipmentmodifications,changesingovernmentalregulations,andthe
constantflowofinformationrelatingtotheuseofmedicines,equipment,anddevices,thereaderis
urgedtoreviewandevaluatetheinformationprovidedinthepackageinsertorinstructionsfor
eachmedicine,equipment,ordevicefor,amongotherthings,anychangesintheinstructionsor
indicationofusageandforaddedwarningsandprecautions.Whilethepublisherandauthorshave
usedtheirbesteffortsinpreparingthiswork,theymakenorepresentationsorwarrantieswith
respecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaimall
warranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantabilityorfitnessfora
particularpurpose.Nowarrantymaybecreatedorextendedbysalesrepresentatives,writtensales
materialsorpromotionalstatementsforthiswork.Thefactthatanorganization,website,or
productisreferredtointhisworkasacitationand/orpotentialsourceoffurtherinformationdoes
notmeanthatthepublisherandauthorsendorsetheinformationorservicestheorganization,
website,orproductmayprovideorrecommendationsitmaymake.Thisworkissoldwiththe
understandingthatthepublisherisnotengagedinrenderingprofessionalservices.Theadviceand
strategiescontainedhereinmaynotbesuitableforyoursituation.Youshouldconsultwitha
specialistwhereappropriate.Further,readersshouldbeawarethatwebsiteslistedinthiswork
mayhavechangedordisappearedbetweenwhenthisworkwaswrittenandwhenitisread.
Neitherthepublishernorauthorsshallbeliableforanylossofprofitoranyothercommercial
damages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages.
AcataloguerecordforthisbookisavailablefromtheLibraryofCongress
HardbackISBN:9781119815037;ePubISBN:9781119815051;ePDFISBN:9781119815044;
ObookISBN:9781119815068.
Coverimage:©DeniseSmit/GettyImages
CoverdesignbyWiley
Setin9.5/12.5ptSTIXTwoTextbyIntegraSoftwareServicesPvt.Ltd,Pondicherry,India
10 9 8 7 6 5 4 3 2 1
v
Contents
Preface xxv
Acknowledgments xxxi
PartI–Introduction 1
1 SequentialDecisionProblems 3
1.1 TheAudience 7
1.2 TheCommunitiesofSequentialDecisionProblems 8
1.3 OurUniversalModelingFramework 10
1.4 DesigningPoliciesforSequentialDecisionProblems 15
1.4.1 PolicySearch 15
1.4.2 PoliciesBasedonLookaheadApproximations 17
1.4.3 MixingandMatching 18
1.4.4 OptimalityoftheFourClasses 19
1.4.5 PullingitAllTogether 19
1.5 Learning 20
1.6 Themes 21
1.6.1 BlendingLearningandOptimization 21
1.6.2 BridgingMachineLearningtoSequentialDecisions 21
1.6.3 FromDeterministictoStochasticOptimization 23
1.6.4 FromSingletoMultipleAgents 26
1.7 OurModelingApproach 27
1.8 HowtoReadthisBook 27
1.8.1 OrganizationofTopics 28
1.8.2 HowtoReadEachChapter 31
1.8.3 OrganizationofExercises 32
vi Contents
1.9 BibliographicNotes 33
Exercises 34
Bibliography 38
2 CanonicalProblemsandApplications 39
2.1 CanonicalProblems 39
2.1.1 StochasticSearch–Derivative-basedandDerivative-free 40
2.1.1.1 Derivative-basedStochasticSearch 42
2.1.1.2 Derivative-freeStochasticSearch 43
2.1.2 DecisionTrees 44
2.1.3 MarkovDecisionProcesses 45
2.1.4 OptimalControl 47
2.1.5 ApproximateDynamicProgramming 50
2.1.6 ReinforcementLearning 50
2.1.7 OptimalStopping 54
2.1.8 StochasticProgramming 56
2.1.9 TheMultiarmedBanditProblem 57
2.1.10 SimulationOptimization 60
2.1.11 ActiveLearning 61
2.1.12 Chance-constrainedProgramming 61
2.1.13 ModelPredictiveControl 62
2.1.14 RobustOptimization 63
2.2 AUniversalModelingFrameworkforSequential
DecisionProblems 64
2.2.1 OurUniversalModelforSequentialDecisionProblems 65
2.2.2 ACompactModelingPresentation 68
2.2.3 MDP/RLvs.OptimalControlModelingFrameworks 68
2.3 Applications 69
2.3.1 TheNewsvendorProblems 70
2.3.1.1 BasicNewsvendor–FinalReward 70
2.3.1.2 BasicNewsvendor–CumulativeReward 71
2.3.1.3 ContextualNewsvendor 71
2.3.1.4 MultidimensionalNewsvendorProblems 72
2.3.2 Inventory/StorageProblems 73
2.3.2.1 InventoryWithoutLags 73
2.3.2.2 InventoryPlanningwithForecasts 75
2.3.2.3 LaggedDecisions 75
2.3.3 ShortestPathProblems 76
2.3.3.1 ADeterministicShortestPathProblem 76
2.3.3.2 AStochasticShortestPathProblem 77
2.3.3.3 ADynamicShortestPathProblem 78
Contents vii
2.3.3.4 ARobustShortestPathProblem 78
2.3.4 SomeFleetManagementProblems 78
2.3.4.1 TheNomadicTrucker 79
2.3.4.2 FromOneDrivertoaFleet 80
2.3.5 Pricing 80
2.3.6 MedicalDecisionMaking 81
2.3.7 ScientificExploration 82
2.3.8 MachineLearningvs.SequentialDecisionProblems 83
2.4 BibliographicNotes 85
Exercises 90
Bibliography 93
3 OnlineLearning 101
3.1 MachineLearningforSequentialDecisions 102
3.1.1 ObservationsandDatainStochasticOptimization 102
3.1.2 IndexingInput𝑥𝑛andResponse𝑦𝑛+1 103
3.1.3 FunctionsWeareLearning 103
3.1.4 SequentialLearning:FromVeryLittleDatato…MoreData 105
3.1.5 ApproximationStrategies 106
3.1.6 FromDataAnalyticstoDecisionAnalytics 108
3.1.7 Batchvs.OnlineLearning 109
3.2 AdaptiveLearningUsingExponentialSmoothing 110
3.3 LookupTableswithFrequentistUpdating 111
3.4 LookupTableswithBayesianUpdating 112
3.4.1 TheUpdatingEquationsforIndependentBeliefs 113
3.4.2 UpdatingforCorrelatedBeliefs 113
3.4.3 GaussianProcessRegression 117
3.5 ComputingBiasandVariance* 118
3.6 LookupTablesandAggregation* 121
3.6.1 HierarchicalAggregation 121
3.6.2 EstimatesofDifferentLevelsofAggregation 125
3.6.3 CombiningMultipleLevelsofAggregation 129
3.7 LinearParametricModels 131
3.7.1 LinearRegressionReview 132
3.7.2 SparseAdditiveModelsandLasso 134
3.8 RecursiveLeastSquaresforLinearModels 136
3.8.1 RecursiveLeastSquaresforStationaryData 136
3.8.2 RecursiveLeastSquaresforNonstationaryData* 138
3.8.3 RecursiveEstimationUsingMultipleObservations* 139
3.9 NonlinearParametricModels 140
3.9.1 MaximumLikelihoodEstimation 141
viii Contents
3.9.2 SampledBeliefModels 141
3.9.3 NeuralNetworks–Parametric* 143
3.9.4 LimitationsofNeuralNetworks 148
3.10 NonparametricModels* 149
3.10.1 K-NearestNeighbor 150
3.10.2 KernelRegression 151
3.10.3 LocalPolynomialRegression 153
3.10.4 DeepNeuralNetworks 154
3.10.5 SupportVectorMachines 155
3.10.6 IndexedFunctions,TreeStructures,andClustering 156
3.10.7 CommentsonNonparametricModels 157
3.11 NonstationaryLearning* 159
3.11.1 NonstationaryLearningI–MartingaleTruth 159
3.11.2 NonstationaryLearningII–TransientTruth 160
3.11.3 LearningProcesses 161
3.12 TheCurseofDimensionality 162
3.13 DesigningApproximationArchitecturesinAdaptive
Learning 165
3.14 WhyDoesItWork?** 166
3.14.1 DerivationoftheRecursiveEstimationEquations 166
3.14.2 TheSherman-MorrisonUpdatingFormula 168
3.14.3 CorrelationsinHierarchicalEstimation 169
3.14.4 ProofofProposition3.14.1 172
3.15 BibliographicNotes 174
Exercises 176
Bibliography 180
4 IntroductiontoStochasticSearch 183
4.1 IllustrationsoftheBasicStochasticOptimization
Problem 185
4.2 DeterministicMethods 188
4.2.1 A“Stochastic”ShortestPathProblem 189
4.2.2 ANewsvendorProblemwithKnownDistribution 189
4.2.3 Chance-ConstrainedOptimization 190
4.2.4 OptimalControl 191
4.2.5 DiscreteMarkovDecisionProcesses 192
4.2.6 Remarks 192
4.3 SampledModels 193
4.3.1 FormulatingaSampledModel 194
4.3.1.1 ASampledStochasticLinearProgram 194
4.3.1.2 SampledChance-ConstrainedModels 195
Contents ix
4.3.1.3 SampledParametricModels 196
4.3.2 Convergence 197
4.3.3 CreatingaSampledModel 199
4.3.4 DecompositionStrategies* 200
4.4 AdaptiveLearningAlgorithms 202
4.4.1 ModelingAdaptiveLearningProblems 202
4.4.2 Onlinevs.OfflineApplications 204
4.4.2.1 MachineLearning 204
4.4.2.2 Optimization 205
4.4.3 ObjectiveFunctionsforLearning 205
4.4.4 DesigningPolicies 209
4.5 ClosingRemarks 210
4.6 BibliographicNotes 210
Exercises 212
Bibliography 218
PartII–StochasticSearch 221
5 Derivative-BasedStochasticSearch 223
5.1 SomeSampleApplications 225
5.2 ModelingUncertainty 228
5.2.1 TrainingUncertainty𝑊1,…,𝑊𝑁 228
5.2.2 ModelUncertainty𝑆0 229
5.2.3 TestingUncertainty 230
5.2.4 PolicyEvaluation 231
5.2.5 ClosingNotes 231
5.3 StochasticGradientMethods 231
5.3.1 AStochasticGradientAlgorithm 232
5.3.2 IntroductiontoStepsizes 233
5.3.3 EvaluatingaStochasticGradientAlgorithm 235
5.3.4 ANoteonNotation 236
5.4 StylesofGradients 237
5.4.1 GradientSmoothing 237
5.4.2 Second-OrderMethods 237
5.4.3 FiniteDifferences 238
5.4.4 SPSA 240
5.4.5 ConstrainedProblems 242
5.5 ParameterOptimizationforNeuralNetworks* 242
5.5.1 ComputingtheGradient 244
5.5.2 TheStochasticGradientAlgorithm 246
x Contents
5.6 StochasticGradientAlgorithmasaSequential
DecisionProblem 247
5.7 EmpiricalIssues 248
5.8 TransientProblems* 249
5.9 TheoreticalPerformance* 250
5.10 WhyDoesitWork? 250
5.10.1 SomeProbabilisticPreliminaries 251
5.10.2 AnOlderProof* 252
5.10.3 AMoreModernProof** 256
5.11 BibliographicNotes 263
Exercises 264
Bibliography 270
6 StepsizePolicies 273
6.1 DeterministicStepsizePolicies 276
6.1.1 PropertiesforConvergence 276
6.1.2 ACollectionofDeterministicPolicies 278
6.1.2.1 ConstantStepsizes 278
6.1.2.2 GeneralizedHarmonicStepsizes 279
6.1.2.3 PolynomialLearningRates 280
6.1.2.4 McClain’sFormula 280
6.1.2.5 Search-then-ConvergeLearningPolicy 281
6.2 AdaptiveStepsizePolicies 282
6.2.1 TheCaseforAdaptiveStepsizes 283
6.2.2 ConvergenceConditions 283
6.2.3 ACollectionofStochasticPolicies 284
6.2.3.1 Kesten’sRule 285
6.2.3.2 Trigg’sFormula 286
6.2.3.3 StochasticGradientAdaptiveStepsizeRule 286
6.2.3.4 ADAM 287
6.2.3.5 AdaGrad 287
6.2.3.6 RMSProp 288
6.2.4 ExperimentalNotes 289
6.3 OptimalStepsizePolicies* 289
6.3.1 OptimalStepsizesforStationaryData 291
6.3.2 OptimalStepsizesforNonstationaryData–I 293
6.3.3 OptimalStepsizesforNonstationaryData–II 294
6.4 OptimalStepsizesforApproximateValueIteration* 297
6.5 Convergence 300
6.6 GuidelinesforChoosingStepsizePolicies 301
6.7 WhyDoesitWork* 303