Table Of ContentCommunications and Control Engineering
Forfurthervolumes:
www.springer.com/series/61
Hyeong Soo Chang (cid:2) Jiaqiao Hu (cid:2) Michael C. Fu (cid:2)
Steven I. Marcus
Simulation-Based
Algorithms
for Markov
Decision Processes
Second Edition
HyeongSooChang MichaelC.Fu
Dept.ofComputerScienceandEngineering SmithSchoolofBusiness
SogangUniversity UniversityofMaryland
Seoul,SouthKorea CollegePark,MD,USA
JiaqiaoHu StevenI.Marcus
Dept.AppliedMathematics&Statistics Dept.Electrical&ComputerEngineering
StateUniversityofNewYork UniversityofMaryland
StonyBrook,NY,USA CollegePark,MD,USA
ISSN0178-5354 CommunicationsandControlEngineering
ISBN978-1-4471-5021-3 ISBN978-1-4471-5022-0(eBook)
DOI10.1007/978-1-4471-5022-0
SpringerLondonHeidelbergNewYorkDordrecht
LibraryofCongressControlNumber:2013933558
©Springer-VerlagLondon2007,2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To JungWonandthree littlerascals, Won,
Kyeong &Min,whochanged mydaysinto
a wholeworld ofwonders andjoys
– H.S.Chang
To myfamily–J.Hu
To mymother,forcontinuoussupport, and to
Lara &David,formixturesofjoy&laughter
–M.C.Fu
To Shelley,Jeremy,and Tobin–S.Marcus
Preface to the 2nd Edition
Markov decision process (MDP) models are widely used for modeling sequential
decision-making problems that arise in engineering, computer science, operations
research,economics,andothersocialsciences.However,itiswellknownthatmany
real-worldproblemsmodeledbyMDPshavehugestateand/oractionspaces,lead-
ingtothewell-knowncurseofdimensionality,whichmakessolutionoftheresult-
ing models intractable. In other cases, the system of interest is complex enough
that it is not feasible to explicitly specify some of the MDP model parameters,
but simulated sample paths can be readily generated (e.g., for random state tran-
sitions and rewards), albeit at a non-trivial computational cost. For these settings,
wehavedevelopedvarioussamplingandpopulation-basednumericalalgorithmsto
overcomethecomputationaldifficultiesofcomputinganoptimalsolutioninterms
of a policy and/or value function. Specific approaches include multi-stage adap-
tive sampling, evolutionary policy iteration and random policy search, and model
reference adaptive search. The first edition of this book brought together these al-
gorithms and presented them in a unified manner accessible to researchers with
varyinginterestsandbackground.Inadditiontoprovidingnumerousspecificalgo-
rithms, the exposition included both illustrative numerical examples and rigorous
theoretical convergence results. This book reflects the latest developments of the
theories and the relevant algorithms developed by the authors in the MDP field,
integrating them into the first edition, and presents an updated account of the top-
ics that have emerged since the publication of the first edition over six years ago.
Specifically, novel approaches include a stochastic approximation framework for
a class of simulation-based optimization algorithms and applications into MDPs
and a population-based on-line simulation-based algorithm called approximation
stochasticannealing.Thesesimulation-basedapproachesaredistinctfrombutcom-
plementarytothosecomputationalapproachesforsolvingMDPsbasedonexplicit
state-spacereduction,suchasneuro-dynamicprogrammingorreinforcementlearn-
ing; in fact, the computational gains achieved through approximations and para-
meterizationstoreducethesizeofthestatespacecanbeincorporatedintomostof
thealgorithmsinthisbook.
vii
viii Prefacetothe2ndEdition
Ourfocusisoncomputationalapproachesforcalculatingorestimatingoptimal
valuefunctionsandfindingoptimalpolicies(possiblyinarestrictedpolicyspace).
Asaconsequence,ourtreatmentdoesnotincludethefollowingtopicsfoundinmost
booksonMDPs:
(i) characterization of fundamental theoretical properties of MDPs, such as exis-
tenceofoptimalpoliciesanduniquenessoftheoptimalvaluefunction;
(ii) paradigmsformodelingcomplexreal-worldproblemsusingMDPs.
In particular, we eschew the technical mathematics associated with defining con-
tinuous state and action space MDP models. However, we do provide a rigorous
theoreticaltreatmentofconvergencepropertiesofthealgorithms.Thus,thisbookis
aimedatresearchersinMDPsandappliedprobabilitymodelingwithaninterestin
numericalcomputation.Themathematicalprerequisitesarerelativelymild:mainly
a strong grounding in calculus-based probability theory and some familiarity with
Markov decision processes or stochastic dynamic programming; as a result, this
book is meant to be accessible to graduate students, particularly those in control,
operationsresearch,computerscience,andeconomics.
We begin with a formal description of the discounted reward MDP framework
in Chap. 1, including both the finite- and infinite-horizon settings and summariz-
ingtheassociatedoptimalityequations.Wethenpresentthewell-knownexactso-
lution algorithms, value iteration and policy iteration, and outline a framework of
rolling-horizoncontrol(alsocalledreceding-horizoncontrol)asanapproximateso-
lution methodology for solving MDPs, in conjunction with simulation-based ap-
proaches covered later in the book. We conclude with a brief survey of other re-
cently proposed MDP solution techniques designed to break the curse of dimen-
sionality.
In Chap. 2, we present simulation-based algorithms for estimating the opti-
mal value function in finite-horizon MDPs with large (possibly uncountable)state
spaces,wheretheusualtechniquesofpolicyiterationandvalueiterationareeither
computationally impractical or infeasible to implement. We present two adaptive
sampling algorithms that estimate the optimal value function by choosing actions
tosampleineachstatevisitedonafinite-horizonsimulatedsamplepath.Thefirst
approachbuildsupontheexpectedregretanalysisofmulti-armedbanditmodelsand
uses upper confidence bounds to determine which action to sample next, whereas
thesecondapproachusesideasfromlearningautomatatodeterminethenextsam-
pledaction.Thefirstapproachisalsothepredecessorofacloselyrelatedapproach
in artificial intelligence (AI) called Monte Carlo tree search that led to a break-
throughindevelopingthecurrentbestcomputerGo-playingprograms(seeSect.2.3
Notes).
Chapter 3 considers infinite-horizon problems and presents evolutionary ap-
proachesforfindinganoptimalpolicy.Thealgorithmsinthischapterworkwitha
populationofpolicies—incontrasttotheusualpolicyiterationapproach,whichup-
datesasinglepolicy—andaretargetedatproblemswithlargeactionspaces(again
Prefacetothe2ndEdition ix
possibly uncountable) and relatively small state spaces. Although the algorithms
are presented for the case where the distributions on state transitions and rewards
are known explicitly, extension to the setting when this is not the case is also dis-
cussed,wherefinite-horizonsimulatedsamplepathswouldbeusedtoestimatethe
valuefunctionforeachpolicyinthepopulation.
In Chap.4,weconsider aglobaloptimizationapproachcalledmodelreference
adaptive search (MRAS), which provides a broad framework for updating a prob-
ability distribution over the solution space in a way that ensures convergence to
anoptimalsolution.Afterintroducingthetheoryandconvergenceresultsinagen-
eral optimization problem setting, we apply the MRAS approach to various MDP
settings. For the finite- and infinite-horizon settings, we show how the approach
canbeusedtoperformoptimizationinpolicyspace.InthesettingofChap.3,we
show how MRAS can be incorporated to further improve the exploration step in
theevolutionaryalgorithmspresentedthere.Moreover,forthefinite-horizonsetting
with both large state and action spaces, we combine the approaches of Chaps. 2
and 4 and propose a method for sampling the state and action spaces. Finally, we
present a stochastic approximation framework for studying a class of simulation-
and sampling-based optimization algorithms. We illustrate the framework through
an algorithm instantiation called model-based annealing random search (MARS)
anddiscussitsapplicationtofinite-horizonMDPs.
In Chap. 5, we consider an approximate rolling-horizon control framework for
solving infinite-horizon MDPs with large state/action spaces in an on-line manner
bysimulation.Specifically,weconsiderpoliciesinwhichthesystem(eithertheac-
tualsystemitselforasimulationmodelofthesystem)evolvestoaparticularstate
thatisobserved,andtheactiontobetakeninthatparticularstateisthencomputed
on-line at the decision time, with a particular emphasis on the use of simulation.
We first present an updatingschemeinvolvingmultiplicativeweights for updating
a probability distribution over a restricted set of policies; this scheme can be used
to estimate the optimal value function over this restricted set by sampling on the
(restricted)policyspace.Thelower-boundestimateoftheoptimalvaluefunctionis
used for constructing on-line control policies, called (simulated) policy switching
andparallelrollout.Wealsodiscussanupper-boundbasedmethod,calledhindsight
optimization. Finally, we present an algorithm, called approximate stochastic an-
nealing,whichcombinesQ-learningwiththeMARSalgorithmofSection4.6.1to
directlysearchthepolicyspace.
Therelationshipbetweenthechaptersand/orsectionsofthebookisshownbe-
low. After reading Chap. 1, Chaps. 2, 3, and 5 can pretty much be read indepen-
dently,althoughChap.5doesalludetoalgorithmsineachofthepreviouschapters,
andthenumericalexampleinSect.5.1istakenfromSect.2.1.Thefirsttwosections
ofChap.4presentageneralglobaloptimizationapproach,whichisthenappliedto
MDPsinthesubsequentSects.4.3,4.4and4.5,wherethelattertwobuilduponwork
inChaps.3and2,respectively.ThelastsectionofChap.4dealswithastochastic
approximationframeworkforaclassofoptimizationalgorithmsanditsapplications
toMDPs.
x Prefacetothe2ndEdition
Chap.1
(cid:2) (cid:8) Sect.4.1
(cid:2) (cid:4) (cid:6) (cid:8)
(cid:2)(cid:2) (cid:4) (cid:6) (cid:8)(cid:8) (cid:10) (cid:12)(cid:14)(cid:16)
(cid:2) (cid:8) (cid:16)
(cid:2)(cid:3)(cid:2) (cid:4)(cid:5)(cid:4) (cid:6)(cid:7) (cid:8)(cid:9) (cid:10)(cid:11) (cid:12) (cid:14) (cid:16)(cid:17)
(cid:12) (cid:14)
Chap.2 Chap.3 Chap.5 Sect.4.3 (cid:12) (cid:14) Sect.4.2
(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:18)(cid:19)(cid:18)S(cid:18)ec(cid:18)(cid:12)(cid:12)(cid:13)t.4(cid:18).4(cid:14)(cid:18)(cid:19)(cid:14)(cid:14)(cid:14)(cid:14)(cid:15)(cid:6)(cid:6)(cid:7)Sect.4.6
Sect.4.5
Finally, we acknowledge the financial support of several US Federal fund-
ing agencies for this work: the National Science Foundation (under Grants DMI-
9988867,DMI-0323220,CMMI-0900332,CNS-0926194,CMMI-0856256,EECS-
0901543,andCMMI-1130761),theAirForceOfficeofScientificResearch(under
Grants F496200110161, FA95500410210, and FA95501010340), and the Depart-
mentofDefense.
Seoul,SouthKorea HyeongSooChang
StonyBrook,NY,USA JiaqiaoHu
CollegePark,MD,USA MichaelFu
CollegePark,MD,USA SteveMarcus
Contents
1 MarkovDecisionProcesses . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 OptimalityEquations . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 PolicyIterationandValueIteration . . . . . . . . . . . . . . . . . 5
1.3 Rolling-HorizonControl . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 SurveyofPreviousWorkonComputationalMethods . . . . . . . 8
1.5 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 PreviewofComingAttractions . . . . . . . . . . . . . . . . . . . 13
1.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 Multi-stageAdaptiveSamplingAlgorithms . . . . . . . . . . . . . . 19
2.1 UpperConfidenceBoundSampling . . . . . . . . . . . . . . . . . 21
2.1.1 RegretAnalysisinMulti-armedBandits . . . . . . . . . . 21
2.1.2 AlgorithmDescription . . . . . . . . . . . . . . . . . . . . 22
2.1.3 AlternativeEstimators . . . . . . . . . . . . . . . . . . . . 25
2.1.4 ConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . 25
2.1.5 NumericalExample . . . . . . . . . . . . . . . . . . . . . 33
2.2 PursuitLearningAutomataSampling . . . . . . . . . . . . . . . . 37
2.2.1 AlgorithmDescription . . . . . . . . . . . . . . . . . . . . 42
2.2.2 ConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . 44
2.2.3 ApplicationtoPOMDPs . . . . . . . . . . . . . . . . . . . 52
2.2.4 NumericalExample . . . . . . . . . . . . . . . . . . . . . 54
2.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3 Population-BasedEvolutionaryApproaches . . . . . . . . . . . . . . 61
3.1 EvolutionaryPolicyIteration . . . . . . . . . . . . . . . . . . . . 63
3.1.1 PolicySwitching . . . . . . . . . . . . . . . . . . . . . . . 63
3.1.2 PolicyMutationandPopulationGeneration. . . . . . . . . 65
3.1.3 StoppingRule . . . . . . . . . . . . . . . . . . . . . . . . 65
3.1.4 ConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . 66
3.1.5 Parallelization . . . . . . . . . . . . . . . . . . . . . . . . 67
3.2 EvolutionaryRandomPolicySearch . . . . . . . . . . . . . . . . 67
xi
xii Contents
3.2.1 PolicyImprovementwithRewardSwapping . . . . . . . . 68
3.2.2 Exploration . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2.3 ConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . 73
3.3 NumericalExamples . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.3.1 AOne-DimensionalQueueingExample . . . . . . . . . . 76
3.3.2 ATwo-DimensionalQueueingExample . . . . . . . . . . 83
3.4 ExtensiontoSimulation-BasedSetting . . . . . . . . . . . . . . . 86
3.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4 ModelReferenceAdaptiveSearch . . . . . . . . . . . . . . . . . . . 89
4.1 TheModelReferenceAdaptiveSearchMethod . . . . . . . . . . . 91
4.1.1 TheMRAS Algorithm(IdealizedVersion) . . . . . . . . . 92
0
4.1.2 TheMRAS Algorithm(AdaptiveMonteCarloVersion). . 96
1
4.1.3 TheMRAS Algorithm(StochasticOptimization) . . . . . 98
2
4.2 ConvergenceAnalysisofMRAS . . . . . . . . . . . . . . . . . . 101
4.2.1 MRAS Convergence . . . . . . . . . . . . . . . . . . . . 101
0
4.2.2 MRAS Convergence . . . . . . . . . . . . . . . . . . . . 107
1
4.2.3 MRAS Convergence . . . . . . . . . . . . . . . . . . . . 117
2
4.3 ApplicationofMRAStoMDPsviaDirectPolicyLearning . . . . 131
4.3.1 Finite-HorizonMDPs . . . . . . . . . . . . . . . . . . . . 131
4.3.2 Infinite-HorizonMDPs . . . . . . . . . . . . . . . . . . . 132
4.3.3 MDPswithLargeStateSpaces . . . . . . . . . . . . . . . 132
4.3.4 NumericalExamples. . . . . . . . . . . . . . . . . . . . . 135
4.4 ApplicationofMRAStoInfinite-HorizonMDPsinPopulation-
BasedEvolutionaryApproaches . . . . . . . . . . . . . . . . . . . 141
4.4.1 AlgorithmDescription . . . . . . . . . . . . . . . . . . . . 142
4.4.2 NumericalExamples. . . . . . . . . . . . . . . . . . . . . 143
4.5 ApplicationofMRAStoFinite-HorizonMDPsUsingAdaptive
Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6 AStochasticApproximationFramework . . . . . . . . . . . . . . 148
4.6.1 Model-BasedAnnealingRandomSearch . . . . . . . . . . 149
4.6.2 ApplicationofMARStoFinite-HorizonMDPs. . . . . . . 166
4.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
5 On-LineControlMethodsviaSimulation . . . . . . . . . . . . . . . 179
5.1 SimulatedAnnealingMultiplicativeWeightsAlgorithm . . . . . . 183
5.1.1 BasicAlgorithmDescription . . . . . . . . . . . . . . . . 184
5.1.2 ConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . 185
5.1.3 ConvergenceoftheSamplingVersionoftheAlgorithm . . 189
5.1.4 NumericalExample . . . . . . . . . . . . . . . . . . . . . 191
5.1.5 SimulatedPolicySwitching . . . . . . . . . . . . . . . . . 194
5.2 Rollout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
5.2.1 ParallelRollout . . . . . . . . . . . . . . . . . . . . . . . 197
5.3 HindsightOptimization . . . . . . . . . . . . . . . . . . . . . . . 199
5.3.1 NumericalExample . . . . . . . . . . . . . . . . . . . . . 200
5.4 ApproximateStochasticAnnealing . . . . . . . . . . . . . . . . . 204
