Reinforcement learning and dynamic programming using function approximators

282 Pages·2010·7.83 MB·English

LucianBus¸oniu,RobertBabusˇka,BartDeSchutter,andDamienErnst
Reinforcementlearningand
dynamicprogrammingusing
functionapproximators

Preface
Controlsystems are making a tremendousimpact on our society. Thoughinvisible
tomostusers,theyareessentialfortheoperationofnearlyalldevices–frombasic
homeappliancestoaircraftandnuclearpowerplants.Apartfromtechnicalsystems,
theprinciplesofcontrolareroutinelyappliedandexploitedinavarietyofdisciplines
suchaseconomics,medicine,socialsciences,andartificialintelligence.
A commondenominatorinthediverseapplicationsofcontrolistheneedtoin-
fluenceormodifythebehaviorofdynamicsystemstoattainprespecifiedgoals.One
approachtoachievethisistoassignanumericalperformanceindextoeachstatetra-
jectoryofthesystem.Thecontrolproblemisthensolvedbysearchingforacontrol
policythatdrivesthesystemalongtrajectoriescorrespondingtothebestvalueofthe
performanceindex. This approachessentially reducesthe problemof finding good
controlpoliciestothesearchforsolutionsofamathematicaloptimizationproblem.
Early work in the field of optimal control dates back to the 1940s with the pi-
oneering research of Pontryagin and Bellman. Dynamic programming(DP), intro-
ducedby Bellman,isstill amongthe state-of-the-arttoolscommonlyusedto solve
optimalcontrolproblemswhenasystemmodelisavailable.Thealternativeideaof
findingasolutionintheabsenceofa modelwasexploredasearlyasthe1960s.In
the1980s,arevivalofinterestinthismodel-freeparadigmledtothedevelopmentof
thefieldofreinforcementlearning(RL).ThecentralthemeinRLresearchisthede-
signofalgorithmsthatlearncontrolpoliciessolelyfromtheknowledgeoftransition
samplesortrajectories,whicharecollectedbeforehandorbyonlineinteractionwith
thesystem.MostapproachesdevelopedtotackletheRLproblemarecloselyrelated
toDPalgorithms.
AcoreobstacleinDPandRListhatsolutionscannotberepresentedexactlyfor
problemswithlargediscretestate-actionspacesorcontinuousspaces.Instead,com-
pactrepresentationsrelyingonfunctionapproximatorsmustbeused.Thischallenge
was alreadyrecognizedwhile the first DP techniqueswere being developed.How-
ever, it has only been in recent years – and largely in correlationwith the advance
of RL – that approximation-based methods have grown in diversity, maturity, and
efficiency,enablingRLandDPtoscaleuptorealisticproblems.
This book provides an accessible in-depth treatment of reinforcement learning
and dynamicprogrammingmethodsusing functionapproximators.We start with a
concise introduction to classical DP and RL, in order to build the foundation for
the remainderofthe book.Next, we presentan extensivereviewof state-of-the-art
approachestoDPandRLwithapproximation.Theoreticalguaranteesareprovided
on the solutionsobtained,and numericalexamplesand comparisonsare used to il-
lustrate the propertiesof the individualmethods. The remainingthree chaptersare
i
ii
dedicatedtoadetailedpresentationofrepresentativealgorithmsfromthethreema-
jor classes of techniques: value iteration, policy iteration, and policy search. The
propertiesandtheperformanceofthesealgorithmsarehighlightedinsimulationand
experimentalstudiesonarangeofcontrolapplications.
We believe that this balanced combination of practical algorithms, theoretical
analysis, and comprehensive examples makes our book suitable not only for re-
searchers,teachers,andgraduatestudentsinthefieldsofoptimalandadaptivecon-
trol, machine learning and artificial intelligence, but also for practitioners seeking
novelstrategiesforsolvingchallengingreal-lifecontrolproblems.
This book can be read in several ways. Readers unfamiliar with the field are
advised to start with Chapter 1 for a gentle introduction,and continue with Chap-
ter 2 (which discusses classical DP and RL) and Chapter 3 (which considers
approximation-basedmethods). Those who are familiar with the basic concepts of
RL andDPmayconsultthelist ofnotationsgivenattheendofthebook,andthen
startdirectlywithChapter3.Thisfirstpartofthebookissufficienttogetanoverview
ofthefield.Thereafter,readerscanpickanycombinationofChapters4to6,depend-
ing on their interests: approximate value iteration (Chapter 4), approximate policy
iterationandonlinelearning(Chapter5),orapproximatepolicysearch(Chapter6).
Supplementaryinformationrelevant to this book,including a complete archive
ofthecomputercodeusedintheexperimentalstudies,isavailableattheWebsite:
http://www.dcsc.tudelft.nl/rlbook/
Comments,suggestions,orquestionsconcerningthebookor theWeb site arewel-
come. Interested readers are encouragedto get in touch with the authorsusing the
contactinformationontheWebsite.
Theauthorshavebeeninspiredovertheyearsbymanyscientistswhoundoubt-
edly left their mark on this book; in particular by Louis Wehenkel, Pierre Geurts,
Guy-Bart Stan, Re´mi Munos, Martin Riedmiller, and Michail Lagoudakis. Pierre
Geurts also provided the computer program for building ensembles of regression
trees, used in several examples in the book. This work would not have been pos-
sible withoutourcolleagues,students, and the excellentprofessionalenvironments
atthe DelftCenter forSystemsandControlof the DelftUniversityof Technology,
theNetherlands,theMontefioreInstituteoftheUniversityofLie`ge,Belgium,andat
Supe´lecRennes,France.AmongourcolleaguesinDelft,JustinRicedeservesspecial
mentionforcarefullyproofreadingthemanuscript.Toallthesepeopleweextendour
sincerethanks.
WethankSamGeforgivingustheopportunitytopublishourbookwithTaylor
&FrancisCRCPress,andtheeditorialandproductionteamatTaylor&Francisfor
their valuable help. We gratefully acknowledge the financial support of the BSIK-
ICISproject“InteractiveCollaborativeInformationSystems”(grantno.BSIK03024)
and the Dutch fundingorganizationsNWO and STW. Damien Ernst is a Research
Associate of the FRS-FNRS, the financial support of which he acknowledges. We
appreciatethekindpermissionofferedbythe IEEEtoreproducematerialfromour
previousworksoverwhichtheyholdcopyright.
iii
Finally, we thank our families for their continual understanding, patience, and
support.
LucianBus¸oniu
RobertBabusˇka
BartDeSchutter
DamienErnst
November2009

Abouttheauthors
LucianBus¸oniuisapostdoctoralfellowattheDelftCenterforSystemsandControl
ofDelftUniversityof Technology,in the Netherlands.He receivedhis PhD degree
(cumlaude)in2009fromtheDelftUniversityofTechnology,andhisMScdegreein
2003fromtheTechnicalUniversityofCluj-Napoca,Romania.Hiscurrentresearch
interests include reinforcement learning and dynamic programming with function
approximation,intelligentandlearningtechniquesforcontrolproblems,andmulti-
agentlearning.
Robert Babusˇka is a full professor at the Delft Center for Systems and Control
of Delft University of Technologyin the Netherlands. He received his PhD degree
(cumlaude) in Control in 1997 from the Delft University of Technology, and his
MSc degree(with honors)in ElectricalEngineeringin 1990fromCzech Technical
University,Prague.Hisresearchinterestsincludefuzzysystemsmodelingandiden-
tification, data-driven construction and adaptation of neuro-fuzzy systems, model-
basedfuzzycontrolandlearningcontrol.Heisactiveinapplyingthesetechniquesin
robotics,mechatronics,andaerospace.
BartDeSchutterisafullprofessorattheDelftCenterforSystemsandControland
at the Marine & TransportTechnologydepartmentof Delft University of Technol-
ogy in the Netherlands. He received the PhD degree in Applied Sciences (summa
cumlaudewithcongratulationsoftheexaminationjury)in1996fromK.U.Leuven,
Belgium.Hiscurrentresearchinterestsincludemulti-agentsystems,hybridsystems
control,discrete-eventsystems,andcontrolofintelligenttransportationsystems.
DamienErnst receivedthe MSc and PhD degreesfromthe Universityof Lie`gein
1998 and 2003, respectively. He is currently a Research Associate of the Belgian
FRS-FNRSandheisaffiliatedwiththeSystemsandModelingResearchUnitofthe
UniversityofLie`ge.DamienErnstspenttheperiod2003–2006withtheUniversity
ofLie`geasaPostdoctoralResearcheroftheFRS-FNRSandheldduringthisperiod
positionsasvisitingresearcheratCMU,MITandETH.Hespenttheacademicyear
2006–2007workingatSupe´lec(France)asprofessor.Hismainresearchinterestsare
inthefieldsofpowersystemdynamics,optimalcontrol,reinforcementlearning,and
designofdynamictreatmentregimes.
v

Contents
1 Introduction 1
1.1 Thedynamicprogrammingandreinforcementlearningproblem . . 2
1.2 Approximationindynamicprogrammingandreinforcementlearning 5
1.3 Aboutthisbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Anintroductiontodynamicprogrammingandreinforcementlearning 11
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Markovdecisionprocesses . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Deterministicsetting . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Stochasticsetting . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Valueiteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Model-basedvalueiteration . . . . . . . . . . . . . . . . . 23
2.3.2 Model-freevalueiterationandtheneedforexploration . . . 28
2.4 Policyiteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.1 Model-basedpolicyiteration . . . . . . . . . . . . . . . . . 31
2.4.2 Model-freepolicyiteration . . . . . . . . . . . . . . . . . . 37
2.5 Policysearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.6 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 41
3 Dynamicprogrammingandreinforcementlearninginlargeandcontin-
uousspaces 43
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2 Theneedforapproximationinlargeandcontinuousspaces . . . . . 47
3.3 Approximationarchitectures . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 Parametricapproximation . . . . . . . . . . . . . . . . . . 49
3.3.2 Nonparametricapproximation . . . . . . . . . . . . . . . . 51
3.3.3 Comparisonofparametricandnonparametricapproximation 53
3.3.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Approximatevalueiteration . . . . . . . . . . . . . . . . . . . . . 54
3.4.1 Model-basedvalueiterationwithparametricapproximation 55
3.4.2 Model-freevalueiterationwithparametricapproximation . 58
3.4.3 Valueiterationwithnonparametricapproximation. . . . . . 62
3.4.4 Convergenceandtheroleofnonexpansiveapproximation . 63
3.4.5 Example:ApproximateQ-iterationforaDCmotor . . . . . 66
3.5 Approximatepolicyiteration . . . . . . . . . . . . . . . . . . . . . 71
3.5.1 Value iteration-like algorithms for approximate policy
evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
vii
viii
3.5.2 Model-free policy evaluation with linearly parameterized
approximation . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5.3 Policyevaluationwithnonparametricapproximation . . . . 84
3.5.4 Model-basedapproximatepolicyevaluationwithrollouts . . 84
3.5.5 Policyimprovementandapproximatepolicyiteration . . . . 85
3.5.6 Theoreticalguarantees . . . . . . . . . . . . . . . . . . . . 88
3.5.7 Example:Least-squarespolicyiterationforaDCmotor . . 90
3.6 Findingvaluefunctionapproximatorsautomatically . . . . . . . . 95
3.6.1 Basisfunctionoptimization . . . . . . . . . . . . . . . . . 96
3.6.2 Basisfunctionconstruction. . . . . . . . . . . . . . . . . . 98
3.6.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
3.7 Approximatepolicysearch . . . . . . . . . . . . . . . . . . . . . . 101
3.7.1 Policygradientandactor-criticalgorithms . . . . . . . . . . 102
3.7.2 Gradient-freepolicysearch . . . . . . . . . . . . . . . . . . 107
3.7.3 Example:Gradient-freepolicysearchforaDCmotor . . . . 109
3.8 Comparisonofapproximatevalueiteration,policyiteration,andpol-
icysearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
3.9 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 114
4 Approximatevalueiterationwithafuzzyrepresentation 117
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2 FuzzyQ-iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.2.1 ApproximationandprojectionmappingsoffuzzyQ-iteration 119
4.2.2 SynchronousandasynchronousfuzzyQ-iteration . . . . . . 123
4.3 AnalysisoffuzzyQ-iteration . . . . . . . . . . . . . . . . . . . . . 127
4.3.1 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.3.2 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.3.3 Computationalcomplexity . . . . . . . . . . . . . . . . . . 140
4.4 Optimizingthemembershipfunctions . . . . . . . . . . . . . . . . 141
4.4.1 Ageneralapproachtomembershipfunctionoptimization. . 141
4.4.2 Cross-entropyoptimization. . . . . . . . . . . . . . . . . . 143
4.4.3 Fuzzy Q-iteration with cross-entropy optimization of the
membershipfunctions . . . . . . . . . . . . . . . . . . . . 144
4.5 Experimentalstudy . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.5.1 DCmotor:Convergenceandconsistencystudy . . . . . . . 146
4.5.2 Two-link manipulator: Effects of action interpolation, and
comparisonwithfittedQ-iteration . . . . . . . . . . . . . . 152
4.5.3 Invertedpendulum:Real-timecontrol . . . . . . . . . . . . 157
4.5.4 Caronthehill:Effectsofmembershipfunctionoptimization 160
4.6 Summaryanddiscussion . . . . . . . . . . . . . . . . . . . . . . . 164
5 Approximatepolicyiterationforonlinelearningandcontinuous-action
control 167
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.2 Arecapitulationofleast-squarespolicyiteration . . . . . . . . . . 168